Multiple Git Identities & SSH

Having multiple Git configurations is inevitable for the busy developer.  It is likely a good thing (you are using version control in multiple settings).  However, setting your development workstation for multiple git accounts can be difficult.  Also, if you want to use Git with ssh, you need to be able to differentiate multiple ssh accounts from each other.  Furthermore, you might desire to use multiple Git platforms at once (public SaaS GitHub, GitHub Enterprise, Git-o-Lite, or Bit Bucket) with each one having both personal and professional repos and accounts.  There are several concepts you need to understand before getting started with using multiple Git identities with ssh:
  • Git protocols
  • Ssh passwordless logins
  • Git remotes
Let’s talk about each of these with respect to what you need to know to get working.

Git Protocols:
Git can use multiple protocols for data transfer communications.  According to the official Git-scm documentation, there are 4:  local, HTTP, SSH, and GIT.  Local is only used, as the name implies, for local Git operations on local disk or even NFS or CIFS mounts.  So, our options for over the network/internet communications are the other 3.  Since the bare Git protocol does not offer much in terms of security (no authentication, no encryption), the optimal choices are HTTPS or SSH.  As the Git documentation mentions, there are some drawbacks to doing HTTPS (server setup, credentials caching and storage, etc), but it is certainly widely used because services such as GitHub make the server side part simple, leaving only the workstation side setup to deal with of which, for OS X users, the keychain makes things nice.  Finally, SSH is ubiquitous and provides good security.  When combined with password-less logins, SSH becomes very convenient as well.

SSH Passwordless Logins:
SSH provides great transport security as well as authentication.  Due to it’s use of public/private key pairs and a configuration file, you can create multiple SSH identities each of which can use their own public/private key pair.  By exchanging your public key with remote servers you connect to, combined with a tracking mechanism and file called known hosts which tracks machine network addresses for added security, SSH allows you to safely login without passwords.  The public/private key pair encryption with the remote host is superior security to text passwords.

Git Remotes:
Git uses the concept of a remote to track the location of the central repository per project.  While the beauty of Git is that the project repos are distributed to all contributors on a project, the best practice is the use a central server for which everyone pushes their changes back for easier tracking.  This central source server in Git is called “origin” by convention.
When you clone a repo on a remote Git server, there is a git dotfile directory created with a config file inside.  This config file tracks the remote origin URL used to locate the Git server.  This configuration file can be edited to provide the functionality you need even after you clone a repo.

Configuring Multiple Git Identities with SSH
  • You have already created SSH keys and organized them in a way you like.
    • Note:  I prefer to have different SSH keys for personal versus professional.  I also find using a naming scheme other than the default helpful in identifying which keypairs are for which use.  I realize this might be considered overkill by some but I prefer this and it also helps make the example more illustrative by showing different keys with different accounts.
  • You have already setup GitHub public, GitHub Enterprise, BitBucket, Stash, Git-o-lite, or GitLab accounts and added your desired SSH public keys to them.
  • You are using OS X or Linux.
    • No offense to my Windows buddies here, but things aren’t the same config wise on Windows and I wanted to keep this post simple.
  • For purposes of this example, I will use multiple accounts in GitHub public, GitHub Enterprise, BitBucket, and Git-o-lite.
    • Note:  this SSH config file is abridged to just show the Git relevant portions.
1) Create a SSH Config file for configuring multiple SSH identities.
cd ~/  
touch .ssh/config
2) Open the new config file and edit it as follows (customizing for your accounts/keys/etc):
vim ~/.ssh/config

#SSH Github Personal Public Account
#Maps to a per repo .git/config email setting to work
User git
IdentityFile ~/.ssh/
IdentitiesOnly yes
#LogLevel DEBUG3

#SSH Github Personal Public Account GISTS
#Maps to a per repo .git/config email setting to work
User git
IdentityFile ~/.ssh/
IdentitiesOnly yes

#SSH Github Professional Public Account
#Maps to a per repo .git/config email setting to work
User git
IdentityFile ~/.ssh/
IdentitiesOnly yes
#LogLevel DEBUG3

#SSH Github Personal Public Account GISTS
#Maps to a per repo .git/config email setting to work
User git
IdentityFile ~/.ssh/
IdentitiesOnly yes

#SSH Bitbucket Personal Public Account
#Maps to a per repo .git/config email setting to work
User git
IdentityFile ~/.ssh/
IdentitiesOnly yes
#LogLevel DEBUG3

#SSH Github Enterprise Account
#Maps to a per repo .git/config email setting to work
User myldapuserid
IdentityFile ~/.ssh/
IdentitiesOnly yes
#LogLevel DEBUG3

#SSH Private Git-o-lite
#Maps to a per repo .git/config email setting to work
User gitolite
IdentityFile ~/.ssh/
IdentitiesOnly yes

3) Finally, you will need to modify the .git/config file in each repo you have cloned and will clone to match the name you specified in the Host section of the .ssh/config file for the URL in the remote “origin” section.
This is because Git is using the Hostname in the URL to match the Git hostname for the SSH key.  Think of this as kind of a DNS lookup by Git (probably a poor analogy) to locate the SSH hostname.
Here are some examples to map to the SSH config file above.  This is especially helpful when you have multiple user accounts on the same Git provider platform (such as GitHub) and need to differentiate user accounts (and associated SSH keys) when the hostname is the same (, etc).
cat ~/git-repos/github-public/personal/project1/.git/config

--- snipped ---

[remote "origin"]
    url =
    fetch = +refs/heads/*:refs/remotes/origin/*

--- snipped ---

cat ~/git-repos/github-public/professional/project2/.git/config

--- snipped ---

[remote "origin"]
    url =
    fetch = +refs/heads/*:refs/remotes/origin/*

--- snipped ---

cat ~/git-repos/bitbucket/personal/project3/.git/config

--- snipped ---

[remote "origin"]
    url =
    fetch = +refs/heads/*:refs/remotes/origin/*

--- snipped ---

cat ~/git-repos/github-enterprise/professional/project4/.git/config

--- snipped ---

[remote "origin"]
    url =
    fetch = +refs/heads/*:refs/remotes/origin/*

--- snipped ---

cat ~/git-repos/git-o-lite/professional/project5/.git/config

--- snipped ---

[remote "origin"]
    url =
    fetch = +refs/heads/*:refs/remotes/origin/*

--- snipped ---

Closing thoughts:
The need to edit your .git/config files on new or cloned repos when the remote origin path needs modification to match your associated SSH config is a little annoying.
I am sure there is a way to automate this to match the paths setup in the .ssh/config file, but I have not embarked to do this just yet.
The problem is you would need to define what SSH identity you want to use by name and then do a post Git Clone “hook” of sorts.  Git clone post hooks don’t exist and so you are left with using some form of git template combined with a script which updates the remote origin URL with the SSH identity Host identity path you desire.  I suppose you could do a sed operation via  bash script executed from some git init customizations combined with a git clone —template, but remembering all these setup steps steps made me just opt for using the simple git command line manually after I clone.
(Note: for all existing cloned git repos you have, you’ll need to go into each one and update the remote origin URL in the .git/config file to take advantage of any custom SSH host name path you are using).
For now, you can use the following command to update the per repo .git/config remote origin URL path:
git remote -v

git remote set-url origin

git remote -v

Note that in the SSH config file examples I provided, there is a line saying #LogLevel DEBUG3.  This is a great way to see live SSH authentication debugging information as you try to authenticate to each remote GIT server.  I left them remarked out just so I could turn them on if I ever needed to troubleshoot a connection.  As you learn how to use these SSH identity approaches to multiple GIT identities, you’ll find this debug command super helpful.


Containers Don’t Really Boot

Docker has been a great advancement for mass consumption of linux based containers.  The maturation of the virtual machine boom that has been happening since the early 2000’s led to mass acceptance and deployment in public and private clouds.  To be sure, asking for bare metal today can be seen as a faux pas without some well-defined use case (like super high IO).  So, now that folks have accepted that slices of CPU, memory, and disk are good enough through well-known hypervisors (kvm, esxi, xen) for most workloads, taking the next step to containers will not be that big of a leap.  Except that now it is more common to run containers on VMs than bare metal.  So now we get a slice of a slice of a slice of resources!
Virtual machines are just what their name implies:  full machines that are virtualized.  This means they have virtual hardware that virtually boots an OS kernel and mounts a filesystem.  The OS doesn’t really know that the hardware it is running on is not real.  Containers on the other hand are not virtual machines.  Containers are fully sandboxed processes using the host machine OS Kernel.  So, when running on a VM, they are slices of VM vCPU, memory, and disk for fully sandboxed processes.  This is the part that had me perplexed for a while until I ventured to understand exactly what happens when an lxc container starts versus a virtual machine.

Boot or Start?

Let’s compare boots of CentOS Linux on virtual machines versus containers:
Virtual Machine/Bare Metal:
  • Power on, the system BIOS loads and looks for sector 0, cylinder 0 of the boot drive (Typically /dev/hda, or /dev/sda)
  • The boot drive contains the MBR which then uses a boot loader such as GRUB (typically in /boot) which locates the kernel and loads it (based on GRUB config)
  • The kernel (vmlinuz) then uncompresses itself into memory
  • Load the temporary RO root filesystem via initramfs (configured in GRUB)
  • The Kernel locates and launches the /init program from within initramfs (/sbin/init)
  • Init determines run level via /etc/inittab and executes startup scripts
  • Per fstab entry, root filesystem completes integrity check and then is re-mounted as RW.
  • You get a shell via /bin/sh
  • Docker tells LXC (now libcontainer) to start a new container using the config in your Dockerfile
    sudo docker run -i -t centos7 /bin/bash 
    • Runs lxc-create or libcontainer equivalent with params (example)
      lxc-create -t mytemplate -f lxc.conf -n mycontainer
  • Docker on rhel/centos/fedora systems use device mapper which uses a sparse file for holding container images here:
  • Docker starts the base image (directory structure) as read only, and creates the new RW (CoW) layer on top of it.
  • Docker gives you a shell via /bin/sh (if you asked for it in the Docker run or a Dockerfile config)Docker configured union filesystem (AUFS, devicemapper, overlayfs) is used for mounted root filesystem

“It is perhaps more precise to say that a linux environment is started rather than booted with containers.”

The entire linux “boot” process that a normal virtual machine goes through is essentially skipped and only the last steps where the root filesystem is loaded and a shell is launched happens.  It is perhaps more precise to say that a linux environment is “started rather than booted”.  I was also further confused by the Docker terminology which uses the word “image” to describe something different from cloud images.  When I hear “image” I think of AMI style full virtual machine images as used in clouds.  These images are different from container images used by Docker.  Docker uses the term “image” to describe what is really a minimal layered root filesystem (union mounted).  This is all a bit confusing at first until you remember that everything in Linux is a file.  If you dig around and take a look at some of the utilities to create these “base images” such as febootstrap/supermin or debootstrap you will see that they are creating clean, consistent directories and files for the linux distro output in various formats such as .img or .tar.  So, the docker “images” are really nothing more than directories and files with pre-populated with the minimum viable set of linux components and utilities you need for a functional linux system.

“This is all a bit confusing at first until you remember that everything in Linux is a file.”

When Docker LXC/libcontainer based containers boot they are really just starting a kind of super “chroot” of processes with totally sandboxed storage and networking.  They don’t need to do a full boot since they are just leveraging the OS kernel of the host system.  All they seem to need are the minimum viable linux system directory structure tools and utilities.  Additionally, because Docker caches content, things run even faster since there is less to download.  These are reasons why containers “boot” or more precisely “start” incredibly fast.  Because you don’t have to go through a fully virtualized system boot process like a virtual machine or bare metal machine, you get productive “process-wise” rapidly in your own super sandboxed linux environment.

Union File Systems and the Neo Image Zeitgeist

One cool thing Docker introduced is the use of union mount layered file systems to control the craziness of working with images.  When I say image “craziness” I might need to clarify with a refresher for those youngsters who didn’t go through the whole “anti-image” zeitgeist of the past 5 years.  Let’s rewind to the early 2000’s when people discovered the ability to create sector by sector disk copies and saved all the time of installing apps over and over again (Ghost anyone?).  Everyone was in love with images then and naturally started using them in VMs when vmware was new and hot.  It was only after years of dealing with homegrown image versions and patching problems that folks started becoming more and more anti-image.  To be sure, many people made many horrible images (especially for Windows machines) that didn’t properly get sanitized (with sysprep or similar tools) before use as a VM template which only served to exacerbate things.

Fast forward to 2005-ish when CM tools like Puppet and later Chef in 2008 were formed in the “anti-image” rebellion.  What people wanted in these modern CM tools was the ability to repeatedly build pristine machines literally from bootstrap.  What this meant to many was no images ever: PXE the world and then Chef it.  As the public cloud took off so did people’s needs to build servers at an increasingly rapid pace. PXE bootstrapping everything was just to slow and often not possible in multi-tenant cloud environments like AWS.  The compromise answer was to create super minimal “base images” (also called JEOS or Stem Cell images) which were super pristine and small.  These base images for full virtual machines booted much faster and the fact that they had very little on them didn’t matter anymore since we could reliably and cleanly install everything declaratively in code using Puppet or Chef.

Fast forward to today and folks often find that full VM’s booted and installed with cookbooks are again not fast enough for them.  Also, the complexity of using some CM tools meant that it was a fair amount of work to stand up your Puppet or Chef environments before they paid off in speed and as a single source of truth.  Enter containers.  However, just getting a container only gives you a pristine base image if you start out with one.  Beyond any pristine base container image, any customizations you might need (like installing your app software components) would require you to get back to the days of image sprawl unless you used modern CM like Puppet or Chef to provision on top of your container base images.  Docker decided to fix this old problem with a new twist by using a union mount for layered or copy on write filesystems.  What they did was take the concept of the pristine base image (which we’ve learned is nothing more than minimum viable linux bistro directories and files with components and utilities) and allow you to leave it in a pristine shape.  They then allow you to layer components on top of each other that you need leaving each layer as read only thin deltas of changes.  The automation (infra as code) offered by Docker is via the Dockerfile where machine DNA is located.  What is still yet to be determined is whether the Dockerfile is enough to get your container in the end state you desire.  For example will layered “super thin delta images” be a replacement for modern CM tools?  Or, more likely, will CM tools like Chef be used to build each thin layer delta image (TLDI).

For Whom the Batphone Calls

A Runbook for Enterprise Adoption of Open Source Software 


Enterprise leaders wanting to do a “Dev and Ops” pivot need to own up to the fact that you have to earn you keep today when choosing this path.
If shit is broke, YOU should have the power to fix it.  This idea of outsourcing all problems to a paid closed source vendor support call can be naive and lazy in many cases.  You should have the power to be your own bat phone.  When you call Batman, you are in essence calling yourself.

Retirement Home Enterprise          

For the longest time, infrastructure and application support teams have operated in a super old workflow that was uninspiring.  They got paid well to be effectively “glorified maintenance/janitorial staff.”  An operations spreadsheet/app had a name on it that says they support X and Y.  When X or Y breaks, they put a ticket in a queue and call people.  These people look at some super basic stuff like 1) Is the server up and pingable?, 2) Is the app process running?, or 3) Is there a vague, nefarious network issue?  If none of these things get hits, they shrug their shoulders and  and open a ticket with the Vendor that they pay support to and then wait.  They then report to some person that reports to some other ops person that they are “working on the issue” with the vendor.  

The vendor then has a support engineer (L1/L2) work the call and ask for sane things like logs and read the case notes.  The enterprise support person just acts like a drone in this case typing whatever the vendor support engineer says and accepts whatever the vendor provides as an answer.  Unless an answer cannot be found, the enterprise support person just accepts the answer/fix and closes the case.  If no fix can be found, it effectively is like reporting a bug (If it is not already well known).  The fixing for the bug can be soon* or indefinite.  Unless the enterprise customer is huge (spends loads on support & products) the bug fix doesn’t get addressed fast.  It gets addressed on the schedule most convenient to the vendor.  These support contracts are not cheap.  This is such a lazy, lame approach.  I call this the “Batphone Mentality” where the vendor software support superhero is always a support phone call away.  This is  especially true for closed source COTS apps whose heavy handed approach actually creates a self-reinforcing cycle of enterprise retirement home “Batphone” mentalities. The problem is that Batman worked for free for Gotham, while enterprises have to pay salaries for multiple external vendor Batmans.

“closed source COTS apps … heavy handed approach actually creates a self-reinforcing cycle of enterprise retirement home “Batphone” mentalities.” 

The Emerging Open-Source Enterprise 

When you decide that you want to control your destiny, get things fixed faster, or add features and functionality to applications and tools you need to run your business, you may be ready to make the open-source switch.  If paying loads of money for support contracts is leaving you beholden to a vendor that is not transparent and open about product bug fixes or feature roadmaps, you are effectively trapped.  Major commercial software vendors with closed source products maintain a kind of hegemony over enterprise customers in the following ways: 

  • Forcing the customer to follow often stringent tech-stack requirements to install and use the product.
  • Forcing the customer to forgo support officially if they try to deviate any part of the tech-stack.
  • Making money off parts of this “required” tech-stack with onerous licensing.
  • Releasing new versions that require forklift upgrades that are uber costly and are often not backward compatible.
  • Purposely not being open with support documentation and requiring pay/account gateways to get it.
  • Forcing a god forsaken license key or licensing process that makes installs painful and limited.
  • Dropping a product completely leaving the customer with no option. 

“Major commercial software vendors with closed source products maintain a kind of hegemony over enterprise customers.”

For years enterprises just accepted this in large part due to the following reasons:

  • They used closed source OS platforms (Windows/AIX/Mainframes) since viable open-source alternatives were not available.
  • They bought software that was closed source because developing the software was hard and costly.
  • Tools needed to write software for these closed-source platforms were not easily accessible or free ($1000+ for Visual Studio is pathetic).
  • Technology perhaps was not a key part their business (non-tech companies).
  • Staff that had requisite skills to overcome (1-4) were not widely available. 

Enter the modern tech era Circa 2007 – 2014 where open-source software is almost an American birthright and as normal and accepted as apple pie.  Open source operating systems, applications, and software development tools are widely available that can solve many (if not most) of the things an enterprise might need (Linux, OpenStack, Nagios, Hadoop, Tomcat, Apache, Rails, NodeJS, and on and on).  Public code sharing repositories like Github, Gitlab, Gitorious, Cloudforge, Sourceforge, and GoogleCode have been a revolution in open-source workflow and project accessibility.  Github in particular has been a big part of this inflection point in social coding.  There has likely been more open-source momentum in the past 7 years than in that past 30.  It is truly an amazing time. Decades from now, folks may look back and ask all us old-timers stary-eyed questions about “what was it like to live during the second open-source revolution?” like we we lived through the equivalent of the roaring twenties or some other historically significant time.

“There has likely been more open-source momentum in the past 7 years than in that past 30.” 

While the open-source movement may have not started precisely in the San Francisco bay area, the area has been home to many leaders in the movement.  Partly perhaps due to the roots (even today) of silicon valley as a haven for free thinkers and futurists.  A place also were it was ok to be distrustful of governments and “the man.”  The bay area hippie culture of the 60’s may have also contributed in part to a tradition of folks wanting things to be free or more accessible that continues.  Today, there is still a vein of this benevolent revolutionary mindset behind every open-source project.  The interesting thing that has happened is that the hippies have been replaced by benevolent libertarian capitalists.  This is very cool.  Many startups in the valley today often release their product day one as an open-source project.  They simultaneously offer paid support for customers.  In many cases, their products are based on open-source products but sold also as SaaS or cloud offerings.  This is an amazing amount of new freedom for the enterprise.

“…the hippies have been replaced by benevolent libertarian capitalists.” 

Be your own Batphone 

If you want to control your destiny, get things fixed faster, or add features and functionality you need on your own schedule, you need to be using open-source software and hiring staff that are not drones or retirees.  Here is proposal for how the modern enterprise can call their own Batphone.

1) Use open-source software
     – get access to the source code now (there is a boat load out there
     – check lists like this (
2) Hire good developers for Apps and good developers for Infra
     – Pay them well
     – Empower them
3) Be very strategic about what commercial, closed source software you buy and use
     – It needs to provide a better ROI for functionality, speed, reliability than an OSS alternative.
     – Try to pick closed source software that at least is largely standards based (it can ease a transition off the product later if needed).
     – Put pressure on the vendor to allow flexibility of the tech stack to use open-source projects (tomcat, node, etc).
4) Train your staff and engineers well
5) Accept the fact that you can now truly control your destiny.

On this last point, it is a function of having skilled developers and engineers that use open source projects and have a specific mentality.  The mentality needed is really an “Anti-Batphone” mentality.  Developers and Engineers have to own up to the fact that they are the last person standing.  They are the last solider in the bunker and have to be McGiver or Will Wheaton to get it working.  They have to believe there is no fucking Batphone.  They are the Batphone. 

A Sample Enterprise anti-Batphone Support Call Flow

1) Some important app is broke
2) Open-source monitoring software alerts the app owner/developer team 
     – Operations sees that same alert and opens a ticket
3) Internal support engineer starts working on it
     – logs, processes, system level debugging
4) Something is broken with the code (a bug)
     – Look at the damn source code, you have it!
5) Commit code changes with the fix to your CI/CD pipeline.
     – Push to prod for the fix
     – close the internal support case.

Paying for Support with an Open Source Vendor 

Paying for support with a vendor that has an open-source version of it’s product is truly a win-win.
You get the luxury of official support.
The open-source vendor gets to make money and you still get the source code.
What if you cannot fix the bug with your internal staff?  What if you need to pay for support?
No problem, do the following: 

1) Locate a vendor that provides the product platform you need as open-source + support
2) Pay these good people for support.
3) Use their product and do the following for support issues
          – Open issues on Github for problems and bugs
          – Fix the bugs yourself and submit pull requests directly to the vendor github
4) Influence product roadmaps by submitting pull requests and collaborating with the vendor:
          – this can be from your own internal staff if they are skilled enough.
          – this can be through skilled contractors that you pay to develop modifications to submit as pull requests to the vendor.
          – this can be through outright paying the vendor for adding the features directly.
5) If for whatever reason the open-source vendor decides to stop working on the project, closes shop, or will not accept your pull requests for product modifications, you still own the source code.
          – you can take over the project
          – you can fork the project and do your own thing

This is an awesome new world with loads of new freedom.  Enterprises need to embrace this and control their destiny by using open-source software and using open-source vendors.  You should have the power to be your own bat phone.  When you call Batman, you are in essence calling yourself. 
















What the Heck is an Architect?

I was never one of those little kids that told mommy and daddy what they wanted to be when they grew up.  I didn’t even start officially working in the “IT” industry until 1996.  I just kind of fell into the career because I enjoyed fixing other people’s computers all the time.  The closest I ever came to wanting a particular job as a teenager was being an architect.  You know the traditional kind like the dad the “Brady Bunch”?  In my mind an architect was always an engineer with a big desk, rulers, calculators, knowledge of complex math, physics, and materials science.  They also were visionary creative types that could come up with amazing designs like the Chrysler building, the TransAmerica Tower, or Falling Water.  In my mind these architects could do designs in an office, but also go out in the field and do some of the engineering work themselves.

The Architect - Mr. Brady


Fallingwater-Frank Lloyd-Wright

Fallingwater-Frank Lloyd-Wright

Fast forward to today when the job title of architect can take on a variety of constructs:  software, infrastructure, political, health care, network, security, traffic, etc.  The job title has morphed into any type of senior design position in some fields.  I myself am part of this trend with the job title “Infrastructure Architect” currently.  However, due to my admiration for the “traditional” architect job, I feel an obligation to at least try and transfer my ideas of what skills translate from the traditional architect into the modern IT infrastructure engineering space.  This has been the cause of some stress for me over the past three years.  I will freely admit that today I worry about what it takes to be proficient in this modern “infrastructure as code” world.  After having a few beers, I sat on my porch and tried to jot down some notes on what troubled me personally and what the causes might be.  I think I have narrowed them down:  personal expectations versus market expectations, time management, and ambition.

Personal Expectations vs Market Expectations

I would be lying to you if I did not admit that I struggle every day personally to try and meet what my own expectations are for an architect in the infrastructure space.  Infrastructure has several “silos/towers” that represent some traditional capability areas.  These are things like servers and virtualization, storage, networking, security, databases, service buses, application hosting, as well as monitoring and performance metrics.  Having deep knowledge in more than one of these towers is what I would say an architect should have.  Due to the amount of knowledge needed to know multiple silos this can take years of work.  In my case this journey has spanned over 17 years since 1997.  There is no formal curriculum so to speak for “architect” so it kind of materializes via technical jobs you take and a little serendipity.
My architect journey kind of materialized like so…

  • Hardware phase (servers/PCs/peripherals)
  • DOS phase (batch programming, emm386 fun days)
  • AS400 and VAX/VMS phase
  • Token ring and ethernet (10base2-10000) phase (jiggle it just a little bit)
  • Novell phase (3 to NDS)
  • Windows phase (over a decade of windows troubleshooting madness)
  • The whole certification industry treadmill phase (mcse/cne/ccnp/blah thank goodness this is dead)
  • The DBA phase (SQL server and Oracle)
  • Network engineering phase (ethernet to satellites to sonet rings to mpls and vpn)
  • Unix/linux phase (Solaris, Slackware, mandrake and RHEL)
  • The Perl/CPAN/Sourceforge phase
  • The php/html/css everything phase
  • The slashdot addiction phase
  • The AWS and Rackspace IaaS phase (ongoing)
  • The OSS and Github revolution phase (ongoing)
  • The ruby and cm transformation (ongoing)
  • The Hacker News addiction phase (ongoing)

The presumed belief is that the more you stuff you learn the more valuable you become from a holistic experience lens.  This is likely true in that you learn over time how multi-level dependencies and other design issues can affect resiliency, performance and scale.  Eventually the need for this kind of holistic knowledge was enough that the modern job titles of “architect” in the IT space arose.  In some ways, the architect position is seen as a more senior role (even though many organizations don’t have an HR job code with architect in it.  To be sure, I’m ambivalent on whether the job title even needs to be official).  The flip side is that going deep in n+3 or more capability silos becomes increasingly difficult to do well given the other responsibilities that often fall on the shoulders of an “architect”.  The funny thing is that these “other responsibilities” often begin to take upwards of 70% or more of the time of an architect type person.  There is a constant balancing act going on where one foot is on the hands on (development/testing/research) side of the fence, and the other foot is in things I call CCE like (collaboration, communication, and evangelism) or leadership side of the fence.  The latter half inevitably results in meeting hell.



Why do these other things increasingly take over the time of an architect type person?  I have thought a lot about this, and I believe it likely boils down in part to the holistic experience thing (others may only have a single silo view of the world and hence need help with understanding about why something could be better another way) and partly because the architect (even though the role may not even be official on paper) is saddled with the overall responsibility that things work cohesively within a given capability area.  When someone has a job responsibility to “make things work” together, it implies that they ideally understand and have proficiency in the technologies involved.  This gets increasingly difficult the more technology areas you are expected to know.  I will be the first to admit that this knowledge responsibility only works when a federated model of architectural ownership is in place.  By this I mean that having multiple architect roles within infrastructure is important and many more within an enterprise is absolutely a necessity.  The old mindset of someone having the job title of ‘Enterprise Architect” is a farce.  It is a feat of impossibility if you expect to have actual architecture efficacy in an IT organization.  The archaic “ivory tower” approach to managing architecture in a top down model is just not feasible.  To be sure, I don’t think it ever was actually effective; and so in my mind the “Enterprise Architect” is dead and has been for a while.  The point I want to make here is that there is just too much to know to be truly effective outside the limited scope of a major capability area.  The ability to be hands on and also do requisite CCE I feel is best within a more manageable size capability area.

“The ‘Enterprise Architect’ is Dead”

Pretty much since 2007 the infrastructure as a service revolution inspired by AWS has been a massive snowball of change.  This change has and continues to be awesome.  I have wholeheartedly embraced it and have been taking part in the transition for years now.  The changes have in part re-invigorated the popularity of the open source movement and with it has come unparalleled opportunities for startups.  It has inspired the creation and success what has become a literal tidal wave of open-source projects on Github.   All one has to do is look at Hacker News on any given day and see that some new project was announced that “fixes this” or “closes this gap in functionality”.  One could argue that a business could start and run without much if any proprietary commercial products today (short of the client devices).  It is an amazing time that we are living in that maybe 30 years from now will be mentioned in history books as the tipping point for mainstream open-source adoption and innovation.  All of this is super cool, especially if you are a small to medium sized startup, and can get everything you need infrastructure-wise from something like AWS or Rackspace or Digital Ocean using an API.  Even better, modern configuration management and containerization sits on top of these IaaS platforms and allows you to install packages and manage build outs rapidly using “infrastructure code” as well.  If you are a software developer and need to be productive in the infrastructure space (even if you don’t know it well) you can do so effectively with just the knowledge of a given API.

Marc Andressen’s famous callout in 2011 that “software is eating the world” has never been more true than in the IT infrastructure space.  I have felt it every day for the past 5 years now.  The challenge, specifically for me, is staying productive, efficient, and helpful in a space that is increasingly more software.  Software development to be precise.  Let’s be honest, even with the cloud NaaS revolution, most enterprises still run the bulk of their infrastructure in house.  Part of this is dealing with legacy apps and the baggage they bring, part of it is a pure cost play where the bandwidth and storage used by an enterprise can still be less in house than using AWS or Rackspace, and finally, part of this is security and performance concerns.  So with many enterprises trying to take part in this “infrastructure as code” revolution while still maintaining significant internal hardware, the need for someone to know both traditional infrastructure management and new code based approaches is critical.  To be sure, a software based management abstraction layer (api) is increasingly meaning that the work an an infrastructure architect does is more dev and less ops.  Since I want to be an architect that does both hands on coding and traditional architecture work (to remain relevant), I need time to focus on development.  Getting the time to do the work and focus is a real challenge.  But dude, the thing is I have to do this to enable infrastructure to provide any value for the app dev teams in the enterprise.  If I can’t help make internal infrastructure as easy to consume and programmable as Amazon AWS, then the Devs increasingly see me (vis-a-vi infrastructure) as a blocker and I don’t blame them.  At least for now, because of the need for supporting legacy and all the painfully inflexible COTS packages (hello Oracle), Infrastructure ends up also having to play a type of luxury one-on-one concierge service for app teams.  I don’t know how long this “concierge” style need will last, but I’m gonna be pushing as much self-service infrastructure via API as fast as I can make it happen.  Things like Openstack and SDN are making this possible for me, but doesn’t address everything needed.

Pressure on Myself  in the Valley

If you buy into the software will eat the world argument (and I feel like I do) then the question becomes:  How can I be the best at my job as an infrastructure architect?
More precisely:

  • What is valuable work?
  • What are valuable skills?

If the answers are only writing code, then there are lots of questions I have the keep me stressed.  First off, is this thing endemic to Silicon Valley only or is more widespread?  I feel like the mass transition is already well underway.  Right now the change is already prevalent in Silicon Valley.  There may be a three to five year delay before it hits middle america, but its probably coming.  But how will these changes (for all their merits) scale?  Are all the old sysadmins that can’t code pretty much SOL?  I think perhaps so.  I said the traditional sysadmin was dead in a paper published last year, but I wonder if middle america can keep up.  I mean this stuff is not exactly easy.  Is this an example of the robotization of everything?  If everyone is writing code, then who is spending time communicating and collaborating (remember this takes a buttload of time) and who maintains the code down the road?  What about all these small IT consulting companies that handle outsourced IT for the space between mom and pop businesses to the enterprise?  Is the “Geek Squad” of the near future going to need staff that are polyglots understanding CI pipelines and Git?   Writing code takes a singular level of focus and time that is already a challenge for even seasoned developers.  The needs for the people (often architects) that sit at the event horizon of software development and collaborative design and communications won’t simply go away.  The difference is now I think they need to be doing both.  Architects, as I have mentioned, often have a lack of time due to the immense gravitational pull of CCE (collaboration, communications, and evangelism) that needs to be done with all the stakeholders/customers of infrastructure.  If I were to surmise that my ideal job time split between “hands on keyboard” time coding and CCE were 50/50, then I’m gonna have to change the way I manage my time.  The balancing act of hands on work and meetings is just way fracking hard.  I may have to start being more selfish with my calendar and time.  I’m just not sure how best to solve this yet.

“The balancing act of hands on work and meetings is just way fracking hard.”  


Another issue I struggle with is the perception that the ability to do both hands on development and CCE or leadership well is commonplace in the valley and elsewhere.  This is buttressed by the examples of exceptional people like Elon Musk, Sergey Brin, Larry Page, Mark Zuckerberg, and Max Levchin as bellwethers for how to work.  I feel like these guys and others like them are truly exceptional people.  They garner media attention because they are success stories, but the translation in the valley is perhaps that their skills are as obtainable as riding a bike.  I feel like this is just not the case.  I don’t necessarily want to lend too much credence to the Malcolm Gladwell “10k hours” argument, but this shit is just not easy.  The competition here is high, especially if you truly want to make a difference and want to think of yourself as good at your job.  Perhaps I am just too hard on myself, but here lately I have gotten to where my blood pressure rises, I get hot, and feel overwhelmed when I read HackerNews.  Sometimes I feel like all the article titles are flying by my eyes like data in the matrix making me feel more inadequate by the minute (the Oracle said I was not “the one”).  Is this is the bane of type A personalities who want to be the best and are competitive?  I rarely think of myself as an “expert” at anything even if I am kicking ass at something.  Sometimes this personality trait of mine helps me and drives me to be better, while other times it exhausts me and gets me down.  Am I a little out of balance with incessantly wanting to be a better programmer, a great speaker and leader, a great skateboarder, up to date on all the latest geopolitics, and be able to write eloquent comments on HN?  Maybe.  I might need to wean myself off HN for a little bit for my own mental health.  I think when you start to get jealous of the gas station cashier for having a simple and less stressful job, it might be time to level set.  Than again, I just want to do a good job in this rapidly changing infrastructure as code landscape.  So what is a modern infrastructure architect?  Is it a software developer, a shoe salesman, or a crisis negotiator?

Wait, sorry I have another meeting to attend.  Let me know if you approve and merge what I perceive to be a market pull request.

Cool Geeks DevOps Meetup

Heya Fellow Cool Geeks!

I and some fellow geeks formed a new DevOps Meetup called “Cool Geeks DevOps.”
The hope is that we will be able to tackle some common DevOps problems in an open forum that is also fun.
I invite everyone to stop by our regular meetups to share the fun.