Multiple Git Identities & SSH

Having multiple Git configurations is inevitable for the busy developer.  It is likely a good thing (you are using version control in multiple settings).  However, setting your development workstation for multiple git accounts can be difficult.  Also, if you want to use Git with ssh, you need to be able to differentiate multiple ssh accounts from each other.  Furthermore, you might desire to use multiple Git platforms at once (public SaaS GitHub, GitHub Enterprise, Git-o-Lite, or Bit Bucket) with each one having both personal and professional repos and accounts.  There are several concepts you need to understand before getting started with using multiple Git identities with ssh:
  • Git protocols
  • Ssh passwordless logins
  • Git remotes
Let’s talk about each of these with respect to what you need to know to get working.

Git Protocols:
Git can use multiple protocols for data transfer communications.  According to the official Git-scm documentation, there are 4:  local, HTTP, SSH, and GIT.  Local is only used, as the name implies, for local Git operations on local disk or even NFS or CIFS mounts.  So, our options for over the network/internet communications are the other 3.  Since the bare Git protocol does not offer much in terms of security (no authentication, no encryption), the optimal choices are HTTPS or SSH.  As the Git documentation mentions, there are some drawbacks to doing HTTPS (server setup, credentials caching and storage, etc), but it is certainly widely used because services such as GitHub make the server side part simple, leaving only the workstation side setup to deal with of which, for OS X users, the keychain makes things nice.  Finally, SSH is ubiquitous and provides good security.  When combined with password-less logins, SSH becomes very convenient as well.

SSH Passwordless Logins:
SSH provides great transport security as well as authentication.  Due to it’s use of public/private key pairs and a configuration file, you can create multiple SSH identities each of which can use their own public/private key pair.  By exchanging your public key with remote servers you connect to, combined with a tracking mechanism and file called known hosts which tracks machine network addresses for added security, SSH allows you to safely login without passwords.  The public/private key pair encryption with the remote host is superior security to text passwords.

Git Remotes:
Git uses the concept of a remote to track the location of the central repository per project.  While the beauty of Git is that the project repos are distributed to all contributors on a project, the best practice is the use a central server for which everyone pushes their changes back for easier tracking.  This central source server in Git is called “origin” by convention.
When you clone a repo on a remote Git server, there is a git dotfile directory created with a config file inside.  This config file tracks the remote origin URL used to locate the Git server.  This configuration file can be edited to provide the functionality you need even after you clone a repo.

Configuring Multiple Git Identities with SSH
Assumptions:
  • You have already created SSH keys and organized them in a way you like.
    • Note:  I prefer to have different SSH keys for personal versus professional.  I also find using a naming scheme other than the default helpful in identifying which keypairs are for which use.  I realize this might be considered overkill by some but I prefer this and it also helps make the example more illustrative by showing different keys with different accounts.
  • You have already setup GitHub public, GitHub Enterprise, BitBucket, Stash, Git-o-lite, or GitLab accounts and added your desired SSH public keys to them.
  • You are using OS X or Linux.
    • No offense to my Windows buddies here, but things aren’t the same config wise on Windows and I wanted to keep this post simple.
  • For purposes of this example, I will use multiple accounts in GitHub public, GitHub Enterprise, BitBucket, and Git-o-lite.
    • Note:  this SSH config file is abridged to just show the Git relevant portions.
Steps:
1) Create a SSH Config file for configuring multiple SSH identities.
cd ~/  
touch .ssh/config
2) Open the new config file and edit it as follows (customizing for your accounts/keys/etc):
     
vim ~/.ssh/config
#######################################################
# CUSTOM .SSH CONFIG FILE FOR MULTIPLE SSH IDENTITIES #
#######################################################

#SSH Github Personal Public Account
#Maps to a per repo .git/config email setting to work
Host github.com-myuserid-personal
HostName github.com
User git
IdentityFile ~/.ssh/id_myuserid-personal@example.com
IdentitiesOnly yes
#LogLevel DEBUG3

#SSH Github Personal Public Account GISTS
#Maps to a per repo .git/config email setting to work
Host github.com-myuserid-personal
HostName gists.github.com
User git
IdentityFile ~/.ssh/id_myuserid-personal@example.com
IdentitiesOnly yes

#SSH Github Professional Public Account
#Maps to a per repo .git/config email setting to work
Host github.com-myuserid-professional
HostName github.com
User git
IdentityFile ~/.ssh/id_myuserid-professional@example.com
IdentitiesOnly yes
#LogLevel DEBUG3

#SSH Github Personal Public Account GISTS
#Maps to a per repo .git/config email setting to work
Host github.com-myuserid-professionall
HostName gists.github.com
User git
IdentityFile ~/.ssh/id_myuserid-professional@example.com
IdentitiesOnly yes

#SSH Bitbucket Personal Public Account
#Maps to a per repo .git/config email setting to work
Host bitbucket.org
HostName bitbucket.org
User git
IdentityFile ~/.ssh/id_myuserid-personal@example.com
IdentitiesOnly yes
#LogLevel DEBUG3

#SSH Github Enterprise Account
#Maps to a per repo .git/config email setting to work
Host github.example.com
HostName github.example.com
User myldapuserid
IdentityFile ~/.ssh/id_myuserid-professional@example.com
IdentitiesOnly yes
#LogLevel DEBUG3

#SSH Private Git-o-lite
#Maps to a per repo .git/config email setting to work
Host gitolite.example.com
HostName gitolite.example.com
User gitolite
IdentityFile ~/.ssh/id_myuserid-professional@example.com
IdentitiesOnly yes


3) Finally, you will need to modify the .git/config file in each repo you have cloned and will clone to match the name you specified in the Host section of the .ssh/config file for the URL in the remote “origin” section.
This is because Git is using the Hostname in the URL to match the Git hostname for the SSH key.  Think of this as kind of a DNS lookup by Git (probably a poor analogy) to locate the SSH hostname.
Here are some examples to map to the SSH config file above.  This is especially helpful when you have multiple user accounts on the same Git provider platform (such as GitHub) and need to differentiate user accounts (and associated SSH keys) when the hostname is the same (github.com, etc).
Examples:
cat ~/git-repos/github-public/personal/project1/.git/config

--- snipped ---

[remote "origin"]
    url = git@github.com-myuserid-personal:myuserid-personal/project1.git
    fetch = +refs/heads/*:refs/remotes/origin/*

--- snipped ---


cat ~/git-repos/github-public/professional/project2/.git/config

--- snipped ---

[remote "origin"]
    url = git@github.com-myuserid-professional:myuserid-professional/project2.git
    fetch = +refs/heads/*:refs/remotes/origin/*

--- snipped ---


cat ~/git-repos/bitbucket/personal/project3/.git/config

--- snipped ---

[remote "origin"]
    url = git@bitbucket.org:myuserid-personal/project3.git
    fetch = +refs/heads/*:refs/remotes/origin/*

--- snipped ---


cat ~/git-repos/github-enterprise/professional/project4/.git/config

--- snipped ---

[remote "origin"]
    url = git@github.example.com:myuserid-professional/project4.git
    fetch = +refs/heads/*:refs/remotes/origin/*

--- snipped ---


cat ~/git-repos/git-o-lite/professional/project5/.git/config

--- snipped ---

[remote "origin"]
    url = git@git-o-lite.example.com:myuserid-professional/project5.git
    fetch = +refs/heads/*:refs/remotes/origin/*

--- snipped ---

Closing thoughts:
The need to edit your .git/config files on new or cloned repos when the remote origin path needs modification to match your associated SSH config is a little annoying.
I am sure there is a way to automate this to match the paths setup in the .ssh/config file, but I have not embarked to do this just yet.
The problem is you would need to define what SSH identity you want to use by name and then do a post Git Clone “hook” of sorts.  Git clone post hooks don’t exist and so you are left with using some form of git template combined with a script which updates the remote origin URL with the SSH identity Host identity path you desire.  I suppose you could do a sed operation via  bash script executed from some git init customizations combined with a git clone —template, but remembering all these setup steps steps made me just opt for using the simple git command line manually after I clone.
(Note: for all existing cloned git repos you have, you’ll need to go into each one and update the remote origin URL in the .git/config file to take advantage of any custom SSH host name path you are using).
For now, you can use the following command to update the per repo .git/config remote origin URL path:
git remote -v

git remote set-url origin git@github.com-myuserid-personal:myuserid-personal/project1.git

git remote -v

Troubleshooting:
Note that in the SSH config file examples I provided, there is a line saying #LogLevel DEBUG3.  This is a great way to see live SSH authentication debugging information as you try to authenticate to each remote GIT server.  I left them remarked out just so I could turn them on if I ever needed to troubleshoot a connection.  As you learn how to use these SSH identity approaches to multiple GIT identities, you’ll find this debug command super helpful.

Sources:
https://git-scm.com/book/ch4-1.html
http://ricardianambivalence.com/2013/09/22/github-for-work-and-play-multiple-accounts/
http://nerderati.com/2011/03/17/simplify-your-life-with-an-ssh-config-file/
http://stackoverflow.com/questions/10228065/git-hooks-is-there-a-clone-hook
http://stackoverflow.com/questions/2141492/git-clone-and-post-checkout-hook/2141577#2141577
http://stackoverflow.com/questions/16363460/set-up-a-default-directory-structure-on-git-init
http://monkeypatch.me/blog/mixing-professional-and-personal-git-configurations.html


Chef Vault with Large Teams

Chef-vault is a tool created by Nordstrom and adopted by Chef as the de facto way to handle secrets management using the Chef platform.  Chef-vault builds on the original Chef encrypted data bags concept¹. Rather than a single shared decryption key, chef-vault creates a separate copy of the data bag item for each node that is granted access, using the existing RSA public/private key pair normally used for Chef API authentication. According to Noah Kantrowitz, who sums it up nicely, “This means you no longer have to worry about distributing the decryption keys, but it also moves chef-vault into the gray area between online and offline storage systems.  Once a server is granted access to a secret, it can continue to access it in an online manner. However granting or revoking new servers requires human interaction. This means chef-vault is incompatible with cloud-based auto-scaling or self-healing systems where nodes come/go frequently. It also inherits the same issues with lack of audit logging as all other data-bag driven approaches”.
Summary of Chef-Vault Functionality:
  • Uses encrypted data_bags, but adds layer of mgmt on top
  • Creates separate copy of encrypted data_bag item for each node granted access
    • No single shared decryption key
  • Uses the existing public/private key pair for the Chef API authentication (node public key)
  • Handles distribution of keys for you using Chef
  • Granting/revoking servers access is a separate human task
    • Cloud based systems which build and destroy nodes requires that new nodes be added/removed each time nodes status changes for vault
    • Requires using the chef-vault local knife commands
Problem:
How to have teams of developers manage secrets in a common manner using chef-vault?
Of course you could say that each individual team has designated individual(s) with knife access to Chef servers and they use knife vault to get the secrets up to Chef and call it a day.  However, this is not always as cut and dry as one might hope.  For example, in my company we chose years ago to tightly restrict knife write access to chef servers only from a CI/CD server via a pipeline.  In other words, there are SDLC process limits and security process limits put in place to limit the risks of “wild west knife access”.  Another reason is that members of teams in larger orgs can come and go, and often you need systems where things are well documented and stored in ways that others can pick up and work where another left them.
Due to the need to have a consistent process as well as limit knife access to the pipeline, we needed a mechanism to get secrets into Vault without funneling them all to a central team every time a new secret is added or removed.  Because Chef Vault uses data_bags which are predominantly stored in version control, the seductive answer is to just throw them on your private version control system like GitHub Enterprise.  However, the moment you do that, your secrets no longer become secret.  What  was needed was a way to encrypt secrets in GitHub in a seamless manner such that a CI/CD pipeline server can ingest them for the express purpose of using knife vault to get it securely to Chef.
The design solution was to use a combination of technologies:  git-crypt + GPG + github + Jenkins + Chef vault.  Let’s dig into the design details.
 
Solution Summary:
  • Create a team based “GPG WoT” (web of trust)
  • Limit exposure of the public key to the folks that need it (dev team, Chef admins, CI server admin) in a localized web of trust.
  • Use git-crypt + the GPG key to seamlessly encrypt your desired secrets for storage in the data_bags/chef-vault git repo.
  • Allow a private CI/CD server instance (like Jenkins) to have your team GPG public key so it can grab your encrypted data_bags/chef-vault secrets and decrypt them
  • Have the CI/CD server instance then run a job which takes the unencrypted data_bags/chef-vault secrets and then uses knife vault to secure them again on the Chef server.

MyOrg Web of Trust (WoT) Using GPG

In order to sanely place secrets like passwords, certificates, and keys in a DVCS like GitHub with only authorized people having access, we need to use GPG keys.  The design is such that the authorized “web of trust” user accounts like an Infrastructure CI/CD server ldap user, Infrastructure Chef admins, or specific Security Engineering team members get a copy of a given team’s public GPG key.  An authorized user account will then be able to grab the data and unencrypt it using the GPG keys.  Depending on your Web of Trust preferences, you may desire to only allow access to your GPG public key to a smaller group.
Essentially, the design is such that each team can have a designated email address or name as an identifier which is used with the GPG public key.  This public key is then sent to ONLY those in your Org Web of Trust (WoT) that are authorized to get the public keys.  In this case we are not actually freely giving out team public GPG keys, but selectively choosing who can get them.  I realize that the beauty of the GPG key in the public internet use case is to make the public key widely available.  Here, we are using it selectively within a given org for purposes of limiting who has access to decrypt.  You are taking some steps to protect your public keys within a team for purposes of getting the shared secrets into a data_bag store (usually Git) securely.  The trick here is to have the data_bag items encrypted in the version control system like a private Github repository such as Github enterprise using GPG keys + Git-Crypt.
The process of getting them into the Chef server as an knife vault encrypted data_bag item then can be handled by a secured job on a CI server such as a private Jenkins instance.
The CI/CD server job must do 2 things:
  1. Grab the GPG encrypted data_bag items from version control (it has the GPG public keys for each team).
  2. Decrypt the GPG encrypted data_bag item and then use knife vault to upload and encrypt using normal Chef Vault processes.
This job workflow allows a distributed team to centrally store secrets in a way that a central job can grab it, decrypt it, and then re-encrypt it using native Chef vault processes.  Of course, when people leave the team, it behooves you ideally to generate a new team based GPG key and then re-encrypt your team secrets using the new GPG key and ensure the CI/CD server job gets the new key for its purposes.  This design also assumes you own a private DVCS system like Github enterprise or a private Github account.  I don’t recommend implementing this design for any open public internet projects, on a non-private Github or bitbucket for example, where mistakes in your workflow and WoT could potentially expose your secrets.  This is a weakness of the design in terms of security and management, but a fair trade off from the alternative of non-standard secrets management among large teams before they get it into chef vault.

Pre-requisites:

  • CI Server & Admins:
    • Working Chef11/12 workstation environment (chef, knife, etc)
      • Chef DK
      • This includes you being a valid Chef user
      • Knife is best reserved to very few people.  Let a CI/CD pipeline do knife tasks.
    • Installed chef-vault gem on your workstation (or just use ChefDK)
      • Recommended to use RVM or rbenv to avoid installing gems in your system ruby or just use ChefDK
  • Org Team members & CI Server & Admins:

Workflow and Dev setup

  • Install git crypt on your local machine in order to encrypt your data bags.
  • Generate and share your team GPG pub key with the central Chef Admins for purposes of getting it to the private CI/CD pipeline server(s).
    • Export your pub key using:
      gpg --export -a "User Name" > public.key
    • This will create a file called public.key with the ascii representation of the public key for User Name (use LDAP userid or email is useful in large orgs)
  • Configure git-crypt for your repo
    • Fork a copy of the chef-repo/myChef repo to your personal GitHub repo (this is the repo where you store data_bags.
    • Clone your forked copy of the myChef repo to your local workstation.
    • Navigate to the data_bags directory and then to the chef-vault data_bag directory, then create a directory for your app tied to the role name
cd /chefRepo/data_bags/chef-vault/
mkdir myApproleProd
  • Create a .gitattributes file to tell git-crypt what files to encrypt in a repo directory
cat > .gitattributes << EOL
secretfile filter=git-crypt diff=git-crypt
*.key filter=git-crypt diff=git-crypt
EOL
  • Initialize the repo directory for git-crypt
    git-crypt init
  • Add your GPG key to git-crypt
    git-crypt add-gpg-user USER_ID
Workflow Summary :  Common Chef Repo Data_bags or Chef 12 Organizations
  • After Chef Admin team adds your team public key, follow the steps below to add a new secret to the chef-vault store within the data_bags root directory in Git.
  1. cd /chefRepo/data_bags/chef-vault
  2. git pull origin master
  3. git-crypt unlock
  4. Add your new data bag item ( make sure it ends with .key and id matches the file name, steps listed below)
    • A vault data bag needs to be checked in a specific directory structure standard (see below for details).
  5. git commit -am “adding encrypted chef-vault item"
  6. git push
  7. Submit pull request to accept additions to the repo
  8. Wait for the Chef CI/CD pipeline server job to:
    • Ingest the new items via the CI server job
    • Decrypt using your team GPG key
    • Knife vault to re-encrypt using chef-vault a given Chef server.
Chef-Vault Directory Design Standards
The data_bags directory structures allow data_bags that contain data_bag items.  This can exist in a sane root directory structure such that we can organize things in a Github repo in a way that is visually appealing.  Once the data_bags and their items are ingested by Chef to SOLR, the root directory structure in GitHub is not present.  Instead there are data_bags (keys) and data_bag items (values).
See details on how Chef views the data_bag directory structure.  Per the Chef docs, “A data bag is a container of related data bag items, where each individual data bag item is a JSON file.  knife can load a data bag item by specifying the name of the data bag to which the item belongs and then the filename of the data bag item.”
Essentially, the form is:
data_bag directory
     data_bag_item json file
You can chose the name the json data_bag_item in Chef vault with a .key extension, but the content should have “id” and “key” section.
{
 "id": "my_secret_item_name",
 "key": "value"
}
An example directory structure is below:
The bag_name should be updated based on your application role.
myChefrepo
├── data_bags
│   ├── chef-vault
│   │   └── bag_name_matching_role
│   │      └── fooapp-secrets.key
│   │      └── barapp-secrets.key
  • We (CI/CD job) expects the bag_name to be added under the chef-vault directory
  • We (CI/CD job) also expect the bag_name to match the role name.
    • The vault data_bag would be created and available only for nodes with corresponding with chef role.
      • e.g: If you expect a vault item to be applied to a node with a chef role of myAppenvProd
      • The structure would look like this:
├── data_bags
│   ├── chef-vault
│   │   └── myApproleProd
│   │      └── myApp-secrets.key
  • For the item to be encrypted the data_bag_item should end with file extension “.key” and the contents should be in JSON format.
    • This is a design preference (.key), but the JSON format is a Chef data_bag thing (see here).

Design for sharing encrypted Items in a DVCS for Chef using chef-vault

chef-vault-automation-github


REFERENCE:

Here are some common use cases or knife-vault that the CI/CD server job logic can execute.
 
Chef-vault commands used by the CI Server automation (Examples)
Create Vault Item
knife vault create fooapp fooapp-secrets \ -J data_bags/chef-vault/apps/fooapp/fooapp-secrets.json \ -A "adminuser1,adminuser2" -S "role:fooapp-server"
Show Vault Item
knife vault show fooapp fooapp-secrets -Fjson 
Delete Vault Item
knife vault delete fooapp fooapp-secrets
Delete old node
knife vault update fooapp fooapp-secrets \ -S "role:fooapp-server"
Update list of Admins
First see who has access:
knife search fooapp 'id:fooapp-secrets -a clients
Next change the membership:
knife vault update fooapp fooapp-secrets \ -J data_bags/chef-vault/apps/fooapp/fooapp-secrets.json \ -A "adminuser3,adminuser2" -S "role:fooapp-server"

Update/Rotate/Refresh Keys/secrets
Rotate keys for a single vault item:

knife vault rotate fooapp fooapp-secrets

Rotate all keys for all vault items:

knife vault rotate all keys

References:
  1. https://coderanger.net/chef-secrets/
  2. http://engineering.ooyala.com/blog/keeping-secrets-chef
  3. https://github.com/Nordstrom/chef-vault/blob/master/KNIFE_EXAMPLES.md
  4. http://jtimberman.housepub.org/blog/2013/09/10/managing-secrets-with-chef-vault/