Distributing application credentials in a secure & encrypted manner is critical to maintaining internal security best practices. This proposal aims to describe a method for distributing these credentials in a way that ensures only appropriate applications gain the access they require while affording developers that ability to accomplish their work in an efficient and reproducible manner.

Infrastructure Requirements

In order to separate sensitive data from development data, an infrastructure that mocks real customer data in a development environment is critical. The same methods for distributing credentials can be used for the development environment or static, common credentials can be utilized for simplicity.

This proposal is based on using the Chef configuration management software for managing the servers & distributing credentials. Developers use pre-configured Vagrant instances that mirror production configuration through the use of the same Chef cookbooks, roles, data bags, etc. but a unique ‘vagrant’ environment within Chef. Staging hosts are provisioned to reside within the ‘staging’ environment & production hosts are provisioned with the ‘production’ environment.

Credential Deployment

Credential management is handled using encrypted data bags with another Chef tool called Chef-Vault. Chef-Vault allows for generating encrypted data bags that may only be decrypted on hosts specified in a given Chef search and by the appropriately defined user. These data bags are encrypted with the Chef client key of the destination host so that they can only be decrypted by the same key. Additionally when the data bags are created, they are given specific Chef client names that are allowed modification privileges. Only the registered clients listed as ‘admins’ can make changes to these data bags (remote hosts that are using these data bags for credentials aren’t given admin access to them, they may only read the credentials from the data bags).

Destination hosts are defined through the use of a Chef search rather than listed individually. Therefore it is possible to dynamically provide credentials to multiple hosts of the same classification. Adding an environment to the search limits the scope of the credentials to just the hosts that match the classification criteria within the given environment (ie, only hosts with the ‘web’ role applied in the ‘production’ environment).

Credential Data Bag Structure

A basic data bag is created using the ‘knife vault’ command as follows:

 $ knife vault create creds fooapi \ '{"fooapiuser":"foosecret123"}' \
--search 'role:web' --admins admin \ 
--key .chef/admin-user.pem --mode client

The data bag is generated on the server & each encrypted Chef-Vault data bag contains two items - a data bag containing the encrypted data & another data bag containing the public keys used to do the encryption.

$ knife data bag show creds

Each client that matches the search used in the creation of the data bag & the admin clients public keys are written into the data bag’s ‘*_keys’ file along with the search query itself:

 admins:                admin
clients:               foo-frontend.local.vm
foo-frontend.local.vm: rlU09lCm/RFlVCGuX5vQK3NIMAXQXfb+k8YB29mYBx8OWlT26eY6jkaQCLvu

 id:                    fooapi_keys

 search_query:          role:web

The actual data is stored encrypted in the item without the ‘_keys’ suffix:

$ knife data bag show creds fooapi
  cipher:         aes-256-cbc
  encrypted_data: Sku9wuTTdkFAH148fLeMDO1buCpBsEK2yAxdC419vdY=

  iv:             EwB/ywakoGdaO9If01FbaQ==

   version:        1
id:         fooapi

Using the Encrypted Data

A host can utilize the encrypted data to populate configuration files with appropriate credentials by installing the ‘chef-vault’ gem on the destination host & using it within a Chef recipe to load the data into a hash. The following file was created using an encrypted password stored in the above ‘creds/fooapi’ data bag (note the encrypted data decrypted and written as the password in bold):

 # cat /tmp/mycreds.txt
USER: 'fooapi' PASS: foosecret123

The above file was created using following recipe applied via the run_list in the ‘web’ role which, in turn, was applied to the ‘foo-frontend’ host:

chef_gem "chef-vault"
require "chef-vault"

vault = ChefVault::Item.load("creds", "fooapi")

file "/tmp/mycreds.txt" do
  owner "root"
  group "root"
  mode "0600"
  action :create
  content "USER: 'fooapi' PASS: #{vault['fooapiuser']}\n"

Unique Credentials Per Environment

Because data bags are not restricted to environments it is impossible to limit the use of data bags to specific environments explicitly. However, there are ways that this can be made to work effectively. Since each encrypted data bag was generated with a specific search, the search itself can limit the accessibility of the encrypted data to a given environment by providing the environment as part of the search query:

 $ knife vault create creds fooapi \ 
'{"fooapiuser":"foosecret1234"}' \
--search 'role:web AND chef_environment:production' \
--admins admin --key .chef/admin-user.pem --mode client

This will only allow hosts within the ‘production’ environment to be able to decrypt the data stored in this data bag.

The problem with this approach as it stands is that data bag names must be unique making it impossible to have another ‘creds/fooapi’ data bag for the ‘staging’ environment as well. A work-around for this is to name the data bag according to the environment it will be available to as follows:

 $ knife vault create creds staging-fooapi \ 
'{"fooapiuser":"foosecret1234"}' \
--search 'role:web AND chef_environment:staging' \
--admins admin --key .chef/admin-user.pem --mode client

A recipe can then load the appropriate data bag using the ‘chef_environment’ node attribute to define which data bag to load:

vault = ChefVault::Item.load("creds", "#{node.chef_environment}-fooapi")

This will load the contents of the ‘production-fooapi’ data bag if the host is part of the production environment.

Updating Encrypted Data

Any service credential should be updated on a regular basis. This can also be done via Chef-Vault simply by overwriting the existing encrypted data with new data. Only changes are required when making updates:

 $ knife vault update creds production-fooapi \
'{"fooapiuser":"newsecret6789"}' \
--key .chef/admin-user.pem --mode client

Adding or removing admins is also fairly easy:

 $ knife vault update creds production-fooapi \
--admins admin,jdoe --key .chef/admin-user.pem --mode client

 $ knife vault remove creds production-fooapi \
--admins jdoe --key .chef/admin-user.pem --mode client

Hosts can be added to the list of those allowed to decrypt data simply by re-running ‘knife vault’ with the ‘update’ sub-command. This will re-run the search and add any hosts that didn’t already exist to the keys list:

 $ knife vault update creds production-fooapi \
--key .chef/admin-user.pem --mode client

Removing a host requires using the ‘update’ sub-command but including a modified search that excludes the host:

 $ knife vault update creds production-fooapi \
--search 'role:web AND chef_environment:production NOT name:badhost' \
 --key .chef/admin-user.pem --mode client

Development Environments

Development is best done in sandboxed environments with data that only contains a representation of customer data, not actual customer data. These environments don’t have the strict security requirements a production environment must have. For development, static, common credentials can be used. For instance, in a Vagrant setup it may be sufficient to simply use ‘vagrant’ for both the username & password (most Vagrant base boxes are configured this way for the default user & root).

There are two common ways to satisfy the requirements of a development environment:

Dedicated Development Credential Data Bags

Probably the best approach is simply to distribute encrypted data bags within a new Chef environment (eg, ‘vagrant’) that contain the credentials used to access the services within the same environment. This approach simplifies things in that it maintains the same process & infrastructure required for both staging & production without any mechanical or logical changes. The downside being that these credentials must also be maintained like production credentials with the overhead of the encryption process.

Logic to Use Plaintext Attributes

Alternatively, logic can be coded into the recipes that use a conditional to determine if the recipe is being run in a development environment and, if so, use plaintext attributes containing the credentials instead of the encrypted data bags. An example of how this would look is as follows:

chef_gem "chef-vault"
require "chef-vault"

 if node.chef_environment != 'vagrant'
  vault = ChefVault::Item.load("creds", "#{node.chef_environment}-fooapi")
  password = vault['fooapiuser']
  password = node['myservice']['password']

 file "/tmp/mycreds.txt" do
  owner "root"
  group "root"
  mode "0600"
  action :create
  content "USER: 'fooapi' PASS: #{password}\n"

The obvious downside to this approach is the additional code required to handle the different ways of accessing the password information.

Notes & Considerations

There are a few issues that have cropped as I was working through this solution that I’ll address here. With additional time it shouldn’t be hard to overcome these issues or improve upon the efficiency of the steps laid out here.

Admins Should Be Users

There seems to be some confusion about the way admins work with Chef-Vault. According to the limited documentation an admin can be either a client or user instance in Chef. However, I had consistent issues using a client as the admin for these demonstrations. I removed the ‘admin’ client & added it back in as a user then used the ‘--key’ option to specify the ‘admin’ user’s private key file. This solved a recurring error when trying to make modifications or view the encrypted data bags (“ERROR: OpenSSL::PKey::RSAError: padding check failed”). This issue is discussed in the GitHub issues for the project: https://github.com/Nordstrom/chef-vault/issues/43

Race Conditions in Provisioning

Using searches to limit which hosts can decrypt data is a great way to dynamically limit access to the encrypted information, but it also introduces a high likelihood of race conditions during the initial provisioning process. For instance, the ‘web’ role I created that runs the ‘mycreds’ cookbook with recipe that tries to write the ‘mycreds.txt’ file using the encrypted data fails because the node isn’t yet registered as an authorized decrypting node (the public key isn’t included in the list of keys in the data bag). Therefore, the run fails and the node doesn’t have the required ‘web’ role in its run_list required for adding it to the allowed hosts list in the data bag keys list. We’re stuck with a catch-22.

One solution would be to populate the password with a default bogus value, but this goes against the idea of idempotent Chef runs. This would also lead to a service starting with the wrong credentials causing connection errors and other problems that would be troublesome.

Another solution would be to pass on the recipe or cookbook attempting to use the encrypted data until it is available. This is safer in that it won’t attempt to complete any of the other tasks in the cookbook that may lead to those same troublesome issues noted above. The system would merely have to be converged a second time after the encrypted data bag was updated to allow the host to decrypt the data and complete the convergence successfully.

Next Post Previous Post