System operators managing security often (maybe always) worry about secrets used to access their networks and resources that will get out or be setup incorrectly, exposing their systems.
If this describes someone you know, you can ease their worries with system hardening. Akash Mahajan (@makash) told his personal journey with system hardening in his talk, The Secrets in Our Clouds. Perhaps his experiences could help you implement or improve system hardening for your own system.
Akash is with Appsecco, which is a relatively small company with typical production and staging requirements:
They run multiple web applications with integrated authentication
They run multiple websites with authentication and authorization
In most cases, they terminate SSL/TLS at NGINX and reverse proxy to internal apps
They generate TLS certificates using Let's Encrypt
Whenever they add authentication as a middle-ware they rely on a SSO provider
Their apps need access to certain secrets, such as passwords and authentication tokens
Akash said he started his system hardening journey with, "what I know best: installing and configuring software." He chose Vault and Goldfish because of its well-written installation instructions, and most of the complexity is hidden. Additionally, it seemed easy enough to expand.
Once your software is installed, you have to answer where to store the secrets and what secrets do we need to worry about. Examples of secrets include:
Certificates
SSH keys
API tokens
Passwords (database, system, user, etc.)
Anything else you need to store safely, such as audit logs that may contain sensitive details and TOTP generation keys
After this was setup for Akash, he still had nagging doubts and gnawing fears in the back of his mind:
Is it the best way to manage secrets?
Should I use something cloud specific?
How resilient is this?
How will I automate to make it reproducible?
What if I make a mistake in configuration and it leaks data out
What if my backups fail?
What did Akash do? He switched hats from operations to security and didn't panic. He took a step back and thought of, "abstractions which are simple enough for me to understand." Some concepts that helped Akash include plaintext keys, secure storage, secure access, and the ability to choose which users see which secret.
Some attributes that helped Akash:
If a digital secret is stolen, you still have your copy
The best way to store something important to you is encrypted, not in plaintext
If we encrypt something, there exists a key in plaintext
This plaintext key needs to be protected
This plaintext key should have secure storage, and we should prove that we are authorized to access it when we want it
Our plaintext key can be stored inside the Vault behind keys with multiple people.
A cloud is someone else's computer. You can't store it in plaintext, but you can be rest assured AWS, Azure, etc. have done their security homework.
If you plan to store secrets in your cloud, Akash offered a few things to think about:
Storage security - Can you expect better security than the cloud provider?
Access security - Can your internal processes do better security than the cloud provider?
Automation possibilities - If you do go ahead, what are the parts that will not be automated?
Cooking required - If you plan to have a multi-cloud strategy, do you need to write your applications, automation scripts to have wrapper functions to ensure that you can plug-in/plug-out cloud provider specific code?
Compliance - Do you have a specific compliance requirement which requires you to think differently about all this?
Akash wrapped up his talk with a recommendation to start simple, with two approaches for managing secrets: one for humans and another for systems.
For humans:
Setup shared Keepass databases synced
Setup a Vault server behind Nginx and create entries for SSH keys and database passwords
Use a provider which can match company directory to make it easy to revoke access to an employee if required
For systems:
Use the same cloud provider where your servers are based
If possible, deploy a serverless app that exposes secrets both from the Cloud KMS and the Vault server.
If you are craving more on DevOps in Modern Infrastructure? Binge watch any of the 20 DevOps in Modern Infrastructure sessions, free of charge, from All Day DevOps.