Every application uses secrets to function. These secrets include usernames and passwords, API keys, and other similar private keys. Applications running inside Kubernetes are no exception. Unfortunately, Kubernetes has a reputation for not being able to keep a secret. Is that reputation valid? In this talk, Omer Levi Hevroni explores potential solutions that show Kubernetes can in fact keep a secret.
Who handles secrets?
A centralized management team oversees secrets in some companies as part of running deployments. The alternative model, though, is Super Devs with full responsibility for writing code, deploying it, and monitoring it. On the one hand, a centralized team manages secrets securely. On the other hand, Super Devs yield a simpler and more scalable organization. That said, Super Devs need good tools to support them, to minimize mistakes, and to make systems secure by design.
GitOps
Developers are already familiar with Git. As a result, there are many upsides to relying on Git for the entire workflow, referred to as GitOps. Typically, code and Kubernetes manifest files live in the repository. Then, they are deployed from there into Kubernetes pods. However, what about the application secrets that must be stored securely?
Requirements
The goal was to find a solution that met these requirements:
- Embraces GitOps
- Integrates natively with Kubernetes
- Stores secrets securely by design
For this analysis, security related to the Kubernetes pod is out of scope. For example, SSH access to the pod and secret leakage from the pod are interesting subjects but not part of this analysis.
Option 1: Kubernetes Secrets
Kubernetes Secrets are the official Kubernetes means of storing secrets securely. They're intended for use on SSH keys, OAuth tokens, and passwords. By comparison, Kubernetes Secrets are safer and more flexible than deploying directly in the pod or a docker image.
Here's the issue though. Kubernetes Secrets store usernames and passwords as base-64 encoded strings. As a result, they're obscured from casual browsing, but this is still the same as being in plaintext. To emphasize, text encoding isn't secure. As a result, Kubernetes Secrets as a solution are immediately complicated.
Encrypting secrets
What about encrypting secrets? Sealed secrets and Helm secrets are two options. Sealed secrets result in encrypted secrets in the manifest file that are decrypted in the pod. On the one hand, that seems to do the trick. However, there are still issues. First, for sealed secrets, anyone with access to the cluster can decrypt the secrets. Second, you're coupled to the specific cluster and deployment method for Helm secrets. Lastly, any changes to the encrypted secrets require decryption, and this affects other configurations within the same file.
Evaluation
Compared to the requirements, this solution integrates natively with Kubernetes, but both the GitOps and "secure by design" requirements are only met with serious limitations.
Option 2: HashiCorp's Vault
The next option relies on HashiCorp's Vault system to store secrets securely. Vault lives side by side with the applications. First, developers put app secrets into Vault. Then, the applications contact Vault to receive plaintext secrets as needed. The workflow sounds simple but has an issue: access control.
Fairly significant permissions issues exist on Vault as compared to the requirements. First, some developers need access to manage secrets, while others only need read access. Consequently, you’ll still require a centralized management team. In addition, Vault receives secrets directly, rather than from Git, so GitOps isn't supported. Lastly, central management of secrets in Vault has advantages but one big disadvantage. Specifically, there's no single source of truth for application configuration inclusive of code, Kubernetes manifest, and secrets. As a result, troubleshooting is more difficult.
Evaluation
Compared to the requirements, this solution integrates natively with Kubernetes but doesn't use GitOps. Also the "secure by design" requirement depends on specific usage.
Option 3: Kamus
Both mainstream solutions didn't meet the needs for protecting secrets in Kubernetes. In time, inspiration came from Travis CI. Specifically, Travis supports a command-line function to encrypt parameters. In similar fashion, could Kubernetes support such a function? With this in mind, Kamus was born.
Kamus encrypts secrets for a specific application. Subsequently, only that application receives the plaintext. As a result, even if other pods try to decrypt the secret, they'll fail. First, Kamus encrypts the secret to go into the Git repository using the intended application's service account token. Then, the pod sends the encrypted secret and its Kubernetes service account to Kamus to receive plaintext secrets. In effect, the service account provides the assurance for Kamus to decrypt secrets for the application.
Additional details
Kamus leverages cloud encryption services like Azure Key Vault, thus hardware security modules (HSMs) are supported. Also, Kamus supports cloud resource definitions (CRDs) for Kubernetes Secrets.
The permission model for Kamus is straightforward. First, users can encrypt on a limited basis but cannot decrypt. Next, a pod can encrypt but can only decrypt its own secrets.
Evaluation
Compared to the requirements, this solution meets them all: full GitOps support, native Kubernetes integration, and secure secrets by design.
Wrapping up
Can Kubernetes keep a secret? Based on this talk, yes. To be sure, you need the right tool for your business. To that end, Kamus enables your Super Devs to fly higher. Watch the presentation below.
Written by Daniel Longest
With over a decade in the software field, Daniel Longest has worked in basically every possible role, from tester to project manager, development manager to enterprise architect. He has deep technical experience in .NET and database application development. After several experiences with agile transformations and years spent coaching and mentoring developers, he's passionate about how organizational design, engineering fundamentals, and continuous improvement can be united in modern software development.