How to Manage GitOps Secrets: A Detailed Guide

GitOps is becoming increasingly popular. More and more companies have started using Git as the source of truth for their infrastructure and application configuration. However, with its advantages comes also challenges. For example, if all your configuration is stored in Git, how do you manage secrets? You can't simply commit your passwords and tokens in clear text to the Git repository even if that repository is private and only a few people have access to it. In this post, you'll learn how to manage GitOps secrets securely. Stay tuned.

GitOps vs Secrets

If you've never used GitOps before, here's a short introduction for you. GitOps is a way of managing infrastructure and application configuration purely via Git repositories in a declarative manner. Here's how it works: You store all the configuration in Git, and then you have a GitOps tool installed somewhere that constantly monitors changes to that Git repository and applies infrastructure and application changes once it detects that something changed in the repository. The whole point of GitOps is that you have one, centralized, single point of truth for all your infrastructure and application configuration. GitOps is most commonly used with Kubernetes.

But as mentioned in the beginning of this post, there are some challenges when using GitOps. And the biggest one is secret management. There will be many secrets that your infrastructure will require. Your application configuration is probably full of secrets too. And it ought to go without saying that storing secrets in the Git repository in plain text is a security vulnerability. That's true even if that repository is private. You need a different solution for that, but ideally something that still works in a GitOps manner. This means it would be great not to have a separate process to define secrets. I'll show you how it can be done.

Secrets the GitOps Way

There are two popular ways of solving this problem. They work quite differently, but both achieve the same outcome: the ability to store secrets or their references in a Git repository. Which one you choose will depend on your company's needs. Let's discuss both of them.

SealedSecrets

We already established that you can't store secrets in plain text in a Git repository. But how about storing them in a non-plain-text version? That's precisely what the SealedSecrets tool does. It allows you to encrypt your secrets and only store their encrypted version in your Git repository. Simple as that.

How does SealedSecrets work, you ask? You install a SealedSecrets controller on your Kubernetes cluster and the kubeseal binary on your local machine. SealedSecrets will generate private and public keys for encrypting the secrets. Before committing a secret to a Git repository, you'll use kubeseal binary to encrypt it. Then, in an encrypted form, it's totally safe to store it in a repository, and only the SealedSecrets controller running in your Kubernetes cluster will be able to decrypt it. Quite smart, if you ask me.

How to Use SealedSecrets

First, follow the installation instructions for SealedSecrets here. Once you have it up and running, you can try to seal your first secret with kubeseal. Let's create a simple Kubernetes secret definition YAML file and use kubeseal to seal it.

apiVersion: v1
kind: Secret
metadata:  
  name: example-secret
type: Opaque
data:
  username: my-username
  password: super-secret-password

Once you have the file, you can pipe its content to the kubeseal binary.

cat secret.yaml| kubeseal --controller-name=sealed-secrets-controller --format yaml > sealed-secret.yaml

And if you now take a look at the created sealed-secret.yaml file, you'll see that the actual username and password values are encrypted.

$ cat sealed-secret.yaml
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: example-secret
  namespace: default
spec:
  encryptedData:
    password: AgC7jlVk(...)eb+XOk5/99fKHk=
    username: AgAHbCU7(...)hIgv5D6LDYopF4n
  template:
    data: null
    metadata:
      creationTimestamp: null
      name: example-secret
      namespace: default
    type: Opaque

This file is now safe to be stored in a Git repository since only a SealedSecrets controller that was used to encrypt this file will be able to decrypt it.

But how do you consume that secret in your cluster? It's very straightforward. You can directly apply that sealed file to your cluster, and the SealedSecrets controller running on it will automatically unseal it and create a standard Kubernetes secret resource from it. Let's take a look.

$ kubectl apply -f sealed-secret.yaml
sealedsecret.bitnami.com/example-secret created

$ kubectl get secret
NAME                                 TYPE                 DATA   AGE
example-secret                       Opaque               2      7s

From now on, you can use example-secret just like any standard Kubernetes secret.

ExternalSecrets

Another way to store secrets for your GitOps needs is using ExternalSecrets. It works differently than SealedSecrets but also solves the problem of storing plain text secrets in a Git repository. ExternalSecrets does this by removing the need to store the actual secret in your repository. Instead, your secret can be safely stored in a secret vault, and you only need to store a reference to a secret in your repository.

So instead of having, for example, the actual username and password in a file in your Git, you'll instead have a file that says something like "this username is password is stored in that secret vault, under this key." And then it's the external secret operator's job to go and fetch the actual value for you when you need it. Let's try that.

Using ESO

The external secrets operator can be installed just like any other tool using Helm. You can follow the installation and initial configuration steps here. Once you have ExternalSecrets up and running, using it is quite simple. You first need to add your secrets to the secret vault that you want to use and then create an ExternalSecrets reference file. This file will be a replacement for your typical Kubernetes secret definition file.

As explained before, working with ESO means referencing the actual secrets from an external secret vault. So, you create an external secret resource, and the external secret operator will fetch the actual secret from an external vault in the background and create an actual Kubernetes secret for you. Here's an example:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-externalsecret
spec:
  refreshInterval: 3h
  secretStoreRef:
    name: azure-keyvault
    kind: SecretStore
  target:
    name: database-secret
    creationPolicy: Owner
  data:
  - secretKey: database-secret-dev
    remoteRef:
      key: database-secret-dev

That is an ExternalSecrets definition file that tells ESO to fetch the value of the database-secret-dev key from the Azure Key Vault and create a Kubernetes secret called database-secret from it. As you can see, we don't have actual secret values in this file, so storing it in a Git repository is perfectly fine.

It's the same when it comes to consuming secrets. You simply apply that ExternalSecrets definition file to your cluster, and the ESO operator will automatically fetch the secret from the defined secret vault and create an actual Kubernetes secret from it.

$ kubectl apply -f external-secret.yaml
externalsecret.external-secrets.io/database-externalsecret created

$ kubectl get secret
NAME                                 TYPE                 DATA   AGE
example-secret                       Opaque               2      12m
database-secret                      Opaque               2      4s

Summary

As you can see, the GitOps secrets problem can be solved. It's not even that difficult. However, it does include some extra steps and tools. But once the initial setup is done, it doesn't take much more daily effort to manage secrets securely in your GitOps practices.

If you want to learn more about secrets or GitOps, you can find more content on our blog.