Kubernetes Secrets: How to Use Them Securely

Storing sensitive values is a problem as old as software itself. In 2016, Uber experienced a massive data breach that exposed 57 million users’ personal information—all traced back to a hardcoded AWS credential discovered in a GitHub repository.

While we have successfully established that hardcoding secrets such as API keys and passwords is bad practice, correctly storing them is a different story, and the issues from 2016 are still prevalent today (8 years later…). In a 2024 report by Sophos, 77% of attacks saw compromised credentials as an initial access method and 56% as a root cause. 

With organizations going cloud-native and moving their workloads to Kubernetes (k8s), it is only right that they know how to avoid these issues in Kubernetes. In this post, we will discuss Kubernetes secrets—the Kubernetes built-in security resource—and how to use them properly to secure your applications on Kubernetes. 

Kubernetes secrets: What are they, and what are they used for?

Secrets in Kubernetes are a resource used to store sensitive values or credentials. In practice, this removes the need to hardcode your API keys within your deployment manifest or pod definition. Out of the box, Kubernetes provides eight types of secrets, each with a unique function. Here’s a breakdown:Each secret type enforces specific data field requirements, helping prevent configuration errors when storing different credentials. 

1. Opaque: The default type for storing arbitrary data like API keys or application passwords. Example: storing a third-party API key for a payment gateway.
2. kubernetes.io/service-account-token: Stores tokens that pods use to interact with the Kubernetes API. Example: allowing a monitoring pod to query cluster metrics.
3. kubernetes.io/dockercfg: Contains credentials for private container registries using the legacy Docker config format. Example: pulling images from a private Docker Hub repository.
4. kubernetes.io/dockerconfigjson: Similar to dockercfg but uses the newer JSON config format. Example: authenticating with multiple private container registries.
   
5. kubernetes.io/basic-auth: Stores credentials for HTTP basic authentication. Example: protecting an internal dashboard with username/password.
   
6. kubernetes.io/ssh-auth: Holds SSH credentials. Example: allowing pods to clone private Git repositories.
   
7. kubernetes.io/tls: Contains TLS certificates and private keys. Example: setting up HTTPS for your ingress endpoints.
   
8. bootstrap.kubernetes.io/token: Used during node registration to authenticate new nodes joining the cluster. Example: automating node addition in an auto-scaling cluster.

Why You’d Use Kubernetes Secrets 

One excellent example of how secrets can be used is the popular Kubernetes tool, cert-manager. This controller automatically generates TLS certificates and stores them back into the cluster as kubernetes.io/tls type secrets, which your workloads can then use to encrypt traffic. In addition, you can also use Kubernetes secrets to:

• Store database credentials that your applications need to establish connections, eliminating the need to hardcode these values in your application code or environment variables.
• Manage API keys for external services like payment processors, email providers, or monitoring tools your applications depend on.
• Store SSH keys needed by CI/CD pipelines to clone private repositories during build processes.
• Keep registry credentials for pulling container images from private repositories, ensuring your deployments can access proprietary container images.
• Store OAuth tokens and other authentication credentials used by microservices to communicate securely within your cluster.
• Maintain JWT signing keys used by authentication services to generate and validate user tokens.
  

Are there any limitations to using Kubernetes secrets?

Lack of Encryption by Default

While Kubernetes provides a good range of choices for secret management, it still has a few limitations, the largest being the lack of encryption by default. While Kubernetes does support encryption at rest through KMS providers, this requires additional configuration and maintenance.

Kubernetes Secrets are Immutable

Another limitation is that Kubernetes secrets are immutable by default. Immutable means that they cannot be modified after creation. This immutability is intended to promote stability but creates some challenges when you need to rotate credentials or update sensitive values. Each update requires creating a new secret and updating all references to it.

Limited RBAC Configuration

While functional, Kubernetes’s built-in RBAC (Role-Based Access Control) system offers limited granularity for secret access control. This factor can become a challenge in larger organizations where different teams need varying levels of access to different secrets. 

For example, you cannot grant a team read access to specific fields within a secret—they either get access to the entire secret or none at all. This limitation often forces teams to create separate secrets for each access level, increasing management complexity.

How to Use Kubernetes Secrets Securely

Securely using secrets in Kubernetes often requires a combination of techniques and tools. In this section, we will explore a few best practices for using them. 

Enable Encryption at Rest

As we mentioned earlier, Kubernetes secrets are stored as plaintext in etcd by default. While this provides convenience, enabling encryption at rest for production environments is crucial. Configure a KMS provider to encrypt your secrets. While no team wants their cluster breached, this adds an extra layer of security and one more hurdle an attacker must overcome.

A WAF (Web Application Firewall) complements these measures by providing an additional layer of protection specifically designed for web traffic, regardless of how secrets are managed within the cluster.

You can configure encryption at rest using the EncryptionConfiguration resource, which typically looks like this:

apiVersion: apiserver.config.k8s.io/v1

kind: EncryptionConfiguration

resources:

  - resources:

      - secrets

    providers:

      - aescbc:

          keys:

            - name: key1

              secret: <32-byte-key>

The resources.resources field is an array of Kubernetes resource names (resource or resource.group) that should be encrypted, such as Secrets, ConfigMaps, or other resources. The providers field can be used to specify one of the supported providers. 

Implement RBAC Controls

While not the most robust, there are ways to ensure you follow the principle of least privilege with your secrets. Here is a quick example:

Create a cluster role:

kubectl create clusterrole secrets-manager \

    --verb=get,list,create,update \

    --resource=secrets \

    --namespace=production-apps

Create a dedicated service account:

kubectl -n production-apps create serviceaccount app-secrets-manager

Bind the role to the service account with a cluster role binding:

kubectl create clusterrolebinding manage-production-secrets \

    --role=secrets-manager \

    --serviceaccount=production-apps:app-secrets-manager \

    --namespace=production-apps

Restrict Secret Access to Specific Containers

When deploying applications, limit secret access to only the containers that require them. Instead of mounting secrets at the pod level, specify secret mounts or environment variables for individual containers. This step reduces the exposure surface of sensitive data within your pods.

You can achieve this by:

• Mounting secrets as volumes to specific container paths rather than pod-wide shared volumes.
• Setting environment variables from secrets only in containers that need them.
• Using subPath when mounting secret volumes to prevent exposing the entire secret volume to a container.
  

In practice, this looks like:

apiVersion: v1

kind: Pod

metadata:

  name: multicontainer-pod

spec:

  containers:

  - name: app-container

    image: app:latest

    volumeMounts:

    - name: api-creds

      mountPath: "/etc/api/credentials"

      readOnly: true

  - name: sidecar-container

    image: sidecar:latest

    # No access to api-creds secret

  volumes:

  - name: api-creds

    secret:

      secretName: api-credentials

      defaultMode: 0400

Consider Using External Secret Store Providers 

If you’re running Kubernetes in a cloud environment, you likely already have access to a Key Management System (KMS). The External Secrets Operator (ESO) bridges the gap between Kubernetes and these external secret stores.

ESO allows you to integrate your cloud provider’s secret management systems with your Kubernetes clusters. Rather than storing sensitive data directly in Kubernetes, ESO fetches secrets from your external store and injects them as regular Kubernetes secrets into your cluster. Here’s what it looks like:

apiVersion: external-secrets.io/v1beta1

kind: ExternalSecret

metadata:

  name: aws-secret

spec:

  refreshInterval: "15s"

  secretStoreRef:

    name: aws-store

    kind: SecretStore

  target:

    name: secret-to-be-created

  data:

  - secretKey: api-key

    remoteRef:

      key: production/api-credentials

      property: api-key

The manifest defines an ExternalSecret resource that:

• Checks for updates every 15 seconds (refreshInterval).
• References a configured secret store (aws-store).
• Creates a Kubernetes secret named secret-to-be-created.
• Pulls the api-key value from a remote secret path production/api-credentials.

The main advantage of this approach is that ESO handles all the synchronization between your external secret store and Kubernetes, automatically creating and updating the Kubernetes secret when the source changes.

Implement Auditing and Monitoring

In addition to fine-grained access control, it’s crucial to have a big picture of what’s happening within your cluster, which is exactly what Kubernetes audit logs provide. It is particularly useful for continuously monitoring secrets usage. As the Kubernetes docs put it:

1. Auditing allows cluster administrators to answer the following questions:

• What happened?
• When did it happen?
• Who initiated it?
• On what did it happen?
• Where was it observed?
• From where was it initiated?
• To where was it going?

Bringing Outside Secrets In 

Kubernetes secrets provide robust capabilities for managing sensitive data within your cluster, from basic credentials to TLS certificates, with multiple layers of security controls. However, not all your resources and sensitive data live within Kubernetes.

The Apono Connector for Kubernetes helps bridge this gap by securely connecting your Kubernetes cluster with outside resources. By running within your environment, the connector maintains a clear separation between your infrastructure and external services while providing unified access management. 
Book a demo to see Apono in action.