Hello, Devs 👋 !.

We continue the Kubernetes and Spring Ecosystem journey. In this story, we’ll explore Kubernetes Secrets.

· Overview
∘ What are Kubernetes Secrets?
∘ Types of Secret
· Working with Secrets
∘ Spring Data MongoDB
∘ Creating Secrets Using a Manifest File
∘ How to Use Kubernetes Secrets
∘ Deploy the container image to Kubernetes
· What are the limitations of Kubernetes Secrets?
· Conclusion
· References

This series of stories shows how to use Kubernetes in the Spring ecosystem. We work with a Spring boot API and Minikube to have a lightweight and fast development environment similar to production.

• Lab1 (Spring Boot/K8S): Deploy Spring Boot application on Kubernetes
• Lab2 (Spring Boot/K8S): Kubernetes health probes with Spring Boot
• Lab3 (Spring Boot/K8S): Mastering ConfigMaps in Kubernetes
• 👉 Lab4 (Spring Boot/K8S): Using Kubernetes Secrets in Spring Boot

In the previous lab (lab 3), we saw how to configure the spring boot application on Kubernetes to use ConfigMaps. In this story, we’ll learn how to store sensitive configuration data that must be handled securely (e.g., database credentials, API keys, etc.) in the API.

Overview

What are Kubernetes Secrets?

A Kubernetes Secret is an object that contains a small amount of sensitive data such as login usernames and passwords, tokens, keys, etc. Using a Secret means we don’t need to include confidential data in the application code. It is to reduce the risk of exposing sensitive data while deploying applications on Kubernetes.

Secrets are similar to ConfigMaps but are specifically intended to hold confidential data. They’re also maps that hold key-value pairs.

Types of Secret

Kubernetes provides several built-in types for some common usage scenarios. These types vary in terms of the validations performed and the constraints Kubernetes imposes on them.

• Opaque Secrets: Opaque Secrets are used to store arbitrary user-defined data. Opaque is the default Secret type, meaning that when creating a Secret and we don’t specify any type, the secret will be considered Opaque.
• ServiceAccount token Secrets: A kubernetes.io/service-account-token type of Secret is used to store a token credential that identifies a ServiceAccount. This is a legacy mechanism that provides long-lived ServiceAccount credentials to Pods. It is important to note that When using this Secret type, we need to ensure that the kubernetes.io/service-account.name annotation is set to an existing ServiceAccount name.

apiVersion: v1
kind: Secret
metadata:
  name: secret-sa-sample
  annotations:
    kubernetes.io/service-account.name: "sa-name"
type: kubernetes.io/service-account-token
data:
  extra: YmFyCg==

• Docker config Secrets: We use a Docker config secret to store the credentials for accessing a container image registry. You use Docker config secret with one of the following type values: kubernetes.io/dockercfg (store a serialized ~/.dockercfg which is the legacy format for configuring the Docker command line.) or kubernetes.io/dockerconfigjson(store a serialized JSON that follows the same format rules as the ~/.docker/config.json file, which is a new format for ~/.dockercfg)

apiVersion: v1
kind: Secret
metadata:
  name: secret-dockercfg
type: kubernetes.io/dockercfg
data:
  .dockercfg: |
    eyJhdXRocyI6eyJodHRwczovL2V4YW1wbGUvdjEvIjp7ImF1dGgiOiJvcGVuc2VzYW1lIn19fQo=    

• Basic authentication Secret: The kubernetes.io/basic-auth type is provided for storing credentials needed for basic authentication. When using a basic authentication Secret, the data field must contain at least one of the following keys: username (the user name for authentication) and password(the password or token for authentication)

apiVersion: v1
kind: Secret
metadata:
  name: secret-basic-auth
type: kubernetes.io/basic-auth
stringData:
  username: admin # required field for kubernetes.io/basic-auth
  password: t0p-Secret # required field for kubernetes.io/basic-auth

• SSH authentication secrets: The builtin type kubernetes.io/ssh-auth is provided for storing data used in SSH authentication. When using an SSH authentication, you must specify a ssh-privatekey key-value pair in the data (or stringData) field as the SSH credential to use.

apiVersion: v1
kind: Secret
metadata:
  name: secret-ssh-auth
type: kubernetes.io/ssh-auth
data:
  # the data is abbreviated in this example
  ssh-privatekey: |
    UG91cmluZzYlRW1vdGljb24lU2N1YmE=    

• TLS secrets: The kubernetes.io/tls Secret type is for storing a certificate and its associated key that are typically used for TLS. When using a TLS secret, we must provide the tls.key and the tls.crtin the configuration’s data (or stringData) field.

apiVersion: v1
kind: Secret
metadata:
  name: secret-tls
type: kubernetes.io/tls
data:
  # values are base64 encoded, which obscures them but does NOT provide
  # any useful level of confidentiality
  tls.crt: |
    LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNVakNDQWJzQ0FnMytNQTBHQ1NxR1NJYjNE
    UUVCQlFVQU1JR2JNUXN3Q1FZRFZRUUdFd0pLVURFT01Bd0cKQTFVRUNCTUZWRzlyZVc4eEVEQU9C
    Z05WQkFjVEIwTm9kVzh0YTNVeEVUQVBCZ05WQkFvVENFWnlZVzVyTkVSRQpNUmd3RmdZRFZRUUxF
    dzlYWldKRFpYSjBJRk4xY0hCdmNuUXhHREFXQmdOVkJBTVREMFp5WVc1ck5FUkVJRmRsCllpQkRR
    VEVqTUNFR0NTcUdTSWIzRFFFSkFSWVVjM1Z3Y0c5eWRFQm1jbUZ1YXpSa1pDNWpiMjB3SGhjTk1U
    TXcKTVRFeE1EUTFNVE01V2hjTk1UZ3dNVEV3TURRMU1UTTVXakJMTVFzd0NRWURWUVFHREFKS1VE
    RVBNQTBHQTFVRQpDQXdHWEZSdmEzbHZNUkV3RHdZRFZRUUtEQWhHY21GdWF6UkVSREVZTUJZR0Ex
    VUVBd3dQZDNkM0xtVjRZVzF3CmJHVXVZMjl0TUlHYU1BMEdDU3FHU0liM0RRRUJBUVVBQTRHSUFE
    Q0JoQUo5WThFaUhmeHhNL25PbjJTbkkxWHgKRHdPdEJEVDFKRjBReTliMVlKanV2YjdjaTEwZjVN
    Vm1UQllqMUZTVWZNOU1vejJDVVFZdW4yRFljV29IcFA4ZQpqSG1BUFVrNVd5cDJRN1ArMjh1bklI
    QkphVGZlQ09PekZSUFY2MEdTWWUzNmFScG04L3dVVm16eGFLOGtCOWVaCmhPN3F1TjdtSWQxL2pW
    cTNKODhDQXdFQUFUQU5CZ2txaGtpRzl3MEJBUVVGQUFPQmdRQU1meTQzeE15OHh3QTUKVjF2T2NS
    OEtyNWNaSXdtbFhCUU8xeFEzazlxSGtyNFlUY1JxTVQ5WjVKTm1rWHYxK2VSaGcwTi9WMW5NUTRZ
    RgpnWXcxbnlESnBnOTduZUV4VzQyeXVlMFlHSDYyV1hYUUhyOVNVREgrRlowVnQvRGZsdklVTWRj
    UUFEZjM4aU9zCjlQbG1kb3YrcE0vNCs5a1h5aDhSUEkzZXZ6OS9NQT09Ci0tLS0tRU5EIENFUlRJ
    RklDQVRFLS0tLS0K    
  # In this example, the key data is not a real PEM-encoded private key
  tls.key: |
    RXhhbXBsZSBkYXRhIGZvciB0aGUgVExTIGNydCBmaWVsZA==    

• Bootstrap token Secrets: The bootstrap.kubernetes.io/token Secret type is for tokens used during the node bootstrap process. We typically create a bootstrap token Secret in the kube-system namespace and named in the form bootstrap-token-<token-id> where <token-id> is a 6 character string of the token ID.

apiVersion: v1
kind: Secret
metadata:
  name: bootstrap-token-5emitj
  namespace: kube-system
type: bootstrap.kubernetes.io/token
data:
  auth-extra-groups: c3lzdGVtOmJvb3RzdHJhcHBlcnM6a3ViZWFkbTpkZWZhdWx0LW5vZGUtdG9rZW4=
  expiration: MjAyMC0wOS0xM1QwNDozOToxMFo=
  token-id: NWVtaXRq
  token-secret: a3E0Z2lodnN6emduMXAwcg==
  usage-bootstrap-authentication: dHJ1ZQ==
  usage-bootstrap-signing: dHJ1ZQ==

Working with Secrets

There are several options to create a Secret:

• Using kubectl
• Using a configuration file
• Using the Kustomize tool

In this story, we’ll use the configuration file to define secrets in our existing Spring Boot API. The purpose consists of adding a MongoDB Atlas connection string in the API.

Let’s start by cloning the main Spring Boot project.

$ git clone https://github.com/anicetkeric/spring-boot-k8s.git -b main

Spring Data MongoDB

Add the Spring Data MongoDB dependency to the pom.xml file.

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-mongodb</artifactId>
</dependency>

Then, in the application.yaml file, we need to add the connection URI:

  spring:
  application:
    name: spring-boot-k8s
  data:
    mongodb:
      uri: mongodb+srv://${MONGODB_LOGIN}:${MONGODB_PWD}@${MONGODB_URL}/${MONGODB_DATABASE}?${MONGODB_OPTS}

We keep sensitive data like the username and password to the MongoDB database inside Kubernetes Secret and the other data in ConfigMaps.

Creating Secrets Using a Manifest File

We must encode the secret data using base64. To convert the username and password to base64, run the following command:

$ echo -n 'dbusername' | base64
ZGJ1c2VybmFtZQ==

$ echo -n 'dbpassword' | base64
ZGJwYXNzd29yZA==

After encoding the username and password we need to create a Kubernetes secrets manifest which will hold the encoded values of the MongoDB username and password.

apiVersion: v1
kind: Secret
metadata:
  name: mongodb-secret
type: Opaque
data:
  username: ZGJ1c2VybmFtZQ==
  password: ZGJwYXNzd29yZA==

Here is the content of ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: mongodb-config
data:
  # property-like keys; each key maps to a simple value
  host: cluster0.xpkhnmq.mongodb.net
  database: library_db
  option: retryWrites=true&w=majority

How to Use Kubernetes Secrets

The following are the three main ways a Pod can use a Secret:

• As container environment variables
• As files in a volume mounted on one or more of its containers.
• By the kubelet when pulling images for the Pod — imagePullSecrets.

The Kubernetes control plane also uses Secrets; for example, bootstrap token Secrets are a mechanism to help automate node registration.

The complete deployment configuration file uses both themongodb-secretSecret and the mongodb-config ConfigMaps.

apiVersion: v1
kind: ConfigMap
metadata:
  name: mongodb-config
data:
  # property-like keys; each key maps to a simple value
  host: cluster0.xpkhnmq.mongodb.net
  database: library_db
  option: retryWrites=true&w=majority
---
apiVersion: v1
kind: Secret
metadata:
  name: mongodb-secret
type: Opaque
data:
  # replace the secret values with yours
  username: ZGJ1c2VybmFtZQ==
  password: ZGJwYXNzd29yZA==
---
apiVersion: v1
kind: Service
metadata:
  name: book-api-service
spec:
  selector:
    app: backend
  ports:
    - protocol: TCP
      port: 8081
      targetPort: 8080
  type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend-book-api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: backend
      environment: dev
  template:
    metadata:
      labels:
        app: backend
        environment: dev
    spec:
      containers:
        - name: book-api
          image: spring-boot-k8s:latest
          ports:
            - containerPort: 8080
          imagePullPolicy: Never
          livenessProbe:
            httpGet:
              path: /actuator/health/liveness
              port: 8080
            initialDelaySeconds: 15
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 1
          readinessProbe:
            httpGet:
              path: /actuator/health/readiness
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 1
          env: # array of environment variable definitions
            - name: MONGODB_URL
              valueFrom: # select individual keys in a ConfigMap
                configMapKeyRef:
                  name: mongodb-config
                  key: host
            - name: MONGODB_DATABASE
              valueFrom:
                configMapKeyRef:
                  name: mongodb-config
                  key: database
            - name: MONGODB_OPTS
              valueFrom:
                configMapKeyRef:
                  name: mongodb-config
                  key: option   
            - name: MONGODB_LOGIN
              valueFrom:
                secretKeyRef:
                  name: mongodb-secret
                  key: username   
            - name: MONGODB_PWD
              valueFrom:
                secretKeyRef:
                  name: mongodb-secret
                  key: password

The environment variable that consumes the Secret key should populate the Secret’s name and key in env[].valueFrom.secretKeyRef.

Deploy the container image to Kubernetes

Now we are ready to deploy this app to Kubernetes.

$ kubectl apply -f k8s.yaml

Check that the application is running:

The secrets are created successfully.

As we see, everything works fine. Let’s try to add the books to the database.

We have the data in the database.

What are the limitations of Kubernetes Secrets?

While Kubernetes Secrets provide a convenient way to manage sensitive data, they have some limitations. Here is a (non-exhaustive) list of secrets limitations.

• Limited encryption: by default, secrets are stored unencrypted in the API server’s underlying data store (etcd). Kubernetes does support encryption, but the encrypting key needs separate management.
• Size limit: Individual Secrets are limited to 1MiB in size. This is to discourage the creation of very large Secrets that could exhaust the API server and kubelet memory. We can use a resource quota to limit the number of Secrets (or other resources) in a namespace.
• Limit Base64 encoding: Storing such manifest files in a Git repository is highly insecure as it is easy to decode the Base64-encoded data. Anyone can accidentally commit the manifest files into the Git repositories, thus exposing sensitive information, such as credentials, tokens, or ssh keys.
• Secrets as plain text: When a pod needs to access the secrets, it is provided by Kubernetes as environment variables or mounted as files containing plain-text secrets. These secrets become accessible to everyone with access to the pod. Further, there is an increased risk of the value getting exposed in places like debug logs.

For more guidelines to manage and improve the security of the Secrets, refer to Good practices for Kubernetes Secrets.

Conclusion

Well done !!. In this story, we have learned how to store sensitive configuration data that must be handled securely (e.g., database credentials, API keys, etc.) in the API.

The complete source code of this series is available on GitHub.

You can reach out to me and follow me on Medium, Twitter, GitHub, Linkedln

Support me through GitHub Sponsors.

Thanks for reading!

References

• https://kubernetes.io/docs/concepts/configuration/secret/
• https://spacelift.io/blog/kubernetes-secrets
• https://kubernetes.io/docs/concepts/security/secrets-good-practices/

👉 Link to Medium blog