Background
In enterprise environments, sensitive information like API keys, passwords, and certificates must be managed securely. Kubernetes, widely used for container orchestration, stores such data as secrets. However, plain-text Kubernetes secrets are not safe for version control systems or manual handling.
To solve this, Sealed Secrets encrypt sensitive data so it can be safely stored and shared. Deploying and managing these Sealed Secrets efficiently in Amazon AKS requires automation to ensure security, scalability, and ease of use.
Challenge
Enterprises often struggle with several challenges in this area. Security risks arise from storing secrets in plain text, making them vulnerable to unauthorized access or leaks, and manual handling increases the likelihood of errors. Managing secrets across multiple environments such as development, staging, and production is complex and time-consuming, and the process becomes increasingly unmanageable as workloads and environments grow. Additionally, the lack of a built-in mechanism to automate the encryption and deployment of Kubernetes secrets necessitates the development of custom automated solutions to integrate with existing tools and workflows.
Proposed solution
A Jenkins pipeline, combined with a Docker-in-Docker (DIND) container, provides a robust solution to these challenges by automating the secure management of Sealed Secrets in AKS clusters. This pipeline encrypts secrets using the Sealed Secrets Controller, ensuring that only the AKS cluster can decrypt them, thereby eliminating manual secret management. Centralized through Jenkins, the solution ensures consistent secret generation and deployment across environments while archiving the generated secrets and certificates for future use. By encrypting secrets before storage, the solution ensures they are safe from unauthorized access and compliant with security standards, while Kubernetes role-based access controls (RBAC) further restrict access.
https://github.com/regspweek41/jenkins-sealed-secrets.git1. Prepare Workspace
This stage sets up the environment for processing secrets and generating artifacts.
- Create Directories: Create working directories to store temporary files and artifact
Benefits
mkdir -p /tmp/jenkins-k8s-apply mkdir -p ${WORKSPACE}/sealed-secrets-artifacts rm -f /tmp/jenkins-k8s-apply/* || true ls -la /tmp/jenkins-k8s-apply || echo "Directory is empty"
2. Process Base64 Encoded Secrets
This step involves decoding the provided secrets YAML file.
- Write Base64 Data to a File: Save the base64-encoded input as a temporary file
echo ${SECRETS_YAML} > /tmp/jenkins-k8s-apply/secrets.yaml.b64
base64 --decode < /tmp/jenkins-k8s-apply/secrets.yaml.b64 > /tmp/jenkins-k8s-apply/secrets.yaml
ls -l /tmp/jenkins-k8s-apply/secrets.yaml
head -n 5 /tmp/jenkins-k8s-apply/secrets.yaml | grep -v 'data:' || echo "File appears to be empty"
3. Apply Kubernetes Config & Fetch Public Certificate
This stage fetches the Sealed Secrets Controller’s public certificate.
- Check Kubeconfig File: Ensure that the kubeconfig file is present and accessible
ls -l ${KUBECONFIG}
Fetch Public Certificate: Use kubeseal to retrieve the public certificate from the Sealed Secrets Controller
docker run --rm \
-v ${KUBECONFIG}:/tmp/kubeconfig \
-v /tmp/jenkins-k8s-apply/secrets.yaml:/tmp/secrets.yaml \
docker-dind-kube-secret kubeseal \
--controller-name=sealed-secrets \
--controller-namespace=kube-system \
--kubeconfig=/tmp/kubeconfig \
--fetch-cert > /tmp/jenkins-k8s-apply/sealed-secrets-cert.pem
Validate the Certificate: Check if the public certificate was successfully fetched
ls -l /tmp/jenkins-k8s-apply/sealed-secrets-cert.pem
4. Create Sealed Secrets
This stage encrypts the secrets using the public certificate.
- Generate the Sealed Secret YAML File: Use the
kubesealtool to encrypt the secrets
docker run --rm \
-v ${KUBECONFIG}:/tmp/kubeconfig \
-v /tmp/jenkins-k8s-apply/secrets.yaml:/tmp/secrets.yaml \
-v /tmp/jenkins-k8s-apply/sealed-secrets-cert.pem:/tmp/sealed-secrets-cert.pem \
docker-dind-kube-secret sh -c "kubeseal \
--controller-name=sealed-secrets \
--controller-namespace=kube-system \
--format yaml \
--cert /tmp/sealed-secrets-cert.pem \
--namespace=${NAMESPACE} \
< /tmp/secrets.yaml" > ${WORKSPACE}/sealed-secrets-artifacts/sealed-secrets.yaml
5. Create Documentation
Document the details of the process for future reference.
- Generate README File: Record metadata and details about the process
echo "Generated on: $(date)" > ${WORKSPACE}/sealed-secrets-artifacts/README.txt
echo "Namespace: ${NAMESPACE}" >> ${WORKSPACE}/sealed-secrets-artifacts/README.txt
echo "Controller: sealed-secrets" >> ${WORKSPACE}/sealed-secrets-artifacts/README.txt
echo "Controller Namespace: kube-system" >> ${WORKSPACE}/sealed-secrets-artifacts/README.txt
The proposed solution integrates seamlessly with existing CI/CD workflows and AWS services, making it easy to adopt and scalable to handle multiple clusters and namespaces. It enhances security by encrypting secrets, saves time by automating manual processes, and simplifies compliance with regulatory requirements. By centralizing secret management and reducing operational overhead, this solution provides a user-friendly and efficient approach for DevOps teams to securely handle sensitive information in Kubernetes environments.
