Don't underestimate the Kubernetes Secret. It's far more than just a base64-encoded YAML object. Secrets sit at the critical intersection of security, application delivery, and cluster ops. Getting them wrong is a major source of incidents.
Here’s a deep dive into what makes them both powerful and fragile:
1. Lifecycle & Storage
Secrets are created via the API server and stored in etcd. By default, they are only base64-encoded (which is not encryption). When mounted into a Pod, they reside in tmpfs (volatile memory) and are ephemeral, living only as long as the Pod itself.
2. Access Control is Critical
A single overly permissive RBAC policy can turn kubectl get secrets into a critical breach. Principle of Least Privilege isn't a suggestion here; it's a requirement. Always scope RoleBindings and ClusterRoleBindings to the specific secrets a service account needs.
3. Encryption is Non-Negotiable
In Transit: TLS protects communication with the API server.
At Rest: The default base64 in
etcdis not secure. You must enable an EncryptionConfiguration using a strong provider (e.g., Cloud KMS, HashiCorp Vault, etc.) for production environments.
4. Mounting Mechanics & Leak Risks
The Secret's key becomes the filename and the value becomes the file content. This elegant system is also a common source of leaks:
Environment variables can be accidentally dumped in logs.
Debug shells (
kubectl exec) might expose mounted secret files.Application error reports can inadvertently include secret values.
5. The Rotation Problem
Kubernetes does not provide native secret rotation. Updating a secret requires recreating it and then restarting any Pods that consume it. To solve this, you need external tooling:
External Secrets Operator (ESO): Syncs secrets from external stores (AWS Secrets Manager, Azure Key Vault, etc.).