Secrets Management: Protecting the Keys to Your Kingdom

Every system has secrets: API keys, database credentials, TLS certificates, encryption keys, OAuth tokens. How you manage them determines whether a single compromise cascades into a full breach. Secrets management is not a tool choice -- it is an architectural discipline.

Secret Types Taxonomy

Secrets
├── Credentials
│   ├── Database passwords
│   ├── Service account keys
│   ├── SSH keys
│   └── API keys / tokens
├── Certificates
│   ├── TLS/SSL certificates
│   ├── mTLS client certificates
│   ├── Code signing certificates
│   └── CA private keys
├── Encryption Keys
│   ├── Data encryption keys (DEK)
│   ├── Key encryption keys (KEK)
│   ├── KMS master keys
│   └── Application-level encryption keys
├── Tokens
│   ├── OAuth access/refresh tokens
│   ├── JWT signing keys
│   ├── Webhook secrets
│   └── Session secrets
└── Configuration Secrets
    ├── Connection strings
    ├── SMTP credentials
    ├── Third-party API secrets
    └── License keys

Tool Comparison

Capability	HashiCorp Vault	AWS Secrets Manager	GCP Secret Manager	SOPS (Mozilla)	Sealed Secrets
Type	Full platform	Managed service	Managed service	File encryption	Kubernetes-native
Dynamic secrets	Yes (DB, AWS, PKI)	No	No	No	No
Auto-rotation	Yes (built-in)	Yes (Lambda-based)	Yes (Cloud Functions)	N/A	N/A
Access control	ACL policies + Sentinel	IAM policies	IAM policies	GPG/KMS keys	Cluster-scoped
Audit logging	Built-in	CloudTrail	Cloud Audit Logs	Git history	K8s audit
Multi-cloud	Yes	AWS only	GCP only	Yes (any KMS)	Kubernetes only
GitOps compatible	External Secrets Operator	External Secrets Operator	External Secrets Operator	Native (encrypted in repo)	Native
Complexity	High (operate cluster)	Low	Low	Low	Low
Cost	Self-hosted or HCP	$0.40/secret/month	$0.06/secret/version	Free	Free
Best for	Multi-cloud, dynamic secrets	AWS-native workloads	GCP-native workloads	GitOps, small teams	K8s-only environments

Architecture: Secrets Delivery Patterns

┌─────────────────────────────────────────────────────┐
│              Secrets Management Platform             │
│        (Vault / AWS SM / GCP SM / Azure KV)         │
└───────────────┬──────────────┬──────────────────────┘
                │              │
    ┌───────────▼───────┐  ┌──▼──────────────────┐
    │  CI/CD Pipeline   │  │  Runtime Injection   │
    │                   │  │                      │
    │  - Build-time     │  │  - Sidecar (Vault    │
    │    injection      │  │    Agent Injector)   │
    │  - OIDC auth to   │  │  - Init container    │
    │    vault          │  │  - CSI Secret Store  │
    │  - Short-lived    │  │  - Env var injection │
    │    tokens         │  │  - Mounted volumes   │
    └───────────────────┘  └──────────────────────┘

    ┌──────────────────────────────────────────────┐
    │  GitOps Path (encrypted at rest in repo)     │
    │                                              │
    │  SOPS / Sealed Secrets / External Secrets    │
    │  Operator reconciles from source of truth    │
    └──────────────────────────────────────────────┘

Rotation Patterns

Pattern	Description	Frequency	Complexity	Downtime Risk
Manual rotation	Human changes secret, updates consumers	Ad-hoc	Low	High (missed consumers)
Scheduled rotation	Automated job rotates on a cadence	30-90 days	Medium	Low (if tested)
Dynamic secrets	Secret generated per-session, auto-expires	Per request	High (initial setup)	None
Dual-secret rotation	Two active versions, rotate one at a time	Continuous	Medium	None
Zero-standing-secrets	No persistent secrets, JIT generation via OIDC/IRSA	Per request	High	None

Anti-Pattern Catalog

Anti-Pattern	Why It Is Dangerous	What To Do Instead
Secrets in source code	Leaked via git history, forks, logs	Use a secrets manager, scan with truffleHog/gitleaks
Secrets in environment variables	Visible in process listings, crash dumps	Mount as files, use CSI driver, or sidecar injection
Shared service accounts	No attribution, impossible to rotate	Per-service identity, short-lived credentials
Long-lived API keys	Large blast radius if compromised	Dynamic secrets, OIDC federation, auto-rotation
Secrets in CI/CD config files	Accessible to anyone with repo access	OIDC auth to vault, no static secrets in pipelines
Copy-pasting secrets via Slack/email	No audit trail, persisted in chat history	Use a secrets manager with UI, 1Password for teams
No rotation policy	Compromised secrets remain valid indefinitely	Enforce rotation via policy, automate with Lambda/CronJob

Strategic Recommendations

Start with a scan. Before choosing a tool, understand your current exposure. Run truffleHog or gitleaks against your entire git history. The results will be sobering and will justify investment.

Embrace OIDC federation. Modern CI/CD platforms (GitHub Actions, GitLab CI) support OIDC tokens that can authenticate directly to cloud providers and Vault -- eliminating static CI/CD secrets entirely.

Dynamic secrets are the endgame. Vault's dynamic secrets (database credentials generated per pod, expired after TTL) eliminate the rotation problem entirely. This is the architecture to aim for.

Encrypt secrets in git for GitOps. SOPS with age or cloud KMS lets you store encrypted secrets alongside application code. Combined with External Secrets Operator, this is the cleanest GitOps pattern.

Resources

:::