Secret Management: The Problem of Secrets in Software
Secrets (credentials, API keys, certificates, access tokens) are the most sensitive elements of a software infrastructure. A single exposed secret can compromise entire systems, databases, and cloud accounts. According to GitGuardian research, in 2024 over 12 million secrets were found exposed in public repositories on GitHub.
Secret management is not just about "not committing passwords to code." It's a complete system covering the generation, secure storage, distribution, rotation, and revocation of secrets throughout their entire lifecycle.
What You'll Learn
- Architecture of a secret management system
- HashiCorp Vault: setup, configuration, and integration
- AWS Secrets Manager and cloud alternatives
- Automatic credential rotation strategies
- Secret detection in repositories with GitLeaks and TruffleHog
- Common anti-patterns and how to avoid them
Anti-Patterns: How NOT to Manage Secrets
Before exploring solutions, let's look at the most common mistakes that lead to credential exposure:
- Hardcoded in code: passwords and API keys written directly in source code
- .env files in repository: configuration files with secrets accidentally committed
- Unprotected environment variables: secrets passed as env vars without encryption
- Secrets shared via chat: credentials sent on Slack, Teams, or email
- Never-rotated secrets: static credentials that are never changed
- Over-broad permissions: API keys with access to all services instead of the minimum required
The Cost of an Exposed Secret
An exposed secret in a public repository is discovered by automated bots on average within 24 seconds. In less than a minute, malicious actors can begin exploiting the credential. The average remediation time in organizations without automated processes is 327 days.
HashiCorp Vault: The Reference Standard
HashiCorp Vault is the most widely adopted enterprise secret management solution. It offers secure storage, controlled access, audit logging, and support for dynamic secret rotation. Vault can generate credentials on-demand with limited TTL, eliminating the problem of static secrets.
Vault Setup with Docker
# docker-compose.vault.yml
version: "3.8"
services:
vault:
image: hashicorp/vault:1.15
container_name: vault
ports:
- "8200:8200"
environment:
VAULT_DEV_ROOT_TOKEN_ID: "dev-root-token"
VAULT_DEV_LISTEN_ADDRESS: "0.0.0.0:8200"
cap_add:
- IPC_LOCK
volumes:
- vault-data:/vault/data
volumes:
vault-data:
Basic Operations with Vault CLI
# Configure the client
export VAULT_ADDR="http://127.0.0.1:8200"
export VAULT_TOKEN="dev-root-token"
# Enable KV v2 secrets engine
vault secrets enable -version=2 kv
# Write a secret
vault kv put kv/myapp/database \
username="dbadmin" \
password="super-secret-password" \
host="db.internal.company.com"
# Read a secret
vault kv get kv/myapp/database
# Read a specific field
vault kv get -field=password kv/myapp/database
# Create an access policy
vault policy write myapp-read - <<EOF
path "kv/data/myapp/*" {
capabilities = ["read", "list"]
}
EOF
# Create a token with limited policy
vault token create -policy=myapp-read -ttl=1h
Dynamic Secrets: On-Demand Credentials
One of Vault's most powerful features is dynamic secrets: credentials generated at request time with a limited TTL. When the TTL expires, Vault automatically revokes the credentials. This eliminates the static secrets problem.
# Enable the database secrets engine
vault secrets enable database
# Configure PostgreSQL connection
vault write database/config/mydb \
plugin_name=postgresql-database-plugin \
allowed_roles="readonly" \
connection_url="postgresql://{{username}}:{{password}}@db:5432/mydb?sslmode=disable" \
username="vault_admin" \
password="vault_admin_password"
# Create a role with dynamic credentials
vault write database/roles/readonly \
db_name=mydb \
creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
default_ttl="1h" \
max_ttl="24h"
# Get dynamic credentials
vault read database/creds/readonly
AWS Secrets Manager
For organizations operating on AWS, AWS Secrets Manager offers a managed service for secret management with built-in automatic rotation, native support for RDS, Redshift, and DocumentDB, and seamless integration with AWS services.
Secret Management Solutions Comparison
- HashiCorp Vault: multi-cloud, self-hosted or managed (HCP), maximum flexibility
- AWS Secrets Manager: AWS native, automatic RDS rotation, easy to use
- Azure Key Vault: Azure native, AD integration, HSM support
- GCP Secret Manager: GCP native, IAM integration, versioning
- Kubernetes Secrets: base64 encoded (not encrypted!), use with external-secrets-operator
Secret Detection in Repositories
Prevention is the first line of defense. Secret detection tools analyze code looking for exposed credentials before they reach the remote repository.
GitLeaks: Pre-Commit Hook
# .pre-commit-config.yaml
repos:
- repo: https://github.com/gitleaks/gitleaks
rev: v8.18.0
hooks:
- id: gitleaks
# .gitleaks.toml - custom configuration
[extend]
useDefault = true
[[rules]]
id = "custom-api-key"
description = "Custom API Key pattern"
regex = '''MYAPP_API_KEY\s*=\s*['"]([A-Za-z0-9_-]+)['"]'''
secretGroup = 1
entropy = 3.5
[allowlist]
paths = [
'''test/.*''',
'''.*_test\.go''',
'''.*\.md'''
]
CI/CD Integration
# .github/workflows/secret-detection.yml
name: Secret Detection
on:
push:
branches: [main, develop]
pull_request:
jobs:
gitleaks:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: GitLeaks scan
uses: gitleaks/gitleaks-action@v2
env:
GITHUB_TOKEN: