Secrets Management Best Practices: Beyond Secret Scanning
Secret scanning catches credentials in your code — but that is only one piece of the puzzle. True secrets management covers the entire lifecycle: how secrets are generated, stored, distributed, rotated, and revoked. This guide goes beyond scanning to cover the operational practices that prevent secret leaks from happening in the first place.
Why Secrets Management Matters
new secrets were leaked on GitHub in 2023 alone
average time for attackers to exploit a leaked AWS key
average time to detect and contain a credential breach
Leaked secrets are among the fastest paths from code to compromise. API keys, database credentials, and private keys give attackers direct access to systems without needing to exploit any vulnerability. The problem is not just preventing leaks — it is managing the full lifecycle of every secret in your organization.
The Secrets Lifecycle
Create
Generate secrets with sufficient entropy. Use cryptographic random generators, not predictable patterns.
Store
Store in a dedicated vault with encryption at rest, access control, and audit logging. Never in code, env files, or wikis.
Rotate
Automatically rotate on a schedule (30–90 days) and immediately after any suspected compromise.
Revoke
Revoke and delete secrets when no longer needed. Decommission service accounts, rotate after employee offboarding.
Secrets Management Best Practices
Use a Centralized Vault
Store all secrets in a dedicated secrets manager (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager). Centralization provides audit trails, access controls, and rotation in one place.
Prefer Short-Lived Tokens
Use tokens that expire after 15–60 minutes instead of long-lived API keys. AWS STS, GCP workload identity, and JWT with short TTLs minimize the blast radius of leaked credentials.
Automate Rotation
Configure automatic rotation for database passwords, API keys, and service account credentials. Manual rotation does not scale and creates gaps during rotation windows.
Implement RBAC for Secrets
Not every service needs every secret. Scope secret access by service, environment, and team. A web server should not have access to database admin credentials.
Never Store Secrets in Code
No secrets in source code, environment files committed to git, Dockerfiles, CI/CD configs, or wiki pages. Use .gitignore, pre-commit hooks, and secret scanning to enforce this.
Encrypt Secrets in Transit
Fetch secrets over TLS. Use the vault's SDK rather than writing secrets to disk. Prefer in-memory injection over environment variables when possible.
Audit Secret Access
Log every secret read, write, and rotation event. Alert on unusual access patterns — a service reading a secret it has never accessed before is a signal.
Plan for Compromise
Have a runbook for leaked secrets: immediate revocation, rotation of affected credentials, forensic analysis of what was accessed, and notification of affected parties.
Secrets Management Tools Comparison
| Tool | Type | Auto-Rotation | Dynamic Secrets | Best For |
|---|---|---|---|---|
| HashiCorp Vault | Self-hosted / HCP | Yes | Yes | Multi-cloud, dynamic secrets |
| AWS Secrets Manager | Managed (AWS) | Yes | No | AWS-native workloads |
| GCP Secret Manager | Managed (GCP) | Manual | No | GCP-native workloads |
| Azure Key Vault | Managed (Azure) | Yes | No | Azure-native workloads |
| CyberArk | Enterprise | Yes | Yes | Enterprise privileged access |
| Doppler | SaaS | Yes | No | Developer-first teams |
Runtime Secrets Protection with eBPF
Secret scanning catches credentials in code before deployment. But what about secrets being accessed at runtime? eBPF monitoring detects when processes read credential files (/root/.aws/credentials, /etc/shadow, .env files) or access secrets through environment variables — catching both legitimate and malicious secrets access in real time.
Where Secrets Actually Leak: Common Exposure Vectors
Most teams assume secret leaks happen when a developer accidentally commits an API key to a public repository. That vector is real, but it is only one of many. Understanding the full attack surface is the first step to closing it. In practice, secrets escape through a surprisingly wide range of channels, and each one needs its own control.
Git History, Not Just HEAD
Deleting a secret from the current commit does not remove it from history. A key committed and reverted three years ago is still retrievable with a single git log -p command. Scanners must analyze the full commit history, and any secret ever committed must be treated as compromised and rotated.
CI/CD Logs and Build Artifacts
Pipelines frequently echo environment variables during debugging, embed credentials in build artifacts, or write them into cached layers. Attackers who compromise a CI system — or simply read public build logs — harvest credentials without ever touching source code.
Container Images and Layers
A Dockerfile that copies a .env file or runs an export command bakes the secret into an image layer permanently, even if a later layer deletes the file. Anyone who can pull the image can extract every intermediate layer with standard tooling.
Infrastructure-as-Code State Files
Terraform state files store resource attributes in plaintext, including generated database passwords and access keys. State stored in an unencrypted S3 bucket or committed to git is a high-value target that most secret scanners overlook.
Ticketing Systems, Wikis, and Chat
Credentials pasted into Jira tickets, Confluence pages, and Slack threads persist indefinitely and are searchable by anyone with workspace access. Several public breach post-mortems trace initial access to a credential found in an internal wiki.
Logs and Error Traces at Runtime
Applications that log full request headers, connection strings, or exception context routinely write bearer tokens and passwords into log aggregation platforms — where retention policies keep them for months and access controls are often looser than production.
Secret Rotation Strategies That Do Not Break Production
Rotation is where most secrets programs stall. Teams agree that credentials should rotate every 30–90 days, then discover that rotating a database password takes down every service that cached the old value. The fix is not to rotate less — it is to design rotation so that it is safe to run at any time. Three patterns cover the vast majority of cases.
Two-Key Overlap (Dual Credentials)
Provision two valid credentials at once — key A and key B. Services use key A while key B is rotated, then traffic shifts to key B while key A is rotated. AWS access keys, most API providers, and signing keys support this natively. There is never a moment when zero valid credentials exist, so rotation cannot cause an outage.
Dynamic, Just-in-Time Secrets
Instead of rotating a shared static password, eliminate it. HashiCorp Vault's database secrets engine creates a unique, short-lived database user for each service instance on demand and revokes it when the lease expires. A leaked credential is worthless within minutes, and rotation becomes a non-event because every credential is already ephemeral.
Identity-Based Access (No Secret at All)
The strongest rotation strategy is removing the secret entirely. AWS IAM roles for EC2 and EKS, GCP workload identity federation, Azure managed identities, and OIDC-based CI/CD authentication (GitHub Actions to AWS, for example) let workloads prove who they are cryptographically. There is nothing to store, leak, or rotate.
Whichever pattern you adopt, rotation must be tested the same way you test backups: by actually running it. A rotation runbook that has never executed in production will fail during an incident, which is precisely when you need it most. Schedule rotation drills quarterly and treat a failed rotation as a sev-2 engineering defect, not an operational annoyance.
Handling Secrets in CI/CD Pipelines
CI/CD systems are among the most credential-dense environments in any organization: they hold deploy keys, cloud credentials, registry logins, and signing keys — often with production-level privileges. The 2023 CircleCI breach demonstrated the blast radius: a single compromised build platform forced thousands of customers to rotate every secret they had ever stored in it. Hardening pipeline secrets is therefore not optional.
Platforms like TigerGate integrate secret scanning directly into the code scanning workflow alongside SAST and IaC checks, so a leaked credential surfaces in the same pull request feedback loop developers already use — rather than in a separate security tool nobody reads.
Secrets Management and Compliance Requirements
Secrets management is not just good hygiene — it is an auditable control in every major compliance framework. SOC 2 Trust Service Criteria CC6.1 requires logical access controls over credentials, and auditors increasingly ask how secrets are stored, who can read them, and how rotation is evidenced. PCI-DSS Requirement 8 mandates credential rotation and prohibits shared accounts, while ISO 27001 Annex A covers privileged access management and cryptographic key handling.
The practical implication: your secrets program needs to produce evidence, not just security. A vault with audit logging enabled generates that evidence automatically — every read, write, and rotation is timestamped and attributable to an identity. Compare that to spreadsheet-tracked shared passwords, where proving compliance means reconstructing history by hand. Teams pursuing SOC 2 or ISO 27001 should prioritize three evidence streams: vault access logs mapped to CC6.1, rotation records demonstrating enforcement of the rotation policy, and secret scanning reports showing continuous monitoring of source code. Runtime detection adds a fourth layer — continuous proof that credential files are not being accessed by unauthorized processes, which maps directly to monitoring criteria like CC7.2.
Frequently Asked Questions
Are environment variables a safe place for secrets?
They are better than hardcoding, but far from ideal. Environment variables are inherited by child processes, appear in crash dumps and debugging endpoints, and are readable via /proc on Linux. Prefer fetching secrets at startup from a vault SDK and holding them only in application memory, or use file-based injection with tight permissions in Kubernetes.
We found a secret in git history. Is rewriting history enough?
No. Treat any committed secret as compromised the moment it lands in the repository — clones, forks, and CI caches may already hold copies. Rotate the credential first, then clean history with tools like git-filter-repo to reduce future noise. Rotation is the remediation; history rewriting is housekeeping.
How is secret scanning different from secrets management?
Scanning is detective: it finds credentials that already leaked into code, config, or images. Management is preventive: vault storage, scoped access, short-lived credentials, and automated rotation reduce the chance a leak happens and shrink the damage when one does. Mature programs run both, plus runtime monitoring to catch access to credentials that scanning cannot see.
What should we do in the first 30 days of a secrets program?
Inventory first: run a full-history scan across every repository and container registry to find existing exposure. Rotate everything found, starting with cloud provider keys. Then stand up a vault for the top ten most sensitive credentials, wire secret scanning into CI as a blocking check, and publish a one-page policy covering storage, rotation cadence, and the leak-response runbook.
Detect Secret Leaks with TigerGate
TigerGate scans code for hardcoded secrets, monitors runtime credential access with eBPF, and audits cloud IAM configurations — closing the loop on secrets security.