Managing sensitive information like API keys, database credentials, and certificates is a critical security challenge for modern applications. HashiCorp Vault provides a centralized, secure solution for storing, accessing, and managing secrets with robust access controls, audit logging, and encryption at rest.
This guide demonstrates practical secrets management using Vault’s Key-Value (KV) secrets engine, covering everything from initial setup to advanced operations.
Prerequisites
- HashiCorp Vault installed and running
- Vault CLI configured and authenticated
- Administrative access to enable secrets engines
- API keys or credentials to store (for testing purposes)
Understanding Vault Secrets Engines
Vault uses secrets engines to handle different types of sensitive data. The Key-Value (KV) engine is the most commonly used for storing static secrets like API keys, passwords, and configuration data.
KV Engine Versions:
- KV v1: Simple key-value storage without versioning
- KV v2: Advanced features including versioning, metadata, and soft deletes
Step 1: Verify and Enable Secrets Engine
Check Current Secrets Engines
Before storing secrets, verify which engines are available:
vault secrets list
Expected Output:
Path Type Accessor Description
---- ---- -------- -----------
cubbyhole/ cubbyhole cubbyhole_12345678 per-token private secret storage
identity/ identity identity_87654321 identity store
sys/ system system_abcdef123 system endpoints used for control
Enable KV Secrets Engine
If the KV engine isn’t mounted at the secret/ path, enable it:
# Enable KV v2 (recommended for production)
vault secrets enable -path=secret kv-v2
# Verify successful enablement
vault secrets list
Post-Enablement Output:
Path Type Accessor Description
---- ---- -------- -----------
secret/ kv kv_98765432 key/value secret storage
cubbyhole/ cubbyhole cubbyhole_12345678 per-token private secret storage
Step 2: Store API Keys and Credentials
Store Individual API Keys
Store your sensitive API keys using organized path structures:
# Store OpenAI API key
vault kv put secret/api-keys/openai \
api_key="sk-proj-your-openai-key-here" \
organization="your-org-id"
# Store Anthropic API key
vault kv put secret/api-keys/anthropic \
api_key="your-anthropic-key-here"
# Store additional service credentials
vault kv put secret/api-keys/github \
token="ghp_your-github-token" \
username="your-github-username"
Store Application Configuration
Group related secrets for easier management:
# Database credentials
vault kv put secret/myapp/database \
username="db_admin" \
password="secure_db_password" \
host="db.example.com" \
port="5432"
# Application settings
vault kv put secret/myapp/config \
app_secret="your-app-secret-key" \
jwt_secret="jwt-signing-key" \
encryption_key="aes-256-key"
Store with Metadata
Add contextual information for better secret management:
vault kv put secret/api-keys/stripe \
api_key="sk_live_your-stripe-key" \
webhook_secret="whsec_webhook-secret" \
created_date="2025-09-25" \
environment="production" \
owner="finance-team"
Step 3: Retrieve Secrets
View Complete Secrets
Retrieve all key-value pairs for a secret:
# Get full secret with metadata
vault kv get secret/api-keys/openai
# Get secret in JSON format (useful for automation)
vault kv get -format=json secret/api-keys/openai
Sample Output:
====== Secret Path ======
secret/data/api-keys/openai
======= Metadata =======
Key Value
--- -----
created_time 2025-09-25T10:30:00.123456Z
custom_metadata <nil>
deletion_time n/a
destroyed false
version 1
========= Data =========
Key Value
--- -----
api_key sk-proj-your-openai-key-here
organization your-org-id
Extract Specific Values
Get individual field values for use in scripts and applications:
# Get only the API key value
vault kv get -field=api_key secret/api-keys/openai
# Get database password
vault kv get -field=password secret/myapp/database
# Store value in environment variable
export OPENAI_API_KEY=$(vault kv get -field=api_key secret/api-keys/openai)
List Available Secrets
Navigate your secrets hierarchy:
# List all secrets in the root
vault kv list secret/
# List secrets in specific directories
vault kv list secret/api-keys/
vault kv list secret/myapp/
Step 4: Advanced Secrets Management
Version Management (KV v2 Only)
KV v2 maintains version history for all secrets:
# View secret metadata and versions
vault kv metadata get secret/api-keys/openai
# Get specific version
vault kv get -version=1 secret/api-keys/openai
# Update secret (creates new version)
vault kv put secret/api-keys/openai \
api_key="sk-proj-updated-key" \
organization="new-org-id"
# Check version history
vault kv metadata get secret/api-keys/openai
Secret Lifecycle Management
Manage secret versions and deletions:
# Soft delete (recoverable)
vault kv delete secret/api-keys/old-service
# Delete specific versions
vault kv delete -versions=1,2 secret/api-keys/openai
# Permanently destroy versions (irreversible)
vault kv destroy -versions=1,2 secret/api-keys/openai
# Undelete soft-deleted secret
vault kv undelete -versions=3 secret/api-keys/old-service
Patch Operations
Update individual fields without affecting others:
# Update only the API key, preserve other fields
vault kv patch secret/api-keys/stripe \
api_key="sk_live_new-stripe-key"
# Add new field to existing secret
vault kv patch secret/myapp/config \
redis_url="redis://cache.example.com:6379"
Step 5: Integration with Applications
Environment Variable Loading
Create scripts to load secrets into environment variables:
#!/bin/bash
# load-secrets.sh
# Load API keys
export OPENAI_API_KEY=$(vault kv get -field=api_key secret/api-keys/openai)
export ANTHROPIC_API_KEY=$(vault kv get -field=api_key secret/api-keys/anthropic)
# Load database credentials
export DB_USERNAME=$(vault kv get -field=username secret/myapp/database)
export DB_PASSWORD=$(vault kv get -field=password secret/myapp/database)
export DB_HOST=$(vault kv get -field=host secret/myapp/database)
# Start application with loaded secrets
./start-application.sh
Application Integration Examples
Python Application:
import subprocess
import os
def get_vault_secret(path, field):
"""Retrieve secret from Vault"""
result = subprocess.run([
'vault', 'kv', 'get', '-field=' + field, path
], capture_output=True, text=True)
if result.returncode == 0:
return result.stdout.strip()
else:
raise Exception(f"Failed to retrieve secret: {result.stderr}")
# Usage
openai_key = get_vault_secret('secret/api-keys/openai', 'api_key')
db_password = get_vault_secret('secret/myapp/database', 'password')
Troubleshooting Common Issues
Permission Denied (403) Error
Symptoms:
Error making API request.
Code: 403. Errors: * preflight capability check returned 403
Cause: KV secrets engine not enabled at the specified path
Solution:
# Check mounted secrets engines
vault secrets list
# Enable KV v2 at secret/ path if missing
vault secrets enable -path=secret kv-v2
No Handler for Route Error
Symptoms:
Error: no handler for route 'secret/mykey'
Cause: Incorrect path format or missing secrets engine
Solutions:
# Verify secrets engine is mounted
vault secrets list
# Check if using correct KV version
vault kv get secret/mykey # KV v2 (recommended)
Secret Not Found
Symptoms:
No value found at secret/data/mykey
Cause: Secret doesn’t exist or incorrect path
Solutions:
# List available secrets to verify path
vault kv list secret/
# Check if secret exists in different location
vault kv list secret/api-keys/
Security Best Practices
Path Organization
Recommended Structure:
secret/
├── api-keys/
│ ├── openai
│ ├── anthropic
│ └── github
├── databases/
│ ├── production
│ └── staging
└── applications/
├── webapp/config
└── api/credentials
Access Control
Implement least-privilege access:
# Create policy for application access
vault policy write myapp-policy - <<EOF
path "secret/data/myapp/*" {
capabilities = ["read"]
}
path "secret/data/api-keys/openai" {
capabilities = ["read"]
}
EOF
# Apply policy to authentication method
vault write auth/userpass/users/myapp-user \
password="secure-password" \
policies="myapp-policy"
Audit and Monitoring
Enable comprehensive logging:
# Enable audit logging
vault audit enable file file_path=/vault/logs/audit.log
# Monitor secret access patterns
tail -f /vault/logs/audit.log | grep "secret/data"
Regular Maintenance
Maintenance Checklist:
- Rotate API keys and credentials regularly
- Review and clean up unused secrets
- Monitor secret access patterns for anomalies
- Update secret metadata with ownership and expiration info
- Regular backup of Vault data and configuration
Advanced Use Cases
Secret Templates
Create reusable secret templates for consistent structure:
# Template for service API keys
vault kv put secret/templates/api-service \
api_key="REPLACE_WITH_ACTUAL_KEY" \
environment="REPLACE_WITH_ENV" \
owner="REPLACE_WITH_TEAM" \
created_date="REPLACE_WITH_DATE"
Automated Secret Rotation
Implement automated rotation for dynamic secrets:
# Database dynamic secrets (requires database secrets engine)
vault secrets enable database
vault write database/config/my-database \
plugin_name=mysql-database-plugin \
connection_url="root:password@tcp(localhost:3306)/" \
allowed_roles="my-role"
HashiCorp Vault provides enterprise-grade secrets management with the flexibility to accommodate various use cases and security requirements. By implementing proper secret storage, retrieval, and lifecycle management practices, organizations can significantly improve their security posture while maintaining operational efficiency.
