LiteLLM requires multiple sensitive credentials — OpenAI and Anthropic API keys, a database password, and a master key for its own API. Storing them in native Kubernetes secrets is the path of least resistance. It’s also an audit finding — base64 encoding is not encryption, any cluster admin can read them, and there’s no audit trail of who accessed what. That’s not secrets management, it’s credential storage with an obscurity layer that satisfies no control requirement.

This post uses HashiCorp Vault to store credentials with encryption at rest and policy-controlled access. The External Secrets Operator syncs them into Kubernetes automatically, and nothing sensitive lives in your manifests or version control.

Vault Configuration

Enable a KV v2 secrets engine scoped to LiteLLM, populate it with credentials, then create a policy granting read-only access to those paths. The policy is what limits ESO to exactly what it needs and nothing else — a least-privilege boundary enforced at the Vault layer rather than relying on cluster RBAC alone.

# Enable KV secrets engine
vault secrets enable -path=litellm kv-v2

# Store credentials
vault kv put litellm/api-keys \
  master-key="sk-your-secure-master-key" \
  salt-key="sk-your-secure-salt-key" \
  openai-key="sk-your-openai-api-key" \
  anthropic-key="sk-your-anthropic-api-key"

vault kv put litellm/database \
  url="postgresql://litellm:password@postgres:5432/litellm?sslmode=require"

# Create read-only policy for ESO
vault policy write eso-litellm-policy - <<EOF
path "litellm/data/api-keys" {
  capabilities = ["read"]
}

path "litellm/data/database" {
  capabilities = ["read"]
}
EOF

External Secrets Operator

ESO is the bridge between Vault and Kubernetes. It watches your ExternalSecret resources, fetches the referenced values from Vault, and creates or updates the corresponding Kubernetes Secret objects automatically — including on rotation. When a credential is rotated in Vault, ESO picks up the new value within the refresh interval and updates the secret without any manual intervention.

helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets \
  --namespace external-secrets-system --create-namespace

Option 1 — Token Auth

Generate a token scoped to the ESO policy and store it in a Kubernetes secret. ESO uses it to authenticate to Vault.

vault token create -policy=eso-litellm-policy -period=768h

Copy the token value from the output, then create the secret and SecretStore:

apiVersion: v1
kind: Secret
metadata:
  name: vault-token
  namespace: ai-services
type: Opaque
stringData:
  token: your-vault-token-here
---
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: ai-services
spec:
  provider:
    vault:
      server: "https://vault.example.com:8200"
      path: "secret"
      version: "v2"
      auth:
        tokenSecretRef:
          name: "vault-token"
          key: "token"

Option 2 — Kubernetes Auth (Recommended)

Vault validates ESO’s service account JWT directly. No static token, no manual rotation, no credential stored in the cluster.

# Enable Kubernetes auth
vault auth enable kubernetes

# Configure it with your cluster's API server and CA
vault write auth/kubernetes/config \
  kubernetes_host="https://your-cluster-api:6443" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

# Create a role binding ESO's service account to the policy
vault write auth/kubernetes/role/eso-litellm \
  bound_service_account_names=external-secrets \
  bound_service_account_namespaces=external-secrets-system \
  policies=eso-litellm-policy \
  ttl=1h

The SecretStore replaces the token reference with a Kubernetes auth block — no Secret manifest needed:

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: ai-services
spec:
  provider:
    vault:
      server: "https://vault.example.com:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: kubernetes
          role: eso-litellm

ExternalSecret

Each entry in data maps a Vault key to a Kubernetes secret key. The refreshInterval controls how often ESO checks for updated values.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: litellm-secrets
  namespace: ai-services
spec:
  refreshInterval: 15s
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: litellm-secrets
    creationPolicy: Owner
  data:
  - secretKey: LITELLM_MASTER_KEY
    remoteRef:
      key: litellm/api-keys
      property: master-key
  - secretKey: LITELLM_SALT_KEY
    remoteRef:
      key: litellm/api-keys
      property: salt-key
  - secretKey: OPENAI_API_KEY
    remoteRef:
      key: litellm/api-keys
      property: openai-key
  - secretKey: ANTHROPIC_API_KEY
    remoteRef:
      key: litellm/api-keys
      property: anthropic-key
  - secretKey: DATABASE_URL
    remoteRef:
      key: litellm/database
      property: url

LiteLLM Deployment

The deployment pulls every credential from the litellm-secrets object ESO manages. STORE_MODEL_IN_DB enables PostgreSQL persistence and RUN_MIGRATION handles schema setup on first boot.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: litellm
  namespace: ai-services
spec:
  replicas: 2
  selector:
    matchLabels:
      app: litellm
  template:
    metadata:
      labels:
        app: litellm
    spec:
      containers:
      - name: litellm
        image: ghcr.io/berriai/litellm:main-stable
        ports:
        - containerPort: 4000
        env:
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: litellm-secrets
              key: DATABASE_URL
        - name: STORE_MODEL_IN_DB
          value: "True"
        - name: RUN_MIGRATION
          value: "True"
        - name: LITELLM_MASTER_KEY
          valueFrom:
            secretKeyRef:
              name: litellm-secrets
              key: LITELLM_MASTER_KEY
        - name: LITELLM_SALT_KEY
          valueFrom:
            secretKeyRef:
              name: litellm-secrets
              key: LITELLM_SALT_KEY
        - name: OPENAI_API_KEY
          valueFrom:
            secretKeyRef:
              name: litellm-secrets
              key: OPENAI_API_KEY
        - name: ANTHROPIC_API_KEY
          valueFrom:
            secretKeyRef:
              name: litellm-secrets
              key: ANTHROPIC_API_KEY
        livenessProbe:
          httpGet:
            path: /health/liveliness
            port: 4000
          initialDelaySeconds: 40
          periodSeconds: 30
        readinessProbe:
          httpGet:
            path: /health/readiness
            port: 4000
          initialDelaySeconds: 10
          periodSeconds: 5
        resources:
          requests:
            memory: "512Mi"
            cpu: "200m"
          limits:
            memory: "1Gi"
            cpu: "1000m"

Service & Ingress

apiVersion: v1
kind: Service
metadata:
  name: litellm-service
  namespace: ai-services
spec:
  selector:
    app: litellm
  ports:
  - port: 4000
    targetPort: 4000
    name: http
  type: ClusterIP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: litellm-ingress
  namespace: ai-services
  annotations:
    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  tls:
  - hosts:
    - litellm.example.com
    secretName: litellm-tls
  rules:
  - host: litellm.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: litellm-service
            port:
              number: 4000

Verification

# Confirm ESO synced the secrets from Vault
kubectl get externalsecret -n ai-services
kubectl describe externalsecret litellm-secrets -n ai-services

# Check pods are running
kubectl get pods -n ai-services
kubectl logs -n ai-services deployment/litellm

# Test health endpoint
kubectl port-forward -n ai-services svc/litellm-service 4000:4000
curl http://localhost:4000/health

# Test the API
curl -X POST https://litellm.example.com/v1/chat/completions \
  -H "Authorization: Bearer YOUR_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Control Mapping

The architecture here — Vault as the authoritative credential store, ESO as the sync layer, Kubernetes secrets as the runtime delivery mechanism — addresses several controls that native Kubernetes secret management cannot satisfy on its own.

IA-5 (Authenticator Management) requires that credentials are protected and managed through their lifecycle. The long-lived Vault token used in Option 1 is technically an IA-5 gap — it’s a static credential that needs to be rotated manually. Kubernetes auth eliminates this by removing the long-lived token entirely and tying access to the pod’s service account identity instead.

AC-3 (Access Enforcement) requires that access to resources is enforced in accordance with policy. The ESO Vault policy grants read access only to litellm/data/api-keys and litellm/data/database. Nothing else in Vault is accessible through that token regardless of what ESO requests. That’s access enforcement at the secrets layer, independent of cluster RBAC.

SC-28 (Protection of Information at Rest) requires that sensitive data stored on the system is encrypted. Vault encrypts all secrets at rest using AES-256-GCM. Native Kubernetes secrets are base64 encoded and stored in etcd — encrypted at rest only if etcd encryption is explicitly configured, which most clusters don’t have enabled by default.

AU-2 and AU-12 (Audit Events and Audit Record Generation) require that access to sensitive data is logged. Vault’s audit log captures every read against the litellm/ path — including the token identity, timestamp, and which keys were accessed. Native Kubernetes secrets have no equivalent access log. If an API key is compromised, the Vault audit log tells you exactly when it was last read and by what identity.

CM-7 (Least Functionality) requires restricting access to only what is necessary for legitimate operation. The ESO policy is the implementation of this at the secrets layer — the token can read two specific paths and nothing else. It can’t write, delete, or access any other path in Vault. That boundary is defined in policy and enforced by Vault, not assumed.

Key Points

• Native Kubernetes secrets are base64 encoded, not encrypted — Vault with etcd encryption disabled is a meaningful security gap that shows up in cluster security assessments
• Vault’s KV v2 engine keeps all credentials out of manifests and version control, with encryption at rest and a full audit trail of every access
• ESO handles sync automatically — rotate a secret in Vault and Kubernetes picks it up within the refresh interval without redeployment
• The ESO policy grants read-only access to specific paths only — a least-privilege boundary enforced at the Vault layer regardless of what cluster permissions exist
• Kubernetes auth is the production approach — no static token, no manual rotation, access tied directly to the pod’s service account identity
• RUN_MIGRATION only needs to be True on first boot; safe to leave on, but can be disabled after the schema is initialized