Key Vault: Securing Exchange API Keys with Zero Trust Architecture

Fri, 14 Jul 2023 16:00:00 +0800

How to isolate exchange API keys from business logic using a dedicated Key Vault service with Zero Trust principles—preventing key leakage even if application pods are compromised.

Situation

The Problem

Systems that integrate with cryptocurrency exchanges face a critical security challenge: business logic pods need to sign API requests, but storing API keys directly in those pods creates a massive attack surface.

Traditional approach:

Store API keys in environment variables or Kubernetes secrets
Each pod has direct access to sensitive credentials
Compromised pod = leaked keys = potential fund loss

Constraints:

Multiple services need to sign requests for different trading strategies
Each service runs in ephemeral Kubernetes pods
Need to support CCXT library’s signing mechanism
Must prevent API keys from ever leaving secure storage

Zero Trust Principle

“Never trust, always verify” - even internal services must prove their identity on every request.

Assume breach - design for the scenario where application pods are already compromised. The system should still prevent key leakage and unauthorized operations.

Task

Build a centralized Key Vault service that:

Stores API keys in GCP Secret Manager (encrypted with Cloud KMS)
Verifies every signing request from business pods
Prevents unauthorized API operations (withdrawals, transfers)
Maintains complete audit trail for compliance

Target KPIs:

Zero API keys stored in application pods
Sub-100ms signing latency
100% request verification coverage
Complete audit log for all signing operations

Action

Architecture Overview

Two-Phase Hybrid Architecture:

Phase 1: Identity Bootstrap (On Pod Startup - Once)
┌─────────────────────┐                          ┌──────────────────────┐
│   Trading Pod       │                          │    Key Vault         │
│                     │                          │                      │
│  1. Gen RSA keypair │──(Register: PubKey)─────▶│  1. Verify SA Token  │
│  2. Read SA Token   │    + SA Token            │     via K8s API      │
│                     │◀─(Session OK)────────────│  2. Store PubKey     │
│  3. Store PrivKey   │                          │     in Redis         │
│     in memory       │                          │     (1hr TTL)        │
└─────────────────────┘                          └──────────────────────┘

Phase 2: High-Frequency Trading (Runtime - Every Request)
┌─────────────────────┐                          ┌──────────────────────┐
│   Trading Pod       │                          │    Key Vault         │
│                     │──(Sign Request)─────────▶│                      │
│  4. Sign payload    │    + RSA Signature       │  3. Verify RSA sig   │
│     with PrivKey    │                          │     (lookup Redis)   │
│                     │◀─(Signed Headers)────────│  4. Check whitelist  │
│  5. Send to Binance │                          │  5. Sign with secret │
└─────────────────────┘                          └──────────────────────┘
         │                                                │
         │                                                ▼
         │                                       ┌─────────────────┐
         │                                       │  GCP Secret     │
         │                                       │  Manager + KMS  │
         │                                       └─────────────────┘
         ▼
┌─────────────────────┐
│  Binance API        │
└─────────────────────┘

Key pattern:

Bootstrap: Use K8s SA Token to establish trust (slow but rare)
Runtime: Use attested RSA keys for speed (fast and frequent)

Similar to TLS session resumption or Kerberos TGT logic.

1. Phase 1: Identity Bootstrap

Challenge: Establish trust anchor for RSA keys Solution: Use K8s Service Account Token to attest RSA public key registration

Kubernetes Configuration: Hardened Token Mounting

Before diving into code, configure the pod to use Projected Volumes with audience restriction:

# ... (Standard Deployment spec) ...
      volumes:
      - name: sa-token
        projected:
          sources:
          - serviceAccountToken:
              path: token
              expirationSeconds: 3600
              audience: key-vault  # ⚠️ Critical: Token only valid for Key Vault

Why this matters:

✅ Blast radius containment: If token is stolen, attacker cannot use it to access K8s API or other services
✅ Audience restriction: Token is cryptographically bound to key-vault audience
✅ Explicit mounting: automountServiceAccountToken: false prevents accidental exposure

Step 1A: Pod Generates Keypair and Registers

// Trading pod startup
import * as crypto from 'crypto';
import * as fs from 'fs';
// ...

// Reusable registration function for rotation
async function registerWithKeyVault() {
  // 1. Generate NEW ephemeral keypair (RSA 2048)
  const { publicKey, privateKey } = crypto.generateKeyPairSync(...);

  // 2. CRITICAL: Re-read token from disk! 
  // Kubernetes automatically updates this file before the old one expires.
  const k8sToken = fs.readFileSync('/var/run/secrets/kubernetes.io/serviceaccount/token');

  // 3. Register public key with Key Vault (attested by SA Token)
  const response = await fetch(`${KEY_VAULT_URL}/register`, {
    headers: { 'Authorization': `Bearer ${k8sToken}` },
    body: JSON.stringify({ publicKey }),
  });

  // 4. Update global state (Hot Swap)
  global.signingKey = privateKey;
  // ... update session expiry
}

Step 1B: Key Vault Verifies and Stores Public Key

app.post('/register', async (req, res) => {
  // 1. Verify SA Token & Extract Pod UID
  const identity = await validateK8sToken(req.token); // Validates signature & audience
  const { podUid } = identity;

  // 2. Rate limiting (Max 1 per 5 min per Pod)
  if (await isRateLimited(podUid)) throw new Error('Rate limit exceeded');

  // 3. Store public key in Redis (Keyed by Pod UID)
  // ⚠️ Critical: Use Pod UID to prevent session hijacking
  await redis.set(`session:${podUid}`, JSON.stringify({
    publicKey: req.body.publicKey,
    namespace: identity.namespace
  }), 'EX', 3600);

  return res.json({ sessionId: podUid });
});

Security benefits:

✅ Trust anchor: K8s API cryptographically verifies pod identity
✅ No impersonation: Attacker cannot register keys without valid SA Token
✅ Session isolation: Each pod has unique session (keyed by UID, not Service Account)
✅ Prevents session hijacking: Compromised Pod A cannot overwrite Pod B’s session
✅ Rate limiting: Max 1 registration per pod every 5 minutes (prevents DoS)
✅ Automatic expiry: Session expires after 1 hour, pod must re-register

2. Phase 2: Runtime Signing

Challenge: Minimize latency for API signing requests

Solution: Use RSA signatures verified against Redis-cached public keys

Step 2A: Trading Pod Signs Requests

// Intercept CCXT signing
exchange.fetch2 = async (path, api, method, params, headers, body) => {
  // ... (Standard CCXT logic) ...

  // Sign payload with RSA private key
  const signature = crypto.sign('sha256', Buffer.from(JSON.stringify(payload)), {
    key: global.signingKey,
    padding: crypto.constants.RSA_PKCS1_PSS_PADDING,
  });

  // Send to Key Vault
  const response = await fetch(`${KEY_VAULT_URL}/sign`, {
    headers: {
      'X-Signature': signature.toString('base64'),
      'X-Session-Id': sessionId, // From Phase 1
    },
    // ...
  });
  
  // ... (Execute actual request with signed headers)
};

Step 2B: Key Vault Verifies RSA Signature

app.post('/sign', async (req, res) => {
  // 1. Lookup public key (Fast: <0.5ms)
  const session = await redis.get(`session:${req.headers['x-session-id']}`);
  if (!session) throw new Error('Session expired');

  // 2. Verify RSA signature
  const isValid = crypto.verify(..., session.publicKey, ...);
  if (!isValid) throw new Error('Invalid signature');

  // 3. Security Checks (Nonce, Path Whitelist)
  await checkNonceAtomic(req.body.nonce); // Redis SET NX
  if (!isPathAllowed(req.body.path)) throw new Error('Forbidden path');

  // 4. Sign with real API key
  const apiKey = await getSecretFromGCP(req.body.secretName);
  // ... (CCXT signing logic) ...
});

Future enhancements for compliance:

Immutable storage: Ship logs to Cloud Storage with retention policy
BigQuery integration: Structured analysis for SOC 2 Type II reports
Correlation: Link audit logs with distributed tracing (Trace ID)

Performance benefits:

✅ Sub-millisecond verification: Redis lookup + RSA verify ~0.5ms
✅ No K8s API dependency: Hot path doesn’t touch control plane
✅ Horizontal scaling: Stateless verification (all state in Redis)
✅ Control plane resilience: API requests continue even if K8s API is slow

3. API Path Whitelist

Critical Security Layer:

Binance API has hundreds of endpoints. We only allow specific operations:

const ALLOWED_PATHS = [
  '/api/v3/order',      // Trading OK
  '/fapi/v1/order',     // Futures OK
];

const FORBIDDEN_PATTERNS = [
  /withdraw/i,          // ⛔️ Block withdrawals
  /transfer/i,          // ⛔️ Block transfers
];

Defense in depth:

Even if an attacker compromises a trading pod and obtains the RSA private key, they cannot:

Execute withdrawal operations (blocked by path whitelist)
Replay old requests (blocked by nonce check)
Sign requests for other services (blocked by service ID binding)

4. GCP Secret Manager Integration

Key management:

Secrets encrypted with Cloud KMS
IAM binding restricts access to specific service account
Cache with TTL to reduce API calls

import { SecretManagerServiceClient } from '@google-cloud/secret-manager';

const secretClient = new SecretManagerServiceClient();

// Create API key secret (Simplified)
async function createApiKeySecret(name: string, apiKey: string, apiSecret: string) {
  // 1. Create secret and add version
  const [secret] = await secretClient.createSecret({...});
  await secretClient.addSecretVersion({...});

  // 2. Set IAM policy: Only Key Vault service account can access
  // ⚠️ Critical: Even project owners cannot see content if not explicitly added
  await secretClient.setIamPolicy({
    resource: secret.name,
    policy: {
      bindings: [{
        role: 'roles/secretmanager.secretAccessor',
        members: [`serviceAccount:key-vault@${projectId}.iam.gserviceaccount.com`],
      }],
    },
  });
}

// Get secret with caching
const secretCache = new NodeCache({ stdTTL: 60 }); // 60 sec cache

async function getSecretFromGCP(secretName: string) {
  if (secretCache.has(secretName)) return secretCache.get(secretName);
  
  // Fetch from GCP (incur latency)
  const [version] = await secretClient.accessSecretVersion({ 
    name: `projects/${projectId}/secrets/${secretName}/versions/latest` 
  });
  
  const payload = JSON.parse(version.payload.data.toString());
  secretCache.set(secretName, payload);
  return payload;
}

// Active invalidation mechanism
// ... (Redis pub/sub logic for immediate cache clearing)

Cache strategy:

60-second TTL: Balance between performance and security
Active invalidation: Immediate cache clear when secret rotates
Redis pub/sub: Synchronize cache across all Key Vault instances
Emergency rotation: If Binance API key compromised, invalidate immediately

Result

Achieved:

✅ Zero API keys in application pods (all in GCP Secret Manager)
✅ 60ms average signing latency (end-to-end including network hop)
✅ Prevented unauthorized API operations via path whitelist
✅ Complete audit trail in Cloud Logging

Security improvements:

Compromised trading pod cannot leak API keys
Attackers cannot execute withdrawal operations even with pod access
All signing requests are logged and traceable

Current Zero Trust coverage

✅ Verify Explicitly
✅ Least Privileged Access
⚠️ Assume Breach: missing real-time anomaly detection

Optimization Roadmap

To achieve 95%+ Zero Trust compliance:

1. mTLS Communication

Certificate-based authentication:

Issue client certificates to each pod
Certificate pinning for Key Vault
Automatic rotation with cert-manager

Service Mesh integration:

Istio/Linkerd for automatic mTLS
Zero-config encryption between services

2. High Availability & Circuit Breaker

Current risk: Key Vault is single point of failure (SPOF)

Design philosophy:

Fail-close over fail-open: If security components fail, reject requests rather than bypass checks
Horizontal scaling: Multiple Key Vault instances behind load balancer
Stateless design: All state in Redis/GCP, allows easy scaling
Circuit breaker: Prevent cascading failures with aggressive timeouts (e.g. 100ms) and fallback logic

3. Migrate to gRPC for Enhanced Security

Current: REST API with JSON over HTTPS

Proposed: gRPC with mutual TLS (mTLS)

Why gRPC is more secure:

Automatic mTLS: Client certificate verification built-in, no manual implementation
Type safety: Protobuf schema prevents injection attacks and malformed requests
Binary protocol: Harder to intercept and tamper with than JSON
HTTP/2 multiplexing: Single TLS handshake for multiple requests (reduced attack surface)
Streaming support: Can implement real-time audit streaming

Trade-offs:

❌ More complex debugging (binary protocol, need tools like grpcurl)
❌ Requires Protobuf schema management and versioning
❌ Less firewall-friendly than REST (HTTP/2 may be blocked)
❌ Steeper learning curve for team

Migration strategy:

Implement gRPC alongside REST (dual protocol support)
Migrate critical operations first (/sign endpoint)
Keep management operations on REST for easier debugging
Monitor performance and security metrics
Gradually deprecate REST once stable

Zero-Trust on ZeroJ's Blog