Multi-Region Kubernetes Migration for FinTech SaaS

The Challenge

A FinTech SaaS company processing $500M in annual transactions faced critical infrastructure limitations:

• Single region: All infrastructure in us-east-1. An AWS outage would take them down completely
• Manual scaling: Engineers SSH’d into servers to scale capacity
• Compliance gaps: Pursuing SOC 2 certification but lacking audit trails and security controls
• No DR plan: Backups existed but had never been tested
• Inconsistent environments: Production bugs that couldn’t be reproduced in staging

With customers demanding 99.95% uptime SLAs and auditors knocking, they needed a modern, resilient platform.

The Solution

Phase 1: Assessment & Planning (Month 1)

Comprehensive audit revealed:

• 42 microservices across 80 EC2 instances
• 3 PostgreSQL databases (largest: 2TB)
• Critical dependencies: Stripe, Plaid, Auth0
• Peak load: 15K requests/second

Architecture decision: EKS over self-managed Kubernetes

• Reduced operational burden
• Automatic control plane upgrades
• Native AWS integrations (IAM, VPC, CloudWatch)

Risk mitigation:

• Parallel run: Keep old infrastructure until migration proven
• Gradual traffic shifting: 1% → 10% → 50% → 100%
• Automated rollback triggers

Phase 2: Foundation (Months 2-3)

Built multi-region Kubernetes infrastructure:

# Terraform: EKS clusters in two regions
module "eks_primary" {
  source = "./modules/eks"
  
  region             = "us-east-1"
  cluster_name       = "fintech-prod-primary"
  node_groups = {
    general = {
      desired_size = 6
      min_size     = 3
      max_size     = 12
      instance_types = ["m5.2xlarge"]
    }
  }
}

module "eks_secondary" {
  source = "./modules/eks"
  
  region             = "us-west-2"
  cluster_name       = "fintech-prod-secondary"
  node_groups = {
    general = {
      desired_size = 3
      min_size     = 3
      max_size     = 12
      instance_types = ["m5.2xlarge"]
    }
  }
}

Implemented service mesh (Istio) for:

• mTLS between all services
• Fine-grained traffic routing
• Circuit breaking and retries
• Observability (distributed tracing)

Set up GitOps with ArgoCD:

# ArgoCD application
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payment-service
spec:
  project: default
  source:
    repoURL: https://github.com/company/k8s-manifests
    targetRevision: main
    path: services/payment
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

All infrastructure changes now go through pull requests. Auditors loved this.

Phase 3: Migration (Months 3-4)

Service-by-service migration strategy:

1. Start with read-only services (analytics dashboard, reporting)
2. Then internal services (admin tools, batch jobs)
3. Finally, customer-facing services (API, payment processing)

Typical service migration:

# Kubernetes deployment with best practices
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
  namespace: production
spec:
  replicas: 4
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  selector:
    matchLabels:
      app: payment-service
  template:
    metadata:
      labels:
        app: payment-service
        version: v2.3.1
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
    spec:
      serviceAccountName: payment-service
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        fsGroup: 1000
      containers:
      - name: payment
        image: 123456789.dkr.ecr.us-east-1.amazonaws.com/payment-service:v2.3.1
        ports:
        - containerPort: 8080
          name: http
        env:
        - name: DATABASE_URL
          valueFrom:
            secretKeyRef:
              name: payment-db-credentials
              key: url
        - name: STRIPE_API_KEY
          valueFrom:
            secretKeyRef:
              name: stripe-credentials
              key: api_key
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "1Gi"
            cpu: "1000m"
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
---
apiVersion: v1
kind: Service
metadata:
  name: payment-service
  namespace: production
spec:
  selector:
    app: payment-service
  ports:
  - port: 80
    targetPort: 8080
  type: ClusterIP
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: payment-service
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payment-service
  minReplicas: 4
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80

Database migration:

• Aurora PostgreSQL with cross-region replication
• Blue/green switch minimized downtime to <60 seconds
• Extensive testing with production-like data

Phase 4: Disaster Recovery & Compliance (Month 5)

Implemented automated DR testing:

#!/bin/bash
# Monthly DR drill script

# 1. Fail over to secondary region
kubectl config use-context eks-us-west-2
kubectl apply -f manifests/production/

# 2. Update DNS to point to secondary
aws route53 change-resource-record-sets --hosted-zone-id Z123 \
  --change-batch file://failover-dns.json

# 3. Run smoke tests
./scripts/smoke-tests.sh

# 4. Verify transactions processing
./scripts/verify-payments.sh

# 5. Measure RTO/RPO
echo "Failover completed in $(date -u +%s - START_TIME) seconds"

Run monthly. First test took 90 minutes. Latest: 15 minutes.

SOC 2 compliance measures:

• Audit logs for all kubectl commands (shipped to immutable S3 bucket)
• Pod security policies enforced
• Network policies isolating services
• Secrets encrypted with AWS KMS
• Regular vulnerability scanning (Trivy + Snyk)

# NetworkPolicy: Payment service can only talk to database and Stripe
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: payment-service
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: payment-service
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: api-gateway
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: postgres
  - to:
    - namespaceSelector: {}
      podSelector:
        matchLabels:
          k8s-app: kube-dns
  - to:  # Stripe API
    - ipBlock:
        cidr: 54.187.0.0/16

The Results

Infrastructure Metrics

• Migration: 42 services migrated with zero customer-facing downtime
• RTO: 24 hours → 15 minutes (99.3% improvement)
• RPO: 4 hours → 5 minutes (97.9% improvement)
• Deployment frequency: 8x/week → 40x/week

Cost Impact

• Infrastructure costs increased 18% (multi-region + overhead)
• But: No more manual intervention, outage risk massively reduced
• ROI: Positive after first prevented outage

Compliance

• SOC 2 Type II: Passed audit on first attempt
• Audit findings: Zero critical or high-severity issues
• Auditors specifically praised disaster recovery testing

Team Impact

• Engineers deploy confidently (GitOps + automated rollbacks)
• On-call burden reduced 60% (better observability, auto-healing)
• New engineer onboarding: 3 weeks → 4 days

Lessons Learned

1. Test your DR plan obsessively
We found 14 issues during DR drills that would have caused real outages. Monthly testing made failover routine.

2. Service mesh complexity is real
Istio added operational overhead. But the security, observability, and traffic control were worth it for this use case.

3. GitOps is transformative for compliance
Every change has a pull request. Auditors can see who changed what, when, and why.

4. Parallel run saved us
Keeping the old infrastructure running during migration let us catch issues without impacting customers.

5. Observability before migration
We instrumented services before moving them. This let us compare behavior pre/post migration.

What They Said

“We went from fearing outages to confidently testing failovers every month. That peace of mind is priceless.”

— CTO

“The SOC 2 audit was the smoothest process we’ve ever had. Our auditors were impressed.”

— Head of Compliance

Long-Term Impact

18 months post-migration:

• Zero multi-hour outages (previous: 2-3 per year)
• Passed SOC 2 Type II renewal with zero findings
• Scaled to 3x traffic without infrastructure changes (auto-scaling works)
• Attracted enterprise customers who required multi-region deployments

Their platform is now a competitive advantage, not a liability.