01Introduction
Taming 12 EKS Clusters with GitOps at Scale
02Problem Statement
- E-commerce platform: 12 EKS clusters (prod, staging, dev) across 3 AWS regions
- Deployments done manually via kubectl; 6-hour release cycles; frequent rollback failures
- Challenge: implement GitOps without disrupting Black Friday traffic peak in 8 weeks
- Config drift between clusters causing environment-specific bugs hitting production
- Goal: sub-30-minute deployments; zero-touch cluster config management
03Proposed Solution
Phase 1
Flux + Atlantis GitOps Control Plane
- Flux CD deployed as GitOps operator across all 12 clusters
- Git repo structure: cluster-specific overlays with Kustomize; base configs shared
- Flux watches Git; any merge to main triggers reconciliation within 5 minutes
- Atlantis for Terraform: all EKS cluster changes via PR + plan approval workflow
- Sealed Secrets for encrypted secret management committed to Git safely
Phase 2
Progressive Delivery & Rollback Safety
- Flagger deployed for canary deployments: 10% to 50% to 100% traffic shifting
- Automated rollback triggered on error rate >1% or P99 latency >500ms
- Cluster bootstrap scripts: new cluster fully configured from Git in under 20 minutes
- Network policies enforced via Flux; no ad-hoc kubectl apply permitted in prod
- Drift detection: Flux alerts on any manual change to cluster resources
04Technologies
- GitOps
- Flux CDFlagger (canary)KustomizeSealed Secrets
- IaC
- Terraform + Atlantis PR workflow
- Platform
- Amazon EKSAWS ALB IngressExternalDNSCert-Manager
- Monitoring
- Datadog (deployment events)CloudWatch Container Insights
- Security
- OPA Gatekeeper policies enforced via Flux