infra-ops: one Terraform + Ansible platform for dev, staging and prod

Challenge

When I joined, each environment had been built by hand at a different time by a different person. Staging didn't match prod, IAM users had access keys older than some employees, and "deploy" meant SSH and a prayer. Every incident started with twenty minutes of discovering what was actually running.

Approach

Terraform for what exists

• Directory per environment (dev/, staging/, prod/) composing versioned modules for networking (VPC, subnets, security groups), compute (EC2, ASGs), storage (S3, RDS) and IAM (least-privilege roles).
• Remote state in S3 with DynamoDB locking, encrypted and versioned. No more "who has the state file".
• Tagging and naming conventions enforced in modules, so cost allocation works without a cleanup project.

Ansible for what runs on it

Roles for PostgreSQL/PostGIS, Nginx + Certbot, and PM2-managed Node services. Terraform outputs feed Ansible inventories, so the two never disagree about an IP. The reasoning behind that split is in Infrastructure as Code: Bridging Terraform and Ansible.

Pipelines that say no

• Every PR runs terraform plan and posts the diff as a comment.
• Auto-apply to dev, manual approval gates for staging and prod.
• OIDC: GitHub Actions assumes a short-lived AWS role per environment. There are no long-lived cloud keys anywhere in the organisation.
• A nightly drift job re-plans every environment and opens a GitHub issue when reality diverges from code.

Hardening defaults

IMDSv2 enforced, UFW and fail2ban baked into the base role, SSM Session Manager instead of SSH keys, secrets in SSM Parameter Store with the move to Vault / External Secrets Operator underway for Kubernetes workloads.

Architecture

What shipped

• One repository that provisions all three environments from the same modules.
• A CI/CD standard adopted by every service: environment-gated promotions, health-check-triggered rollbacks.
• Zero shared credentials; every access path is a role with an audit trail.
• The pattern is open-sourced as infra-ops.

Results

• 40% shorter deployment cycles across services after the pipeline standard rolled out.
• Staging finally predicts prod: the same module versions, the same roles.
• Drift is caught overnight instead of during an incident.

What I'd do differently

• Introduce policy-as-code (OPA/Conftest on the plan JSON) alongside the manual gates; humans approve too many "small" changes.
• Model the Kubernetes migration into the modules earlier so EKS, ArgoCD and the mesh land as module versions rather than a parallel effort.
• The observability side of the same platform is the lgtm-sidecar: a resource-capped observability stack for small teams story.