Cost Sentinel

An AWS budget monitoring and alerting system. When monthly spend is forecast to exceed a configured threshold, AWS Budgets fires an alert through SNS to a Lambda function that stores the alert in S3 and updates a public dashboard. The system runs across two independent Terraform environments. Prod owns the live budget and alerting, dev runs the full stack silently as a canary.

The project started as a first real foray into cloud infrastructure and grew through five post-mortems into a production system with a six-stage CI/CD pipeline, a manual approval gate before prod, iterative security hardening, and eventually its first organic catch of unexpected spend.

The runtime path is simple by design: AWS Budgets detects a threshold crossing and publishes to a KMS-encrypted SNS topic. A Lambda function subscribes, processes the alert, writes it to a versioned S3 bucket, and updates a dashboard JSON file served by a second S3 bucket as a static website. CloudWatch Logs captures Lambda execution with KMS encryption and a 30-day retention policy.

All infrastructure is provisioned by a reusable Terraform sentinel module deployed into two independent environments. Each environment has its own KMS key, SNS topic, Lambda function, S3 buckets, and CloudWatch log group. The environments share a DynamoDB-backed remote state backend with separate state keys and locking.

AWS Budgets is account-scoped. Two budget resources in the same account would both monitor the same spend and both fire alerts on the same threshold crossing. Running a dev environment with its own budget would mean every alert arrives twice.

The solution: prod owns the single account-wide budget and is the authoritative alerting environment. Dev runs the full stack — topic, Lambda, S3, KMS — but with the budget disabled. Dev exercises the SNS-to-Lambda wiring in integration tests without generating live alerts. Stateful prod resources use prevent_destroy to guard against accidental destruction.

The email subscription on the prod SNS topic is managed manually in the Console and kept out of Terraform, so the address never appears in state. The Lambda subscription is Terraform-managed.

Six stages: Build, DeployDev, IntegrationDev, ApproveProd (manual gate), DeployProd, IntegrationProd. CodeBuild packages the Lambda function and validates Terraform on every push. The integration stage invokes Lambda with a test SNS payload, verifies the S3 write, and validates the dashboard JSON. A failure at any stage blocks promotion.

The manual approval gate before prod promotes the current dev-validated commit. Prod never receives a commit that hasn't passed integration tests in dev first. The CI/CD infrastructure itself — pipeline stages, CodeBuild projects, IAM roles — is managed by a bootstrap Terraform stack applied out-of-band, since a pipeline cannot add its own stages.

The prod cutover was designed around a no-monitoring-gap principle: prod was brought live first with dev still alerting, then dev's budget was disabled in a separate pipeline run after prod was verified. At no point was the account unmonitored.

Security hardening happened iteratively across post-mortems rather than upfront. The arc from initial deployment to the final posture covered: customer-managed KMS keys with explicit service principal conditions for Lambda, CloudWatch Logs, and SNS; least-privilege IAM roles with write permissions scoped to project-tagged resources using tag-based conditions; four separate IAM roles by responsibility (build, deploy, integration, Lambda execution) rather than a single deployment role; elimination of all wildcard write permissions; pre-commit hooks for secrets detection on every commit.

Five post-mortems document the project's development history: initial deployment challenges, KMS integration and IAM debugging, CI/CD pipeline hardening, a full security review that raised the assessed posture from 7/10 to 9/10, and the first organic alert.

The fifth post-mortem is the most operationally significant. After completing active development, Cost Sentinel transitioned to passive monitoring. A Route 53 domain registration pushed the account's monthly forecast above the $10 threshold. Cost Sentinel detected it unattended and delivered the alert. The investigation confirmed the source in under a minute. That event directly motivated standing up the production environment — the system had proven it could do its job, and it deserved a proper home separate from the dev environment used to build it.

Terraform, AWS (CodePipeline, CodeBuild, CodeConnections, Lambda, SNS, S3, KMS, Budgets, CloudWatch, DynamoDB), Python, Makefile, pre-commit.

Expected monthly cost under $6: CodePipeline (~$1), CodeBuild (~$0–2 per build minute), Lambda (free tier), S3 (negligible), KMS ($1 per key per month, two keys for dev and prod), AWS Budgets (free for the first two budgets).

github.com/dablotz/cost-sentinel — README, post-mortems, and prod cutover runbook.