TL;DR
Our single staging environment deliberately mirrors production: every change deploys there after merge, behind a feature flag. That meant features landed on main before they'd ever been live-tested. And as cloud coding agents take on more implementation work, humans increasingly need to review a change running, not just its diff. So we built ephemeral environments: every pull request gets its own isolated, full-stack environment (all ~26 services plus its own copy of the database), ready about 15 minutes after the PR asks for one, and destroyed automatically when it closes. Our first design, which deployed only changed services and routed requests with headers, died on our job queues. The design that shipped deploys the full stack per PR and isolates state at the environment boundary, with database isolation powered by Aurora copy-on-write clones.
The pain
Nirvana runs a single staging environment; there is no dev or QA tier below it. That's by design: staging deliberately mirrors production. Every service deploys there after merge, and every change is supposed to ship dark behind a feature flag until it's ready.
That discipline keeps production safe, but it quietly inverts the development loop. The first time a feature runs in a real environment is after it's already on main. Some changes can be exercised locally, but most flows span too many services and too much data for a laptop. Code review happens pre-merge; live testing happens post-merge, so half-proven features accumulate on the main branch, and anything that misbehaves gets fixed (or flagged off) after the fact. QA and product can't click through a branch before it merges, and cross-service changes (a proto change touching two services, say) have no pre-merge test surface at all.
Then a second force made this urgent. Cloud coding agents are taking on more and more implementation work, and an agent-authored PR raises the bar for review: reading the diff isn't enough. The human in the loop needs to see the change running: click through the UI, exercise the flow, watch the data move. That requires somewhere agents can deploy their work automatically, which a single, post-merge, mirror-of-production staging can never be.
The goal, then: every pull request, human- or agent-authored, gets its own isolated, short-lived environment, deployed before merge and torn down when the PR closes, so a change can be live-tested before it ever touches main.
The design that didn't survive contact
Our first instinct was the elegant one: deploy only the services a PR changed, and share everything else with staging. Every request would carry an environment ID header (X-ENV-ID: pr-123), and a routing layer of Envoy sidecars would read it and steer the request to the PR's copy of a changed service, falling through to the shared staging stack for everything else. Minimal compute, minimal duplication: a "sparse overlay" on top of staging.
For this to isolate anything, though, one invariant has to hold everywhere: every hop of every request must carry that header forward. A synchronous gRPC chain can be taught to do that. But the deeper we traced our real request flows, the more places we found where the request, and with it the environment identity, simply ends:
- Background jobs. Almost half of our deployments include a job processor: a service that picks tasks off a shared queue and runs them. A request creates the task, but by the time a worker picks it up, that request is long over, and the header is gone with it. A staging worker can run a PR environment's job (or the reverse) and quietly write the results into the wrong database.
- Everything else asynchronous. Cron jobs start themselves. Queue consumers wake on messages, not requests. Webhooks arrive from vendors who have never heard of our header. Environment identity dies at every one of these boundaries.
Worse, the failure mode is silent. A service that drops the header doesn't throw an error; it just leaks traffic into the wrong environment, and everything looks like it's working. Making the overlay trustworthy would have meant re-architecting how context propagates across every async boundary in the system, and then continuously monitoring request flows end-to-end just to prove the isolation still held. The off-the-shelf tooling that does this is Kubernetes-only; we run on ECS.
We didn't want isolation we'd have to keep proving. So we made the boring choice.
The pivot: full stack, every PR
Instead of deploying the changed services and routing around the rest, we deploy all ~26 services for every PR, but only build images for the services the PR actually changed. Unchanged services reuse the latest image already in our staging registry, resolved at deploy time.
Isolation stops being a routing problem and becomes a naming problem, which is a much better problem to have. Each environment gets its own service-discovery namespace, and a single environment variable rewrites every gRPC address at config-load time, with no per-service code changes.
The deploy itself is three separate Terraform stacks per environment (base networking/DNS, unchanged services, changed services) with separate state, so CI can apply them in parallel: while images for changed services build, the unchanged fleet is already deploying. If any step fails, a rollback job destroys all three stacks automatically. End to end, an environment is ready about 15 minutes after the PR asks for one: the image build and the database provisioning run side by side and dominate the wall clock.
Which brings us to the part of the problem we find genuinely interesting.
The interesting part: a database per PR
A full stack of stateless services is table stakes. The thing that makes a PR environment trustworthy is stateful isolation: if two PRs share a database, one PR's schema migration or test data corrupts the other's run, and neither engineer can trust what they're seeing. We wanted every environment to boot against its own copy of the staging database, with real, representative data in it.
There are plenty of ways to get a database copy: restoring from snapshots, serverless Postgres providers with instant branching, managed branching databases. We landed on Aurora's native copy-on-write cloning, for boring reasons: it stays entirely inside AWS, introduces no new vendor, and required zero application code changes.
Copy-on-write, in one paragraph
When you clone an Aurora cluster, no data is copied. The clone points at the same underlying storage pages as its parent; only when either side writes a page does Aurora copy it and give each cluster its own version. That yields the two properties that matter for ephemeral environments: clone time is independent of database size, and you pay only for the pages an environment actually changes, which for a short-lived PR environment is next to nothing.
Plugging it in
This was the payoff for boring architecture. Every service already reads its database host from configuration injected at deploy time, so pointing an entire environment at its clone is a deploy-time change, not a code change. The clone source is the same database our schema-migration flow keeps current, so a clone taken at any moment has today's schema and today's data. There is no drift between "works in the PR environment" and "works on staging."
Cloning does have platform limits on the number of clones a cluster can support, but because our environments are short-lived, we stay well within those limits.
The rest of the lifecycle
Environments that are cheap to create must be cheap to destroy, or they stop being ephemeral. Every environment carries a TTL: a scheduled EventBridge rule fires the destroy workflow when the TTL expires, and closing or merging the PR destroys it immediately. The destroy tears down the service stacks and the database clone in parallel. Nothing about an environment outlives the PR that asked for it.
What we got
- Pre-merge, end-to-end validation. A reviewer can click a URL and exercise the actual change, against real services and a real, isolated database, before approving. Features get live-tested before they land on
main, not after. - Review the behavior, not just the diff. This is what makes agent-authored PRs reviewable at scale: the agent deploys its change, and the human judges it running.
- Writes are provably isolated. Rows created in an environment's clone never appear in staging, and vice versa. Schema-changing PRs, previously the scariest category, are now the boring category.
- Staging stopped being a contended resource. A broken branch breaks one environment with one victim, who is also the author.
- The unit economics work. An environment costs a few dollars of compute and a negligible amount of database storage for its lifetime: trivially cheap against the engineer-hours a single staging breakage used to burn.
What we learned
Route around state, and state wins. Our sparse-overlay design failed because job queues, crons, and webhooks don't carry HTTP headers. Any isolation scheme that depends on every hop cooperating will be defeated by the hop you forgot. Isolating at the environment boundary (separate namespace, separate database) is cruder and heavier, and it actually holds.
Buy the mechanism, not the benchmark. Some branching technologies create a database branch in under a second; ours takes minutes. It didn't matter, because the clone runs in parallel with the image build, and our actual requirement was "a trustworthy database per PR with zero code changes." Optimize the requirement, not the leaderboard.
Boring plumbing compounds. The whole database story plugs in through one environment variable that already existed. Years of resisting per-service configuration cleverness paid for itself in one afternoon.
What's next
The stateless tier and the primary database are isolated, and environments can opt into a mocking layer for external vendor APIs, so flows that touch third parties can be exercised end-to-end without real-world side effects. The frontier is what's left of the stateful surface: per-environment isolation for queues and object storage. We'll write about those as they land.
If shared-staging pain sounds familiar, hopefully these trade-offs save you a whiteboard session or two.
%201.jpg)
