AWS has released AWS Emulator, a free, offline copy of AWS that coding agents can test against on a laptop. Finally, a fake that is honest about being fake: it even ships with a folder listing what it gets wrong. That alone puts it ahead of most staging environments we have met. If you build software with agents, this is worth a closer look, because it is really a story about how agents know when their work is correct.
What AWS Emulator is
The announcement came as a guest post by Swami Sivasubramanian on Werner Vogels’ blog, All Things Distributed. The key facts:
- It covers more than 200 AWS services, built from over 400 public API models.
- All of its code was written and tested by agents, and it is public on GitHub under a public-domain licence, commit history and agent-filed issues included.
- Its yardstick is a set of “fidelity tests”: real customer scenarios that agents first tune until they pass against real AWS, so each test records how AWS actually behaves. Only then do agents build the emulator until it passes the same tests.
- Every change goes through a merge gate of automated checks, including security agents doing adversarial reviews.
- A folder called “things-that-do-not-work” lists the behaviours that don’t match real AWS yet.
The most striking example is Swami’s rebuild of a replication engine his team originally took about four months to build. With an agent and no way to check its results, he spent four nights supervising it. With a spec and a test environment, the rebuild took about two hours. His conclusion: “Software quality in the AI era depends on the guardrails that engineers design around agents.”
Why a faithful stand-in matters
An agent can only check its work against what it can see. Give it a mock that someone wrote from memory, and it will happily make the mock pass. Give it something that behaves like the real service, and it can catch its own mistakes before a customer does. The important trick in AWS Emulator is the order: test against reality first, then build the stand-in. The tests describe the real world, not what someone assumed the real world looks like.
What we learnt building Calendrz
Calendrz lives on other people’s APIs. When you’re busy in one calendar, our availability mirroring creates a marker entry in each of your other connected accounts, across Google Calendar and Microsoft 365, so nobody can book that slot. Every interesting bug we have had sits in the gap between what we expect Google or Microsoft to hand back and what they actually hand back.
Here is one from February. Markers were created perfectly, but when the original meeting moved or was cancelled, the markers stayed put. Nothing crashed and no error was logged. The update was simply skipped. The cause was two checks comparing a stored identifier with the wrong kind of value. A brand new marker has nothing stored on it yet, so the check fell through and creation worked. An existing marker, the kind real calendars are full of, always failed the check. Any test that only ever looked at fresh markers would pass every time.
Our founder, Liviu Tudor, fixed it with Claude as his pair, and the fix shipped with a written root cause analysis and updated tests. The lesson stuck: the stand-ins you test against must look like real calendars after months of use, not like the tidy objects you create in a test. Real calendars are messy. They have recurring meetings with one moved instance, time zones on some events and not others, and entries that have been edited a dozen times.
That is also how we run the codebase with agents today. The repository carries its own rules and specialist agents for the backend, the frontend and the database, plus skills for reading production logs and for merging and deploying. Every pull request has to pass the full test suite before it merges. The agents do a lot of the typing. The specs, the tests and the gates decide when the work is done.
What to take from it
We think AWS is pointing at where software engineering is heading: a “lights out” setup, where engineers write the specs and the checks, and agents write, test and ship the code. The engineer’s job moves from writing code to deciding what correct means. That only works if the thing the agent tests against tells the truth.
- Record reality before you fake it. Build your test doubles from real responses, not from the documentation or your memory of it.
- Test the old data, not just the new. Most bugs live in records that have been around for a while.
- Keep a “things that do not work” list. Knowing where your stand-in lies is half the value of having one.
- Make the gate automatic. An agent that can’t merge without passing the checks doesn’t need you to watch it for four nights.
If your own calendars are the messy, real-world kind, with several jobs, clients or side projects, connect them to Calendrz. Auto-blocking keeps every calendar showing when you’re busy, without sharing the details, and we test it against calendars that look a lot like yours 🙂







