End-to-end tests have a reputation problem.<\/p>\n
They’re slow, they’re flaky, and the moment you wire in a real cloud dependency, your CI builds become an exercise in waiting and retrying.\nMost teams I’ve talked to handle this one of two ways: they push everything down into integration tests and call it good (and lose confidence that the full system actually composes),\nor they spin up shared “dev” environments and run E2E against those (and accept the cross-talk, the flake, and the “who broke staging” group chat messages as the cost of doing business).<\/p>\n
We took a third path, and it’s paid off in a way I didn’t fully expect.<\/p>\n
Azure Chaos Studio<\/a> lets customers proactively break their own systems in safe, controlled ways \u2014 to validate that their resilience strategies actually hold up under real failure.\nUnder the hood we have four independently-shipped services: a control plane, an execution plane, a fault-execution plugin host, and a data plane.\nEach one is its own deployable, with its own dependencies, its own long-running-operation (LRO) semantics, and its own surface to test.<\/p>\n We’re heading toward GA of our V2 platform later this year, and the velocity bar has gone up \u2014 particularly as the team has leaned heavily into agent-assisted development. We needed test coverage we could actually trust before we let agents make non-trivial changes.<\/p>\n For a long time, our end-to-end tests ran against shared environments. That’s a familiar story for anyone working on real cloud services:<\/p>\n We had a solid integration test layer too. Each service has its own But that’s exactly the limit. Integration tests of that shape can’t tell you whether the four services compose correctly \u2014 whether the LRO state machine in the control plane actually agrees with what the execution plane is polling for,\nwhether the auth flow holds across hops, whether a retry on one service surfaces sensibly two services downstream.\nThat’s the layer where most of our real bugs live, and that’s the layer we didn’t have a trustworthy story for.<\/p>\n About a year ago I read this writeup on hermetic, ephemeral test environments<\/a>,\nand it stuck with me. The core idea: every test gets its own clean, isolated environment, brought up just for that test, with all dependencies running locally.\nNo shared state. No flake from neighbors. Failures are reproducible by construction.<\/p>\n It’s a great vision. The hard part is getting there in a real system with deep cloud dependencies.<\/p>\n Then a conversation with one of the .NET teams connected the dots for me: The hermetic model has three pieces:<\/p>\n In Aspire, that’s all expressed as resources on the AppHost. The same model that drives our local dev loop drives our tests:<\/p>\n The test fixture brings the whole graph up, hands the test a typed That last We’re up to roughly 90 hermetic tests<\/strong>, and they cover meaningfully more than the original “do the services start up” check. The interesting ones are the scenario tests \u2014 end-to-end fault-injection flows driven through the real service graph:<\/p>\n Each of those used to be a careful manual exercise in a shared environment. Now they run on every PR, in parallel, with no cross-talk.<\/p>\n Here’s the part I genuinely didn’t see coming.<\/p>\n When the team started leaning into agent-assisted development in earnest, this test suite quietly became our trust anchor. An agent can propose a meaningful refactor or a non-trivial feature, and we have a real signal \u2014 not just “the unit tests still pass” \u2014 that the change actually composes across services.<\/p>\n Agents don’t have to be perfect. They have to be checkable<\/strong>.<\/p>\n<\/blockquote>\n That distinction is the whole game. Perfection isn’t a realistic bar for any contributor, human or otherwise \u2014 and chasing it tends to slow the team down more than it helps. Checkability is. If the system can tell you, quickly and unambiguously, whether a proposed change holds up end to end, you can move fast and stay honest about it.<\/p>\n Hermetic end-to-end tests turn out to be one of the highest-leverage checks you can give an agent, because:<\/p>\n This isn’t a hypothetical. The last few months of our V2 push would have been a much scarier ride without it.<\/p>\n If you’re considering something similar, a couple of things saved us time:<\/p>\n Aspire didn’t just make hermetic testing possible for us \u2014 it made it the path of least resistance.<\/p>\nThe problem with our previous setup<\/h2>\n
\n
WebApplicationFactory<\/code>-style suite that spins the API up in-process, runs real HTTP through it, and stubs out the data and event collaborators with mocks. Those tests are fast, deterministic, and great at catching regressions inside a single service.<\/p>\nHermetic, ephemeral, per-test environments<\/h2>\n
Aspire.Hosting.Testing<\/code><\/a> was already most of the way there. Aspire already knows how to bring up your full service graph as a process tree. With the testing package, you can do it programmatically \u2014 from inside an xUnit fixture, on every PR, in your CI pipeline.<\/p>\nWhat the setup looks like<\/h2>\n
\n
AddAzureKeyVault(...).RunAsEmulator()<\/code> flow.<\/li>\nvar builder = DistributedApplication.CreateBuilder(args);\n\n\/\/ Emulated dependencies\nvar cosmos = builder.AddAzureCosmosDB(\"cosmos\").RunAsEmulator();\nvar storage = builder.AddAzureStorage(\"storage\").RunAsEmulator();\n\n\/\/ Key Vault emulator comes from the community package\n\/\/ AzureKeyVaultEmulator.Aspire.Hosting (james-gould\/azure-keyvault-emulator)\nvar keyVault = builder.AddAzureKeyVault(\"kv\").RunAsEmulator();\n\n\/\/ Stub for the one dependency we can't emulate\nvar authStub = builder.AddContainer(\"auth-stub\", \"wiremock\/wiremock\")\n .WithHttpEndpoint(targetPort: 8080, name: \"http\");\n\n\/\/ Our actual services\nvar controlPlane = builder.AddProject<Projects.ControlPlane>(\"control-plane\")\n .WithReference(cosmos)\n .WithReference(storage)\n .WithReference(keyVault)\n .WithReference(authStub.GetEndpoint(\"http\"));\n\nvar executionPlane = builder.AddProject<Projects.ExecutionPlane>(\"execution-plane\")\n .WithReference(controlPlane);<\/code><\/pre>\nHttpClient<\/code> for each service, and tears it all down when the test finishes:<\/p>\npublic class HermeticFixture : IAsyncLifetime\n{\n private DistributedApplication _app = null!;\n\n public HttpClient ControlPlaneClient { get; private set; } = null!;\n\n public async Task InitializeAsync()\n {\n var appHost = await DistributedApplicationTestingBuilder\n .CreateAsync<Projects.ChaosStudio_AppHost>();\n\n _app = await appHost.BuildAsync();\n await _app.StartAsync();\n\n ControlPlaneClient = _app.CreateHttpClient(\"control-plane\");\n await _app.ResourceNotifications\n .WaitForResourceHealthyAsync(\"control-plane\");\n }\n\n public async Task DisposeAsync() => await _app.DisposeAsync();\n}<\/code><\/pre>\nWaitForResourceHealthyAsync<\/code> call is one of those quietly important details. Tests don’t run until the service graph is actually ready, and “ready” means real health checks \u2014 not arbitrary sleeps that drift and flake.<\/p>\nWhat we’re actually testing<\/h2>\n
\n
The agent payoff<\/h2>\n
\n
\n
A few practical notes<\/h2>\n
\n
Closing<\/h2>\n