What Microsoft's Outage Says About Your Performance Lab

On November 12, Exchange Online stopped answering for a large part of the world. Outlook went with it, along with Teams calendars, parts of SharePoint, and Copilot. For close to a full day, a lot of businesses ran on phone calls and hallway conversations. Some of us remember when that was just called Tuesday.

The easy takeaway is "they should have load tested." I don't think that's the lesson. Microsoft load tests, and runs some of the most sophisticated performance labs anywhere. Their own preliminary post-incident report explains, in plain terms, why the lab didn't catch this one.

That explanation is worth reading closely, because many teams I talk to have the same gap.
‍

What Microsoft says happened

The preliminary report (the final is due November 20, so details may shift) describes conditions that lined up:

  • A latent defect. A Windows fix built specifically for Microsoft 365's networking gateway servers rolled out in July and ran cleanly for more than three months. Under certain load, lock contention inside an internal Windows process exhausted CPU and memory, and the servers stopped responding.
  • A traffic shift. About 15 hours before the outage, Teams failed over from its alternate gateway to the primary gateway that also serves Exchange Online. Load went up, and the shape of the traffic changed.
  • Reduced capacity. Nearly 14% of gateway capacity was already out of service from an unrelated issue the day before. Microsoft says the remaining headroom was fine on its own.

None of these was enough by itself, but together they hit the defect. Then the system's own resilience made it worse. Traffic from failed servers was routed to healthy ones, which pushed them into the same condition. In under two hours the defect had spread across the fleet, and at the low point more than 80% of servers were unresponsive. Servers brought back into service were buried by queued traffic and fell over again. Standard rollback couldn't help quickly, because the change was three months old.

On the question of why testing missed it, Microsoft named two things: the traffic mix and the hardware SKUs in their labs. The problem got worse on machines with more cores and newer hardware.
‍

"Production-like" is a set of conditions

When people hear "test in a production-like environment," they usually picture a full-scale replica. That's expensive, most organizations will never build one, and it isn't what would have caught this.

The lab was missing fidelity on a few specific dimensions. That's where I'd put the money.

Hardware parity

Lock contention is the kind of defect that hides on smaller boxes. More cores means more threads competing for the same lock, and the latency curve can go from flat to vertical with little warning. If your performance environment runs on last generation's instance types, or on smaller VMs "because it's just test," you are measuring a different system.

Server count matters less than server type. One node on the production SKU can tell you more about per-node saturation than twenty nodes on the wrong one.

Workload model fidelity

In my experience, most load tests replay a traffic mix someone defined early in the project and rarely revisited. Production doesn't hold still. Here, a failover consolidated Teams and Exchange traffic onto shared infrastructure. That changed both the size and the shape of the load those servers handled.

Build the workload model from production telemetry and refresh it on a schedule. Include the conditions you would see after your own failover and consolidation events, too. If you know a failover moves workload X onto cluster Y, then "cluster Y with X added" is a test condition.

Degraded-state performance

This is the one I rarely see tested. Nearly every performance test I've reviewed assumes full capacity. Production rarely has it: nodes are out for patching, a zone is impaired, a dependency is slow.

Ask what happens at peak load with 15% of capacity gone. Then ask what your failover and health-check logic does when the remaining nodes start to struggle. In this incident, routing traffic away from sick servers is what spread the failure, and Microsoft noted that the bulkheads between regional partitions didn't contain it. A system that redistributes load can turn a local defect into a global one. You only learn whether yours does by testing the redistribution under stress.

Recovery under load

Restarting a server into a backlog is its own load test. Queued requests, client retries, and reconnect storms all arrive at once. If your recovery plan assumes a returning node will see normal traffic, that plan hasn't been tested. Bring a node back cold into a backlog and see whether it stays up, and check whether you can admit traffic gradually.

‍


Three months of clean telemetry

The detail I keep coming back to is the three months. The fix shipped in July and behaved. Every check before and after deployment said it was healthy.

I wrote recently about how AI quality degrades silently in production. Performance risk works the same way. A latent defect sits in production waiting for its conditions, and those conditions arrive on their own schedule. No lab will find all of them before release.

What you can do is watch the leading indicators. Lock wait time, CPU per request, and memory growth relative to throughput all move before a server stops answering. A fleet where per-request cost climbs with core count is telling you something long before the day the traffic mix shifts unexpectedly. Feed those signals back into the workload model and the test conditions. That loop is what turns a near-miss into a test case instead of a production incident.


Where to start

You don't need to fix all of this at once. Pick the gap most likely to bite you:

  1. Compare your performance environment's instance types against production. If they don't match, fix that first. It's usually the cheapest change with the biggest payoff.
  2. Rebuild your workload model from the last 30 days of production traffic, and add one post-failover mix.
  3. Rerun your peak test with 15% of capacity removed, and watch what your load balancer and health checks do.
  4. Add one recovery test: bring a failed node back into a backlog and see whether it holds.
  5. Pick two saturation signals and put them on a dashboard someone reviews every week.

Microsoft deserves credit for publishing this level of detail within days. Most organizations never show their work like this, and the rest of us get to learn from it.

Matching production's size is a budget question. Matching its conditions is an engineering decision and you can make it now.

‍

‍

About the author

Lee Barnes
Chief Quality Officer at Forte Group

You may also like

Transform AI into a Scalable Delivery Capability

83% faster delivery. Under 10% rework. See exactly how Xceptor got there.