
On November 12, Exchange Online stopped answering for a large part of the world. Outlook went with it, along with Teams calendars, parts of SharePoint, and Copilot. For close to a full day, a lot of businesses ran on phone calls and hallway conversations. Some of us remember when that was just called Tuesday.
The easy takeaway is "they should have load tested." I don't think that's the lesson. Microsoft load tests, and runs some of the most sophisticated performance labs anywhere. Their own preliminary post-incident report explains, in plain terms, why the lab didn't catch this one.
That explanation is worth reading closely, because many teams I talk to have the same gap.
The preliminary report (the final is due November 20, so details may shift) describes conditions that lined up:
None of these was enough by itself, but together they hit the defect. Then the system's own resilience made it worse. Traffic from failed servers was routed to healthy ones, which pushed them into the same condition. In under two hours the defect had spread across the fleet, and at the low point more than 80% of servers were unresponsive. Servers brought back into service were buried by queued traffic and fell over again. Standard rollback couldn't help quickly, because the change was three months old.
On the question of why testing missed it, Microsoft named two things: the traffic mix and the hardware SKUs in their labs. The problem got worse on machines with more cores and newer hardware.
When people hear "test in a production-like environment," they usually picture a full-scale replica. That's expensive, most organizations will never build one, and it isn't what would have caught this.
The lab was missing fidelity on a few specific dimensions. That's where I'd put the money.
Lock contention is the kind of defect that hides on smaller boxes. More cores means more threads competing for the same lock, and the latency curve can go from flat to vertical with little warning. If your performance environment runs on last generation's instance types, or on smaller VMs "because it's just test," you are measuring a different system.
Server count matters less than server type. One node on the production SKU can tell you more about per-node saturation than twenty nodes on the wrong one.
In my experience, most load tests replay a traffic mix someone defined early in the project and rarely revisited. Production doesn't hold still. Here, a failover consolidated Teams and Exchange traffic onto shared infrastructure. That changed both the size and the shape of the load those servers handled.
Build the workload model from production telemetry and refresh it on a schedule. Include the conditions you would see after your own failover and consolidation events, too. If you know a failover moves workload X onto cluster Y, then "cluster Y with X added" is a test condition.
This is the one I rarely see tested. Nearly every performance test I've reviewed assumes full capacity. Production rarely has it: nodes are out for patching, a zone is impaired, a dependency is slow.
Ask what happens at peak load with 15% of capacity gone. Then ask what your failover and health-check logic does when the remaining nodes start to struggle. In this incident, routing traffic away from sick servers is what spread the failure, and Microsoft noted that the bulkheads between regional partitions didn't contain it. A system that redistributes load can turn a local defect into a global one. You only learn whether yours does by testing the redistribution under stress.
Restarting a server into a backlog is its own load test. Queued requests, client retries, and reconnect storms all arrive at once. If your recovery plan assumes a returning node will see normal traffic, that plan hasn't been tested. Bring a node back cold into a backlog and see whether it stays up, and check whether you can admit traffic gradually.
The detail I keep coming back to is the three months. The fix shipped in July and behaved. Every check before and after deployment said it was healthy.
I wrote recently about how AI quality degrades silently in production. Performance risk works the same way. A latent defect sits in production waiting for its conditions, and those conditions arrive on their own schedule. No lab will find all of them before release.
What you can do is watch the leading indicators. Lock wait time, CPU per request, and memory growth relative to throughput all move before a server stops answering. A fleet where per-request cost climbs with core count is telling you something long before the day the traffic mix shifts unexpectedly. Feed those signals back into the workload model and the test conditions. That loop is what turns a near-miss into a test case instead of a production incident.
You don't need to fix all of this at once. Pick the gap most likely to bite you:
Microsoft deserves credit for publishing this level of detail within days. Most organizations never show their work like this, and the rest of us get to learn from it.
Matching production's size is a budget question. Matching its conditions is an engineering decision and you can make it now.