• Skip to content

Carlone Technology Group

Experienced leadership where execution matters most

  • Home
  • Services
  • Company
  • Contact

The Most Dangerous Word in Change Management Is “Routine”

Two outages on consecutive days, two preliminary explanations, and one word they share.

In mid-November 2026, Outlook, Teams and Copilot were disrupted worldwide for nearly a day. The next afternoon the FAA stopped departures across the country for about five hours, the longest nationwide ground stop in 25 years. Neither was an attack. Officials found no evidence of one at the FAA, and Microsoft traced its outage to its own software. Both investigations are still open, so everything below comes from preliminary accounts.

What stuck with me was the vocabulary. The Transportation Secretary described the work behind the FAA stop as routine maintenance. Microsoft’s account says the change at the root of its outage passed safe deployment and then ran for more than three months without a visible problem. Routine. Safe. Proven.

Two incidents in brief

Microsoft shipped an update to the gateway servers in front of its cloud services in July. It rolled out over several weeks and nothing happened. In November, a Teams failover pushed traffic onto the gateway that also serves Exchange Online, while roughly 14% of gateway capacity was already out of service for an unrelated reason. That combination triggered a lock contention defect in the July update. Servers stopped responding, traffic was redirected to the servers that remained, and those tripped the same defect. At the lowest point, more than 80% of the affected servers were unresponsive.

The FAA story is thinner so far. Officials say a contractor was replacing an aging copper line with fiber at a remote site when a newly installed router overloaded a critical communications system. The Wall Street Journal reports that authorities are focusing on the contractor’s work. The Washington Post reports that the agency had been warned for years about the risks of this kind of repair, and that thousands of flights were canceled. Whether the router was faulty, misconfigured, or simply too much for old equipment is the investigation’s job to say.

I am not going to claim these events share a root cause. They don’t. They share a pattern, and the pattern is the useful part.

“Routine” describes a change’s past, not its present

People call a change routine because it has gone well before. That is a statement about history. It says nothing about whether the environment the change lands in today resembles the one where it went well.

Both incidents had a gap between those two things. Microsoft’s defect sat dormant through three months of normal operation because normal operation never produced the conditions it needed. The FAA’s repair risks were, by the Post’s account, known for years, which means the work felt familiar long before it was safe. Familiarity builds confidence faster than it builds evidence.

A better test for the word: what has changed since the last time this went smoothly? Capacity, traffic mix, hardware generation, staff, vendors, the other work happening nearby. If nobody can answer, the change is familiar, not routine.

A test answers the question it was asked

Microsoft reportedly told customers that its load testing tended to focus on weaker hardware with fewer cores, while the defect behaved worse on newer machines with more of them. The testing was real and the update passed it. The test simply did not look like production.

This gap is the most common one I know of, and it rarely shows up as a failed test. It shows up as an assumption that nobody owns: this environment is representative of that one. Everyone relies on it, and no one signs for it.

Make someone sign for it. Before a consequential release, require a short written list of the ways the test environment differs from production, with a name beside each difference and a decision about whether it matters. Three honest lines beat a page of passing results.

Failures need a conjunction, and reviews check factors one at a time

Take Microsoft’s three conditions separately. A latent defect that had never misbehaved. A Teams failover that was, by one account, not particularly unusual. A 14% capacity shortfall that reportedly sat within the service’s headroom. Any risk review would have waved each of them through, and each was acceptable alone.

The failure needed all three at once. Reviews are built to evaluate factors; incidents are built from combinations. So add one question to the review: what else has to be true for this to hurt us? Then go and check whether those things are already true today. Every known issue the organization has accepted, whether a degraded node, an open ticket, or a workaround, is a standing input to the next change. Most teams file them as separate problems and never ask how they interact.

Recovery is usually designed for the last failure

Microsoft’s automated recovery was not built for a large share of servers going unresponsive at once. Restarting them did not help, because they were overwhelmed again as soon as they came back. Shifting load to healthy servers, which is exactly what a resilient design is supposed to do, fed the defect more victims. Engineers ended up building a fix and applying it to server groups gradually.

The lesson is not that automation is dangerous. Recovery mechanisms get tested against the failures people expected, such as one node, one zone, one bad deploy. The failures that become headlines are the ones where recovery itself changes the load, and the plan to undo the change assumes you are rolling back into the conditions you left.

Rehearse recovery under degradation, not at full health. Take a fraction of capacity out first, then ask what the rollback does to what remains. If the honest answer is that nobody knows, say so in the readiness review, not in the postmortem.

The person doing the work is rarely the person who owns the outcome

Whatever the final FAA findings conclude, the shape of the situation is familiar: one party performs the installation, another owns the system it plugs into, and a third has the authority to halt the work or declare it done. In many organizations those are three different parties with three different definitions of success. The installer’s definition is that the equipment is in. The operator’s is that the service stays up.

When a change is outsourced or handed across a team boundary, the task transfers cleanly and the accountability for the outcome does not. Assign an owner for the operational result of each consequential change, name the person who can stop it, and write down what that person must see in order to do so. Stop authority that depends on someone finding out afterward is not stop authority.

Five questions before anyone calls a change routine

None of this calls for a heavier process. It calls for five questions asked at the point where a change gets its label.

  1. What is different about the environment since this last went well?
  2. In what three ways does our testing differ from production, and who accepted each?
  3. What is already degraded right now, and what happens if it gets worse mid-change?
  4. If this fails in the way we least expect, what is the first recovery action, and could it make things worse?
  5. Who owns the result, who can stop it, and what will they see in time to do it?

If the answers come quickly and specifically, proceed and move fast. If they come slowly or vaguely, the change was never routine. You just hadn’t noticed yet.

Classify by consequence, then let the paperwork follow

The usual reaction to an outage is more governance: more approvals, more forms, more sign-offs on everything. That slows the changes that were safe and leaves the dangerous ones exactly as understood as before. A rule that applies equally to a typo fix and a core network change will be ignored by the people who live with both.

Split the vocabulary instead. “Routine” should mean two claims, each with evidence: this change is well understood, and its failure would be cheap. A change that is well understood but expensive to fail deserves the five questions. A change that is poorly understood deserves them even if failure looks cheap, because cheap is a guess until you have mapped what depends on it.

My position is simple. Retire “routine” as a standalone word in change reviews. Replace it with those two claims and make the team prove each one. The next outage will arrive wearing the label “routine” on its work order, and the organizations that survive it gracefully will be the ones that stopped trusting the label.

Sources

  • Microsoft’s preliminary post-incident account, as reported by The Register, “Microsoft blames monster email outage on software defect introduced in July fix,” Nov. 16, 2026
  • ZDNET, Microsoft outage coverage, Nov. 16, 2026
  • The Wall Street Journal, “Outage That Swamped Air Travel Blamed on ‘Routine’ Work to Replace Copper Wire”
  • The Washington Post, “FAA was warned for years about repair risks leading to nationwide ground stop,” Nov. 14, 2026
  • Associated Press, “Nationwide FAA Ground Stop Blamed on Contractor Replacing Copper Wire”

Let’s Talk

Most of my work begins through existing relationships. Former colleagues and clients reach out when something familiar shows up again and they want a trusted partner to help navigate it.

If this resonates and you are thinking about what comes next, a conversation is usually the best place to start.

Let's talk
  • Home
  • Services
  • Company
  • Contact

Copyright © 2026 · Carlone Technology Group