Last Thursday, Microsoft 365 Exchange Online and related apps like Outlook and Teams went down in at least 74 countries. The next day the U.S. Federal Aviation Administration ordered a full nationwide groundstop. Every airport in the United States stopped sending planes into the sky, and those already in the air had to come back down. On Monday, Chase Bank customers couldn’t get into their accounts for the better part of the day. Then United Airlines suffered an outage. And on Tuesday, the U.S. National Security Agency’s classified communications network went down.
Social media is flooded with conspiracy theories. Talking heads on television and YouTube are blaming everything from nefarious foreign governments to rogue AI agents. For a sober and sobering perspective on this, I turned to former IEEE Spectrum contributing editor Robert N. Charette. He spent the last two decades meticulously analyzing software and systems failures around the globe. No one is better equipped to explain the whys and hows of this parade of horribles to a world waking up to the fact that the infrastructure that undergirds modern life is being held together with the digital equivalent of duct tape and chewing gum.
Is this just a bad couple of weeks or is it the new normal?
Bob Charette has tracked and diagnosed software and system problems for decades. Robert N. Charette
Robert Charette: Oh, I think it’s a bad week, but at the same time, it’s also the new normal. What I mean by that is that over the last 20 years of reporting for Spectrum, I have documented hundreds of outages over the years, and those are just the ones that come to the surface. In a typical week globally, there are hundreds of these glitches, but they’re just not reported or if they are, they only make the local news.
But these latest events are piled together and so everybody’s super-sensitive to them. And they are not minor. Friday’s nationwide groundstop was only the second since 9/11. And this current spate of failures actually goes back a few weeks. On October 28, we had the train crash in Spain because of a safety system that went down. Again what occurred there wasn’t uncommon. To my mind, none of these are uncommon. What is uncommon is that they’re clustered together. If you go back to last October and November, we had a cluster of outages as well. So this isn’t like the first time this has happened. And it’s not going to be the last time, either.
The FAA Groundstop
Let’s break these down and look at some of these incidents in depth, starting with the groundstop on Friday. In his initial assessment, Transportation Secretary Sean Duffy said outdated communications equipment had been overloaded with data and went down. He later described a newly installed router that was sending a flood of connection attempts every few seconds, swamping other routers. That new router was being installed by contractor L3Harris, which was replacing copper wire with fiber at a facility in southern Arkansas. What was your first reaction when you heard about this situation?
Charette: That somebody didn’t do their homework. If you’re taking a look at what happened with the network, they were replacing old Cisco routers with new Juniper routers. The old Cisco routers have been around for about 20 years, I think. They can’t handle as much traffic as a new router can, especially with fiber optics. So they were updating their routers and they were going to be updating thousands of these things throughout the network which is run by L3Harris, right? It’s not run by the FAA. It’s contracted out.

Putting my engineering hat on, I believe that it’s probably an old router that had some peculiar fixes to actually make it work a long time ago. And so when they brought the new router in with the configuration that they had been using elsewhere—because they’ve 400 installed so far without a problem, with 69 operational—when they installed this one, there was something unique about it and it crashed the system. And so the crashing of the system was actually crashing the old network, the old Cisco routers, not the new ones.
I believe that they’re going to find out just like in 2009 when this sort of thing happened before is that they didn’t do the right configuration management. They didn’t have a full understanding of the network and they didn’t understand that particular set of circumstances at that particular location. The interesting thing to me is that the Arkansas location, which was the center point of the problem, and another router in Amarillo, Texas started pinging back and forth with each other. I’m curious about how that actually happened as well. Because the one in Amarillo was installed, I think, three weeks ago without a problem. It’s always something weird that nobody thought about.
So does that indicate a contractor oversight issue, like people at the FAA not keeping tabs on L3Harris?
Charette: It’s partly an issue with the FAA. It’s partly an issue with L3Harris. It’s partly a consequence of the complexity of these systems that we have. Expecting every Cisco router you’re replacing to have exactly the same specs is asking a heck of a lot. You’re making a lot of assumptions. And as I wrote in my very last article before I retired from Spectrum, assumptions you make are risks you take. And I think they made a lot of assumptions about how the old routers were working and how they could configure them and do a plug and replace. That doesn’t happen. And you know, back 25 or 30 years ago, there would have been a lot more care in the test bed.
I’m also interested in what happened at the L3Harris test bed at the simulation-stimulation lab that they used in Melbourne, Florida, and to see how robust it was to deal with these types of problems. But that said, it could have just been somebody just didn’t do the update, like they didn’t do it in 2009. Somebody turned something off and was supposed to turn it back on and they didn’t.
The FAA recently expanded an AI-based traffic tool called SMART to some of its facilities and people on social media pointed at that as the culprit. The FAA and the tool’s developer both denied it had anything to do with the outage. This raises a broader question about the impact of AI on software maintenance. Is what we’re seeing a harbinger of things to come?
Charette: The pessimist in me says yes because if you take a look at where AI is being used and how it’s being paid for within organizations and within government, they’re taking it not out of new capital spend, but out of legacy systems’ operational maintenance. So can I point to this incident being a result of that? I doubt it. But I think what you will likely see is that there’s going to be this giant race between deploying AI—which hopefully will increase network and system resilience—versus stealing from the money you need to keep the lights on in these legacy systems. Because if we leave aside the AI component for the moment and just look at the FAA network, this is a classic case where you’re putting in new hardware that has to communicate with legacy systems and it’s that physical hardware and the software routing which has problems.
Now if you take AI and bolt that onto legacy systems you probably don’t know very well, then you’re going to start seeing problems. It would not surprise me that we’re going to see an increase in failures as we start to move more AI into these systems. One of my old risk maxims is that the risk exists at the interfaces and we don’t know these interfaces very well at all.
There’s a big gap between what people know about their architectures and what they should know about their architectures.
Let’s pivot to the Microsoft outage. On Thursday, November 12th, Microsoft 365’s Exchange Online and related apps like Outlook, Teams, and Copilot stopped working, and they stayed broken for roughly a day. The website Downdetector showed at least 74 countries affected. Microsoft said the failure wasn’t a cyber attack and wasn’t caused by AI agents, but rather a latent defect in the networking gateway to these services. Under certain load conditions, servers became unresponsive, in part because of an earlier network capacity upgrade and a recent network change. Why do crashes like this keep getting through? And how does this relate to the interface issue?
Charette: These are incredibly complex systems and there is probably no one who knows very much about any part of the system in any detail. And so when the update comes, you have an idea of what it’s actually doing but you probably don’t know all the different parameters of what it could do. You have to ask yourself what do I know? Then you have to ask yourself what should I know? And then you have to ask how do I manage the gap? There’s a big gap between what people know about their architectures and what they should know about their architectures, especially when they’re updating them. When you do an update, you have to be very cognizant of those risks.
I don’t think that this is standard practice. I may be wrong. When I talked to legacy maintainers for my “Inside the Hidden World of Legacy IT Systems” article there wasn’t a standard practice of saying, “okay, what happens if everything goes off rails?”

Normally, we design as much as we can. We test where we can and then we hope we can recover as quickly as possible because there isn’t any way to really understand everything that might occur. As Microsoft has found out numerous times doing normal Windows upgrades, they don’t know everything that happens in their systems and there’re so many different configurations that they need to worry about. If you actually take a look at the Outlook error, it affected a group, but it didn’t affect everybody. So, again, how you execute and manage configurations matters a hell of a lot. You need to be prepared for what’s going to occur and when it’s going to occur and who might get whacked and who won’t.
What would you ask of a company like Microsoft after an event like that?
Charette: They already know what they need to do. I would ask them just to do it. You know, it’s what I’ve written about for 20 years in Spectrum: We know how to protect our systems. We just don’t do it. We know how to make sure that we protect our systems. We know how to develop these things correctly. We’re not going to be perfect, but we know that these dangers are out there.
It’s not just this year. Last October, Azure had a problem, same type of deal. I don’t know what you can tell a company who refuses to either learn or decides that it’s worth the risk to deploy these things versus spending the money to keep bad things from happening. It has to be an economic decision. It’s not a technical decision.
JPMorgan Chase, United Airlines, and Hospital Glitches
Now on to the lightning round, Bob. We’re going to fire through three more incidents to get your take. Let’s start with the outage at JP Morgan Chase on Monday that lasted roughly four and a half hours. Chase called it a technical issue and no detailed explanation has appeared. What’s your read?
Charette: Probably just another update error. If you look at the UK for the last 15 years, you see they have online banking outages virtually every month. The government had to threaten them to upgrade their IT to keep that from happening. If there hadn’t been all of these failures this week, nobody would have even noticed.

What about United Airlines, which reportedly had a problem with its weight and balance system that delayed hundreds of flights for about two hours on Monday? United said it wasn’t a cyber incident.
Charette: From what I gathered it was an update to their system. Same thing happened to Alaska Airlines in 2024, and these types of failures happen to other airlines routinely. Glitches that occur with airlines or at airports happen almost once a month. That’s been the case for years. If it wasn’t this week, it would have barely registered.
The trouble is that this is so common that it takes something like this cluster of events for people to actually hear about it because these things have been normalized unless it’s something really consequential. We all just accept that we’re screwed. It just keeps happening over and over again. And sometimes you get these clusters.
That’s not to say you shouldn’t be concerned about it. I’ve been yelling about this for a long time, but I think that what we’re seeing is finally people starting to pay attention to security after ignoring it forever because AI is putting it on steroids. You can see the potential of losing a grid, water systems, other critical infrastructure. It would have been more interesting if they were cyber attacks because that would have at least said that the system was operational. But this is all bog-standard systems blow up. To me, a bigger problem is AI cyber attacks of legacy systems whose owners have been told over and over again that they’re vulnerable and everybody just ignored it anyway.
Master the detail and master the risk.
One of your favorite things to cover back in the day were incidents at small regional hospitals. On November 12, Monadnock Community Hospital in New Hampshire had a server outage. What stands out to you about hospital incidents in general?
Charette: Cyber attacks and ransomware attacks on hospitals have been going on for the last decade and have have increased. Hospital systems have always been very vulnerable, even more so now with modern Electronic Health Records (EHR) systems. There’re just a lot of IT systems within hospitals that have open doors that attackers can use. What encouraged cyber attackers was the US government’s push toward using EHRs as rapidly as possible. It opened up a lot of opportunities for bad actors. If you go back to the late oughts and early teens, people were warning that hospitals were very vulnerable to cyber attacks, especially as they started to cross over to new EHRs, and that was basically ignored.
So given all of that has transpired over the last two decades and over the last two weeks, what should the engineer reading this take away?
Charette: Master the detail and master the risk. Look at these outages, they’re always finding these details they didn’t know about, right? There’re all these unknown interconnections, these unknown processes, these interfaces they didn’t fully understand. We’re in an age where because of this complexity, you have to be twice or three times as careful as you thought you needed to be.
And what’s the takeaway for the non-engineer, the person who just wants their flights to leave on time, or at all?
Charette: Bring a really big book with you like War and Peace. Because there’s nothing you can do. Oh, and don’t lose your charger.
From Your Site Articles
Related Articles Around the Web


Be the first to comment