What we know so far: The nationwide ground stop that froze US air travel on Friday didn't start at a major hub or a busy control center. It began at a small Federal Aviation Administration radio site in El Dorado, Arkansas, where a router installed during routine upgrade work began flooding the agency's private network with connection requests that the older equipment it connected to couldn't handle. Soon after, the links between pilots and controllers and the radar feeds that track aircraft started failing. It resulted in more than 3,500 flights being held for about five hours, leaving tens of thousands of travelers stranded.
The volume of traffic was far beyond anything the network normally handles. "We saw two places that were actually pinging our network at a very high rate, a thousand connections over every four to five seconds," Rebecca Guy, the FAA's technology chief, said on Saturday at a press conference.
El Dorado was one of those two places. The other was a radar facility in Amarillo, Texas, which had been upgraded several weeks earlier and showed a second problem that officials have not described. Engineers took both sites offline, but by then each had been hitting the network hundreds of times per second. That was enough to knock out other parts of the air traffic communications system, and the delays and cancellations carried into the weekend.
The Arkansas work was part of a long-running FAA effort to move its telecommunications from old copper wiring to fiber-optic cable. L3Harris, which owns and operates the network, has worked with the FAA on modernization for more than two decades and says it has finished more than 70% of the planned conversion. Crews chose to work on the El Dorado station because it seemed remote enough that a problem there would cause little harm, an assumption that turned out to be wrong.
The hardware involved is not the kind of router found in a home office. The FAA runs tens of thousands of these networking devices, including routers and switches, and they are larger, industrial-grade units built for harsh environments, Michael McCormick, a professor at Embry-Riddle Aeronautical University in Daytona Beach, Flaorida, told The New York Times. They sit on a closed network meant to keep out cyberattacks. They also have to read and process radar data, flight plans and audio, then send each piece to the right place. The agency's equipment comes from Cisco Systems and Juniper Networks, which Hewlett Packard Enterprise bought last year, and the FAA has been slowly swapping older Cisco routers for newer Juniper models.
Aviation experts think the failure came from a speed mismatch. A new device that pushes out data faster than older equipment can answer can stall the older gear, and the trouble spreads from there. R. John Hansman, a professor of aeronautics and astronautics at the Massachusetts Institute of Technology, said the new equipment pushed older parts of the network beyond their usual pace. "Now you have multiple parts of the old system that are running much faster than they ever ran," he said. "You never had this problem, because you never had all of the elements running at that max speed."
When the network began to falter, McCormick said, it became "unpredictable and unreliable," and stopping departures was the right decision. "It was a smart decision to actually decide to ground-stop all air departures," he said.
Investigators are now trying to determine whether the router was defective or whether something went wrong during installation. Transportation Secretary Sean Duffy confirmed the problem on Sunday during an appearance on Face the Nation. "We actually have both routers with L3Harris, who operates and owns our network. FAA is on site as well. FBI is offering support. We're looking at that right now, and when we know what the root cause of this issue was, was it the install from L3Harris? Was it the router?" he said. "I'm going to tell you and the American people what we find."
The FAA says it has found no evidence of a cyberattack. L3Harris said it is helping with the investigation.
Not everyone accepts the official explanation. Robert W. Mann Jr., who founded an airline analysis and consulting firm, doubted that one device could cause this much damage. "The idea that you would have a single router take down a national network is just not plausible," he said. He saw the outage as evidence of a long-known weakness. "What we really have here is a demonstration that what we knew a decade ago was problematic done at small scale is really, really problematic done at nationwide scale."
The episode raises questions about redundancy in a system where trouble at one small facility could stop flights nationwide. Duffy and other officials blamed the aging infrastructure and said Congress needs to provide more money for upgrades. Experts warned that speeding up the work brings its own risks, since the legacy network must stay in service while new equipment is installed and fully tested. Some suggested doing hardware and software changes during night shifts, when far fewer planes are flying, rather than during busy daytime hours.
