A Brief History of Everything Wireless Blog: The new most expensive software failure

The new most expensive software failure

2019-07-30 [Petri]

The very first Ariane 5 rocket blew up in 1996, with an estimated loss of 370 million dollars.

The cause of the accident was Ariane 5's Inertial Reference System software, which was lifted as-is from the earlier, flawlessly functioning Ariane 4 and installed without comprehensive re-testing.

Unfortunately Ariane 5 accelerated much faster than Ariane 4, creating an overflow in memory: a 64-bit value was tried to be saved in a 16-bit location, causing the IRS to shut down and send diagnostic data to the engine control, which assumed this to be a valid command and tried to change the rocket's direction by 90 degrees at full speed while it was at an altitude of 4 km. As a result, the aerodynamic forces started to rip the rocket apart, and a self-destruct was sent out by the controllers.

There was a backup system for the IRS, but as it was a copy of the main system, it also failed in the same way at the very same moment: a good example of a common-mode error situation that should be avoided in redundant systems.

The irony in this case was that this particular system was not even needed to be active at the point of flight when the overflow and subsequent system shutdown occurred, but it was left running as it had also been active throughout Ariane 4 launches, and any changes to the functionality would have required further testing.

The learning from all this is obvious: if something works flawlessly in configuration A, do not assume that it will behave the same way in configuration B. Test it.

As a singular event, the infamous loss of the Mars Climate Orbiter due to Metric vs. Imperial value mismatch was more costly. But by adding the costs of all of the subsequent delays that the Ariane blowup caused for other Ariane 5 launches in the pipeline, it can be safely argued that Ariane 5 has been the most expensive software error so far, and probably is somewhere in the 500 million dollar range.

Yet that half a billion dollars is now peanuts, even when taking inflation into account:

Step aside, Ariane – there's a new “king” in town.

Boeing announced that the losses during the last financial quarter were the highest ever: 2.9 billion dollars. This is mainly due to the anticipated costs of compensating for the grounding of the worldwide fleet of 737 MAXes, for which Boeing allocated 4.9 billion dollars after taxes.

The grounding is a direct consequence of the issues with the MAX's MCAS system, which I have formerly discussed in depth. The links can be found below.

Yet this may still not be the end of it, as Boeing also announced that it is considering to halt the 737 MAX production, and MAX has been their most profitable product. This would naturally also reflect the profits of the tens of subcontractors in the MAX product line.

As often is the case, there is no single reason for this unfortunate situation, in which a highly respected century-old aerospace company has shed its reputation in one fell swoop. But the root cause is that profits took the front seat, eventually turning the initial savings into huge losses and personal tragedies for the victims of the two 737 MAX accidents.

More information on the background of this case can be found in my earlier blog entries:

737 MAX - The new DC-10?
Fingerpointing has started
MAX turbulence continues
Nine dollars per hour...

Permalink: https://bhoew.com/blog/en/89

Show latest Earlier entries