Part 1: Overview

Doomsday Machines: When software mistakes become lethal.

Background and Summary

The most reliable thing about software is that it's always broken. Sometimes just a little bit, but more often than not it's badly broken. It loses data, lets attackers in to steal our data, and it crashes or needs a restart weekly if not daily.

Letting the attackers in is becoming a pretty serious problem as we see daily breaches in the news. But people are still making money from using software, and it's not like it's ever killed anyone, right?

Wrong. This website and accompanying video series serves as a wake up call about the increasing damage software is doing to society. As the whole world goes digital, the are more opportunities for software to put us in real danger. Even more worrying, we're integrating software into all of our appliances, watches, phones, cars, medical equipment, utilities, banking, and every other aspect of our lives and connecting everything together. This puts us at extreme risk from small problems in one device cascading into enormous regional disasters.

While the concept of bug free software is discussed openly in Silicon Valley and the broader tech industry as an impossibility, the military and aerospace industries have been pioneering approaches and deploying highly robust systems for around half a century. It's time that the rest of the software industry started taking their increasing responsibilities as seriously as other engineering disciplines.

It's not too late to avert doomsday.

Definitions

Doomsday Machine: A hardware or software system (usually digital) that, due to design defects or omissions, through correct and normal use by the user or operator, causes death or destruction of lives i.e. imprisonment. Death and damages due to malicious cyber activity and cyber warfare are a separate categories.

Misconceptions

Software cannot be bug free

Software is broken, but it's not that serious of a problem. While we see the number and severity of breaches, ransomware incidents, and other attacks continue to rise, there is still a prevailing attitude that this is unlikely to happen to us personally.

We must fix all the bugs

I'm not a target

Further, we don't believe that we are important enough to be targets of such hostile attacks.

Cyber attacks are the real danger

What seems to be neglected and misunderstood is that the damage done from defects or bugs, non-malicious attacks is actually a larger and more pressing problem than cyber attacks. This series is an attempt to bring awareness to these issues and prompt necessary changes.

Further, all security vulnerabilities/incidents are caused by defects or omissions in the design of the targeted systems. Weird machine behavior is considered a failure of design.

Variant of Hanlon's Razor: Incompetence and omission is a more dangerous threat than malice.

Back to top