Note: My Web pages are best viewed with style sheets enabled.

Unrated

The CrowdStrike Global Disaster

Proper Procedures Could Have Prevented It

Copyright © 2024 by David E. Ross

In the third week of July 2024, many Windows PCs around the world could not be booted up. Banks, airlines, hospitals, governments, grocery stores, and others saw their PCs repeatedly crash with the "blue screen of death". No, this was not a hostile attack. This was the result of a faulty update to software developed by CrowdStrike that was supposed to provide users with secure systems and protect them from hostile attacks.

Although I retired 21 years ago from my 30-year career as a software-test engineer, I still remember certain key principles that were apparently not followed by CrowdStrike.

• Software updates should be distributed gradually. Then, if a problem is detected, the distribution can be halted before all users are injured. CrowdStrike broadcast its update all at once to all its users worldwide.
• Software updates should not be forced onto users. Users should be notified of the availability of updates and then be allowed to install them (1) after in-house evaluation and testing and (2) when it is most convenient. CrowdStrike forced the update on its users, none of whom had a choice to delay the update.

  I do not allow any updates to be forced onto my or my wife's PC. I download the update installer file and scan it with three different unrelated anti-virus applications. I read the change log for that update. After I disable my Internet connection, I execute the installer file while logging all changes to my Windows registry and to the files on my PC.

• Software updates should be tested by the developer in a series of scenarios that mimic complete systems. The updates should not be released if even a single test shows anomalous results. It appears no such testing was done by CrowdStrike.
• During the distribution of a critical software update, all senior members of the development team must remain on duty to trouble-shoot any adverse events. I do not know if CrowdStrike had any of its development team on duty.

Users of CrowdStrike's software were not entirely blameless. They should have backup facilities for critical computer systems. Those backup facilities should include alternative security applications that are never updated at the same time as the applications in the primary facility. Yes, that can be costly. However, how costly was this fiasco to CrowdStrike's users? (I smell some very large lawsuits.)

21 July 2024

David Ross home