CrowdStrike blames testing software for crashing 8.5 million Windows machines

CrowdStrike has published a post-incident review (PIR) of the buggy update it published that took down 8.5 million Windows machines last week. The detailed post blames a bug in the testing software for not properly validating the content update that was pushed to millions of machines on Friday. CrowdStrike promises to more thoroughly test its content updates, improve error handling, and implement a staggered rollout to avoid a repeat of this disaster.

CrowdStrike’s Falcon software is used by companies around the world to help manage malware and security breaches on millions of Windows machines. On Friday, CrowdStrike released a content configuration update for its software that was supposed to “gather telemetry about potential new threat techniques.” These updates are delivered regularly, but this particular configuration update caused Windows to crash.

CrowdStrike typically issues configuration updates in two different ways. There is what is called Sensor Content that directly updates CrowdStrike’s own Falcon sensor that runs at the kernel level in Windows, and separately there is Rapid Response Content that updates how that sensor behaves to detect malware. A small 40 KB quick response content file caused Friday’s issue.

Actual sensor updates do not come from the cloud and typically include AI and machine learning models that will allow CrowdStrike to improve its sensing capabilities over the long term. Some of these features include something called Template Types, which is code that enables rediscovery and is configured by the separate quick response content type that was delivered on Friday.

On the cloud side, CrowdStrike manages its own system that performs validation checks on content before it is released to prevent an incident like Friday’s from happening. CrowdStrike released two rapid response content updates last week, or what it also calls model instances. “Due to a bug in the content validator, one of the two model instances passed validation despite containing problematic content data,” says CrowdStrike.

While CrowdStrike performs automated and manual testing on sensor content and model types, it does not appear to do as thorough testing on the rapid response content that was delivered on Friday. A March rollout of new model types provided “confidence in the checks performed in the content validator,” so CrowdStrike appears to have assumed that implementing quick-response content would not cause problems.

This assumption caused the sensor to load the problematic fast-response content into its content interpreter and trigger a memory out of bounds exception. “This unexpected exception could not be handled correctly, resulting in a Windows operating system crash (BSOD),” explains CrowdStrike.

To prevent this from happening again, CrowdStrike promises to improve its rapid response content testing using local developer testing, content refresh and rollback testing, along with stress, fuzzing and fault injection testing. CrowdStrike will also perform stability testing and content interface testing on Quick Response Content.

CrowdStrike is also updating its cloud-based content validator to better verify rapid response content releases. “Rechecking is underway to prevent this type of problematic content from being deployed in the future,” says CrowdStrike.

On the driver side, CrowdStrike will “improve the handling of existing errors in the Content Interpreter,” which is part of the Falcon sensor. CrowdStrike will also implement a staggered deployment of Rapid Response Content, ensuring that updates are gradually rolled out to larger portions of its installed base, rather than an immediate push to all systems. Both driver improvements and staggered deployments have been recommended by security experts in recent days.