On Friday 19, 2024 many organizations across the globe have seen disruptions on their windows systems stemming from a faulty update pushed out by CrowdStrike; a prominent cybersecurity company.
CrowdStrike is a Texas-based EDR (Endpoint Detection & Response) solutions company serving over 500 companies in the fortune 1000. Their products run on millions of endpoints across different sectors of the market such as hospitals, airports, financial institutions & universities.
The EDR product requires administrative privileges on the endpoints to monitor, review and update systems. According to the company, it was a channel file [C-00000291.sys] containing the sensor data which caused the windows systems to BSOD(Blue Screen OF Death). While it was not a major update in the EDR component itself but, it was massive enough to bring down the entire IT world running windows systems.
According to the statement CEO of CrowdStrike George Kurtz, “CrowdStrike is actively working with customers impacted by a defect found in a single content update for windows hosts”. He also noted that, “Mac and Linux systems are not impacted. This is not a cybersecurity incident”
Impact
The impact of this faulty update has been chaotic and halted many critical operations across the globe.
Microsoft users in Australia reported the outages first on July 19, with other recognized companies such as Visa, Netflix and Vodafone.
According to BBC, major airports have cancelled multiple flights due to a “technical issue with CrowdStrike”
In a post by Matt Burgess at Wired, “The US Emergency alert systems, which issues hurricane warnings and various 911 services were disrupted”
In the United Kingdom, NHS England confirmed the appointment and patient record system has been impacted.
Dutch broadcasting organizations NOS said “ the forced several flights to be grounded”
Remediation
CrowdStrike has identified and notified their customers of the ongoing update issue and released a workaround while they work at the permanent solution.
The following steps can be taken to recover windows hosts:
1. Boot Windows into Safe Mode or the Windows Recovery Environment
2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
3. Locate the file matching “C-00000291*.sys”, and delete it.
4. Boot the host normally
In an updated statement, CrowdStrike says that “the problematic channel file [C-00000291*.sys” with timestamp of 0409 UTC] has been reverted” and the good version of it is C-00000291*.sys with timestamp of 0527 UTC or later.
Afterthoughts
IT admins are going to have a long weekend to recover from this event. Many organizations have thousands of endpoints affected by this and recovering them would be gruesome task. However, this brings up an important topic of business resiliency. Organizations should focus on maturing their business continuity/resiliency plan if they ever encounter situations like this. With the growing trend of relying on 3rd party applications, events like this most likely to happen in future and it’s a paramount responsibility for leaders to prepare their organizations to recover from any disastrous scenario.