The recent Crowdstrike outage left many businesses scrambling and cyber security professionals on high alert. As one of the leading providers of endpoint protection, Crowdstrike‘s reliability is critical to numerous organizations worldwide.
This blog aims to break down what happened during the outage, what Crowdstrike has communicated, and where things currently stand.
Timeline
Friday July 19 2024, 04:09 UTC (12:09 AM ET)
Crowdstrike released a sensor configuration update to Windows systems as part of its ongoing security operations.
05:27 UTC (1:27 AM ET)
The faulty update was remediated by CrowdStrike.
Early Friday morning
- 911 centers in several states, including Alaska and Arizona, experienced service outages.
- Some hospitals began experiencing technology issues.
- Major airlines including Delta, American, United, and Allegiant grounded flights due to the outage.
- Public transit systems in the Northeast faced delays, impacting transportation services in cities like Washington, DC, and New York City.
Morning
- The White House initiated investigations into the outage.
- Global banks reported service disruptions.
- Emergency declarations were made in response to the outage affecting critical systems.
- Federal agencies, including the Department of Homeland Security, Department of Justice, and Social Security offices, reported service disruptions.
Throughout Friday
- Hospitals across the nation announced impacts, delays in services, and cancellations of non-urgent medical procedures and appointments.
- Blood supply organizations faced challenges in distributing blood to hospitals.
- Financial services and general practitioners’ offices were disrupted.
- TV broadcasters went offline.
- Airports issued advice to passengers due to grounded planes and delayed services.
By Friday afternoon
- CrowdStrike had issued a fix for impacted systems.
- The company stressed that the outage was not tied to a cybersecurity event or attack.
Saturday, July 20, 2024
- CrowdStrike released a detailed blog post explaining the technical aspects of the outage and apologizing for the incident.
What happened?
On July 19, 2024, a widespread outage affected Crowdstrike users, leading to significant disruptions. The incident was triggered by an update that caused the infamous Windows Blue Screen of Death (BSOD) on numerous machines globally.
This BSOD is a critical error screen displayed by the Windows operating system, indicating a severe issue that prevents the system from operating safely.
According to reports, the update from Crowdstrike conflicted with certain Windows processes, leading to system crashes and rendering devices unusable.
As businesses and cyber security teams depend heavily on Crowdstrike for endpoint security, this incident had a profound impact, causing operational delays and increasing vulnerability to cyber threats.
Who was affected?
Here are some of the key sectors and areas affected by the Crowdstrike outage:
- Airlines: Major carriers like Delta, United, American, Allegiant, and airlines in countries such as France, Spain, Australia, and India experienced flight groundings and disruptions.
- Healthcare: Hospitals faced challenges accessing crucial patient data, leading to delays in surgical procedures. For example, York’s Sloan King Cancer facility experienced delays.
- Emergency Services: 911 dispatch centers in places like Alaska and Ohio were affected, forcing local authorities to provide alternative contact numbers.
- Media and Broadcasting: TV broadcasting services were impacted, including the U.K.’s Sky News and various channels in France and the U.S.
- Retail: Some retail businesses faced operational issues, such as McDonald’s Japan, which suspended about a third of its stores due to cash register problems.
- Public Infrastructure: Iconic locations like Times Square in New York City were affected, with digital screens running on Windows temporarily going dark.
- Banking: The financial sector was among those disrupted, though specific details were not provided.
- Cloud Services: Cloud-dependent businesses faced severe disruptions, highlighting concerns about over-reliance on cloud services.
What Crowdstrike said
In response to the outage, Crowdstrike issued several statements to address the concerns of their users.
On their website, CEO George Kurtz wrote:
“We know that adversaries and bad actors will try to exploit events like this. I encourage everyone to remain vigilant and ensure that you’re engaging with official CrowdStrike representatives. Our blog and technical support will continue to be the official channels for the latest updates.
Nothing is more important to me than the trust and confidence that our customers and partners have put into CrowdStrike. As we resolve this incident, you have my commitment to provide full transparency on how this occurred and steps we’re taking to prevent anything like this from happening again.”
What you need to do
CrowdStrike offers a number of remediation steps:
1. Reboot hosts:
- Ensure the host is connected to a wired network.
- Reboot the host to download the reverted channel file.
2. Manual recovery (if crashes persist):
- Follow detailed steps in the provided Microsoft article.
- Note: BitLocker-encrypted hosts may require a recovery key.
3. Automated recovery via bootable USB:
- Follow instructions in the KB article available on the support portal.
- Note: BitLocker-encrypted hosts may require a recovery key.
4. Identification of impacted hosts:
- Use Advanced Event Search Query or updated dashboards available in the support portal.
5. Non-impacted hosts:
- Windows hosts brought online after 2024-07-19 0527 UTC are not impacted.
For continuous updates, visit the CrowdStrike Support Portal.
Meanwhile Microsoft has also provided its customers with a recovery tool to mitigate the impact of the outage. The two options it offers are:
Recovery from WinPE: This method involves creating boot media to repair the device, not requiring local admin privileges. If BitLocker is enabled, manual input of the recovery key might be needed. For third-party disk encryption, refer to the vendor’s guidance.
Recovery from safe mode: This method uses boot media to start the device in safe mode, requiring local admin access for remediation. Suitable for devices with TPM-only protectors or non-encrypted devices. BitLocker-enabled devices may need the recovery key or PIN entered.
