What You Need to Know About the Worldwide Crowdstrike Outage
Hi folks,
Big day with half the world down for a Windows security update, eh? My thoughts go out to all of you working on fixing this. The main story is an explanation of CrowdStrike, what happened, and what folks are having to do to fix it.
And today’s issue is on the house. So hi again free subscribers!
Peace,
Tom
BIG STORY
What is CrowdStrike Falcon and what does it do? Is my computer safe?
The Conversation
How to fix a Windows PC affected by the global outage
MIT Technology Review
What we know about CrowdStrike's update fail that's causing global outages and travel chaos
TechCrunch
Major Windows BSOD issue takes banks, airlines, and broadcasters offline
The Verge
CrowdStrike fixes start at “reboot up to 15 times” and get more complex from there
Ars Technica
Major outages at CrowdStrike, Microsoft leave the world with BSODs and confusion
Ars Technica
CrowdStrike and Microsoft: What we know about the global IT outage
BBC
Massive global IT outage hits banks, airports, supermarkets – and a single software update is likely to blame
The Conversation
Sky News Off Air Briefly, Planes Disrupted During Global IT Outage
Variety
Starbucks mobile ordering falls victim to the CrowdStrike BSOD outage
The Verge
Serious questions to answer after what could be the biggest IT outage in history
Sky News
WHAT HAPPENED
Early Friday in Australia, Windows computers at large companies began seeing the old "Blue Screen of Death" and were unable to reboot. The outage spread as the sun came up in Asia, Europe, and the Americas.
The Blue Screens were caused by a faulty update to CrowdStrike Falcon Sensor's enterprise security product pushed at around 4:09 UTC on July 18, just after midnight Eastern Time. At the same time, Microsoft reported an Azure outage for a cloud region covering the central US because of a configuration change. That is now resolved and was unrelated to the CrowdStrike bug and the BlueScreens but certainly confused the issue.
The CrowdStrike bug affected airports and airlines, causing planes to be grounded. In fact, the US FAA put a ground stop in effect at one point to sort out the chaos caused by the outage. It also affected banks, hospitals, transport services, media including Sky Media which went off the air briefly, and shops including Starbucks which had to close stores because it was unable to take orders.
WHAT IS CROWDSTRIKE FALCON SENSOR?
Falcon is "endpoint detection and response" software or EDR. It proactively monitors a computer's operations and if it detects malicious activity, it attempts to shut it down. It's more sophisticated than just matching viruses with a table of definitions. It looks at internet communications, what programs are running, what files are being opened, and more.
It is privileged software with access to internal systems not usually granted to other programs; a kernel-level driver, if you know what that means, integrated into the core of Windows. If it detects an unusual pattern, it can unilaterally shut down communication.
Falcon is one of the top EDR providers, meaning it is used by a lot of companies, and meaning it affected a lot of the world when it got a major bug. It is not usually used by consumers, so you probably didn't have an issue on your home PC but might have on a work PC. And if you're a company admin, godspeed my friend. We're cheering for you.
CrowdStrike is a security company with around 29,000 corporate customers worldwide. It's famous for catching the attackers who tried to break into Democratic National Committee networks in 2016. It's the company that gives hacker groups animal-themed names like Fancy Bear, Charming Kitten, and Gothic Panda. It had a Super Bowl ad this year.
WHY DID THIS HAPPEN?
We don't have technical details yet, but something in an update to Falcon caused Windows to crash and fail to reboot.
One thing I want to say to fend off some of the usual knee-jerk responses:
CrowdStrike almost certainly did test this update before deploying it. Not all bugs show until pushed at scale. It will be interesting to see why this bug didn't show up in tests.
This isn't necessarily Microsoft's fault. Sky Media, annoyed at being knocked off air, is already calling for Microsoft's head. They're an attractive punching bag and I wouldn't rule out there was something in Windows that might have caused this but I also wouldn't jump to that conclusion.
WHAT ARE THE FIXES?
CrowdStrike has issued a patch as of 5:27 UTC Friday. Rebooting the machine may trigger a download of the reverted channel file. That might take as many as 15 reboots.
If that doesn't work, which reportedly it doesn't always, Microsoft recommends restoring the system to before 4:09 UTC July 18, just after midnight Eastern Time before the buggy version of the patch was pushed. Then you can accept the fixed patch.
That isn't always working, so if it doesn't, you can manually delete the buggy update. If you're fixing just one machine it's not that bad. You boot into safe mode or Windows Recovery, go to the CrowdStrike directory and delete C-00000291*.sys then reboot.
This can't be easily done remotely though. So virtual machines and remote servers cause a particular challenge.
Plus the wide nature of the outage causes some interesting issues. For example, one listener of DTNS noted that they needed their Bitlocker keys to fix the issue but the recovery server was down because of the bug.
HOW BIG IS THIS?
This was NOT a malicious attack but the only comparable outages are from malicious attacks. NotPetya, WannaCry, etc. Security Consultant Troy Hunt apparently said, "basically what we were all worried about with Y2K, except it's actually happened this time."
More Stories
The FBI used an unreleased Cellebrite tool to get into the Trump shooter’s phone
The Verge
Yesterday we learned the news that law enforcement hacking software company Cellebrite appeared to be unable to get into phones running iOS 17.4 as well as Google Pixel 6, 7, and 8 phones that had been turned off. Today we learn that Cellebrite gave the US FBI unreleased software to unlock the phone of the person who shot President Trump. Reportedly it was able to unlock the "newer Samsung phone" in 40 minutes.
Netflix Adds 8 Million Subscribers in Q2, Profits Surge 44%
The Wrap
Netflix will drop a new multiplayer game when Squid Game season 2 premieres this year
Engadget
Netflix is getting more comfortable with ads
The Verge
Netflix Making Hay With Games Based On Its IP, Releasing One A Month
Deadline
Netflix beat expectations, increasing its revenue 17% on the year and growing profits by 44%. It also gained subscribers, raising the number 16.5% on the year, up 8.05 million to 277.65 million subscribers worldwide. Netflix said this quarter it won't add as many subscribers as it did this time last year as the effects of its crackdown on sharing accounts wane. Netflix reported that its ad-supported tier resulted in 45% of new sign-ups in markets where it is offered. And Netflix will ramp up its gaming products, saying it will release one new game related to its own IP every month starting now. A Squid Game-related game will be one of those later this year.
Fiio’s DM13 brings welcome upgrades to Sony’s classic Discman
The Verge
If you miss portable CD players like the Sony Discman or just want to go retro, a Chinese company named Fiio is releasing a new portable CD player called the DM13. For $179 you can play CDs on the go with support for Bluetooth wireless headphones and both 3.5mm wired headphones as well as 4.4mm balanced wired headphones. It has a USB connection to integrate it into a digital sound system as well. The battery gets about 8 hours of playback on a charge. It can play standard CDs as well as MP3, FLAC, WAV, AAC, and WMA audio files burned onto a CD-R. If Fiio sounds familiar, it also showed a portable cassette player at CES this year.
New Galaxy Z phones get fewer pre-orders, more young customers
SamMobile
Samsung says it has taken more than 910,000 pre-orders for the Galaxy Z Fold 6 and Galaxy Z Flip 6 in South Korea, down slightly from the 1.02 million it took for the 5s in the same period last year. About half the orders came from customers in their 20s and 30s, up from 43% last year, so this is appealing to a younger demo slightly. Also about 40% of the orders were for the fold, vs. 30% last year.
For Context
Mobile internet and social media disrupted in Bangladesh amid student protests
The Record
The government of Bangladesh has ordered mobile telecommunication providers to restrict network access for more than 24 hours during student protests that have so far resulted in 6 deaths and hundreds of injuries. This follows several days of social media restrictions. The protests are in relation to a quota system for government jobs.
Samsung has suspended Galaxy Buds 3 Pro shipments
Android Authority
Some of the ear tips were torn among other quality issues. So Samsung is taking a pause to inspect the process and find out what's happening.
Google's goo.gl links will stop working in August 2025
9to5Google
Google stopped generating new "short links" to the goo.gl domain in March of 2019. On August 25, 2025 - so next year - it will stop redirecting any existing links from those short URLs. Starting August 23 this year, Google will provide an interstitial when anyone clicks on a goo.gl short link explaining that the link will stop working next year.
Google is purging ‘low-quality’ Android apps next month
The Verge
Starting August 31, Google requires all apps listed in the Play Store to have a ‘stable, engaging, responsive user experience.’ This is meant to crack down on spam apps that don't do anything or have a single wallpaper etc.
Samsung Electronics and striking union to resume talks on Friday
Reuters
The Samsung strike drags on.
Interesting Reads
FTC blasts Microsoft’s new ‘degraded’ Xbox Game Pass Standard tier and price increases
The Verge
Google Maps cuts prices by 70% for developers after Ola Maps’ free offering
Entrackr
Huawei sues Taiwan's MediaTek over alleged patent infringement
Nikkei Asia
Proton Mail now has a privacy-focused AI writing assistant
Engadget