www.crowdstrike.com
Open in
urlscan Pro
2606:4700::6810:b476
Public Scan
Submitted URL: http://crowdstroke.io/
Effective URL: https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/
Submission: On September 22 via api from TW — Scanned from NL
Effective URL: https://www.crowdstrike.com/falcon-content-update-remediation-and-guidance-hub/
Submission: On September 22 via api from TW — Scanned from NL
Form analysis 0 forms found in the DOM
Text Content
Skip to main contentEnable accessibility for low visionOpen the accessibility
menu
x
See the latest news and announcements from Fal.Con Las Vegas! Visit the Fal.Con
2024 News Center
Skip to Main Content
* Experienced a Breach?
* Small Business
* CrowdStrike Marketplace
* Contact Us
* Blog
English
* * Deutsch
* * English (AU)
* * English (UK)
* * English (US)
* * Español
* * Français
* * Italiano
* * LatAm
* * Português
* * عربى
* * 日本語
* * 繁體中文
* * 한국어
* Platform
Explore Platform
THE DEFINITIVE AI-NATIVE CYBERSECURITY PLATFORM
* Endpoint Security
The leader in EPP and EDR, backed by pioneering adversary intelligence and
native AI.
* Exposure Management The leader in exposure management with complete attack
surface visibility & AI-powered vulnerability management.
* Identity Threat Detection & Response Stop modern attacks in real time with
the only unified platform for identity protection and endpoint security.
* IT Automation Consolidate security and IT with one platform, agent, and
console to cut complexity and cost.
* Threat Intelligence & Hunting The leader in cyber threat intelligence with
world-class research and elite threat hunting to disrupt adversaries.
* Cloud Security The most complete CNAPP with unified agent and agentless
protection, from code to cloud.
* Next-Gen SIEM
The world’s only AI-native SOC platform that consolidates siloed security
tools and data.
* Generative AI Turn hours of work into minutes or seconds with generative
AI workflows for cybersecurity and IT.
* Data Protection
Unified data protection that deploys instantly on existing agents to stop
the theft of sensitive information.
* Workflow Automation Build your own workflows with native security
orchestration, automation, and response (SOAR).
* Services
Prepare
--------------------------------------------------------------------------------
Prepare and train your organization to defend against sophisticated threat
actors using real-life simulation exercises.
SEC Readiness Services
Tabletop Exercise
Red Team/Blue Team Exercise
Adversary Emulation Exercise
Penetration Testing
Respond
--------------------------------------------------------------------------------
Available under a Services Retainer, giving you access to security
consultants and expertise to respond to a breach.
Incident Response
Compromise Assessment
Endpoint Recovery
Network Detection
Experienced a breach?
Fortify
--------------------------------------------------------------------------------
Enhance your cybersecurity practices and controls with actionable
recommendations to fortify your cybersecurity posture.
Maturity Assessment
Technical Risk Assessment
SOC Assessment
Cloud Security Assessment
Identity Security Assessment
Managed Services
--------------------------------------------------------------------------------
Managed Detection & Response
Included in Falcon Complete and backed by CrowdStrike's Breach Prevention
Warranty.
Cloud Detection and Response
The only CDR that unifies world-class threat intelligence and 24/7 services
with the world’s most complete CNAPP.
Additional Services
--------------------------------------------------------------------------------
Cloud Security Services
Identity Protection Services
Falcon LogScale Services
Partner Services
* Why CrowdStrike
WHY CROWDSTRIKE
* Industry Recognition CrowdStrike is the recognized leader in endpoint
protection solutions.
* Customer Stories Don’t take our word for it, hear what our customers have
to say.
COMPARE CROWDSTRIKE
* vs. Microsoft Burdensome operations drive up TCO.
* vs. Palo Alto Networks Hard to deploy, hard to use, harder to manage.
* vs. SentinelOne Weak coverage, can’t stop breaches.
* vs. Wiz Incomplete CNAPP that can't stop breaches.
* vs. Other Competitors See why CrowdStrike is the top choice for
comprehensive cybersecurity.
SOLUTIONS BY TOPIC
* Cloud Detection and Response The only CDR that unifies world-class threat
intelligence and 24/7 services with the world’s most complete CNAPP.
* Zero Trust Real-time breach protection on any endpoint, cloud workload or
identity, wherever they are.
* Ransomware Protection Learn what you can do to stop ransomware threats in
their tracks.
* Observability & Log Management Fills in the gaps, logs everything, and
realizes real-time observability for your entire system.
* Log4Shell Mitigation Get the latest information on this evolving
vulnerability.
SOLUTIONS BY INDUSTRY
* Small Business
* Election Security
* State and Local Government
* Federal Government
* Healthcare
* Education
* Retail
* Learn
Featured Resources
--------------------------------------------------------------------------------
Cybersecurity 101 Glossary
Explanations, examples and best practices on a variety of cybersecurity
topics.
Get Your Threat Landscape
Discover the adversaries targeting your industry.
2024 Global Threat Report
The must-read cybersecurity report of the year.
2024 Threat Hunting Report
CrowdStrike's threat hunting insights from July 1, 2023 to June 30, 2024.
CrowdStrike Blog
--------------------------------------------------------------------------------
Under The Wing
Discover how CrowdStrike protects you against the most advanced attacks.
From The Front Lines
Executive Viewpoint
Counter Adversary Operations
Customer Focused
--------------------------------------------------------------------------------
Free Trial Guide
Customer Support Portal
CrowdStrike University
CrowdStrike Tech Hub
Developer Portal
Knowledge Resources
--------------------------------------------------------------------------------
Customer Stories
White Papers
Webinars
Adversary Universe Podcast
Reports
Logging Guides
Try interactive demos
All Resources
* Company
Connect With Us
--------------------------------------------------------------------------------
Careers
Events
Fal.Con 2024
Falcon Encounter Hands-on Labs
Partner Programs
--------------------------------------------------------------------------------
Channel Partners and Distributors
Service Providers
Strategic Technology Partners
CrowdStrike Marketplace
View All
Become a partner
About Us
--------------------------------------------------------------------------------
Our Story
Board of Directors
Investor Relations
CrowdStrike & F1 Racing
Executive Team
Latest News
Environment, Social & Governance
* Login
Contact Us
--------------------------------------------------------------------------------
* View bundles & pricing
--------------------------------------------------------------------------------
* Platform
--------------------------------------------------------------------------------
* Services
--------------------------------------------------------------------------------
* Why CrowdStrike
--------------------------------------------------------------------------------
* Learn
--------------------------------------------------------------------------------
* Company
--------------------------------------------------------------------------------
* Blog
--------------------------------------------------------------------------------
* Marketplace
--------------------------------------------------------------------------------
* Login
--------------------------------------------------------------------------------
* Contact us
--------------------------------------------------------------------------------
* Experienced a breach?
--------------------------------------------------------------------------------
* Languages
--------------------------------------------------------------------------------
Back
REMEDIATION AND GUIDANCE HUB: CHANNEL FILE 291 INCIDENT
Support Portal Translated resources
* Channel File 291 RCA Exec Summary
* Statements from Our CEO
* Frequently Asked Questions
* Preliminary Post Incident Review
* How Do I Remediate?
* Third Party Vendor Information
* Additional Resources
Support Portal Contact us
Page last updated 2024-08-06 2119 UTC
Updated 2024-07-31 1638 UTC (Final Post-Incident Measurement Report)
Using a week-over-week comparison, ~99% of Windows sensors are online as of July
29 at 5pm PT, compared to before the content update. We typically see a variance
of ~1% week-over-week in sensor connections.
Updated 2024-07-25 1954 UTC
Using a week-over-week comparison, greater than 97% of Windows sensors are
online as of July 24 at 5pm PT, compared to before the content update.
Updated 2024-08-06 1600 UTC
CHANNEL FILE 291 RCA EXEC SUMMARY
This document provides an executive summary of the findings of CrowdStrike’s
Root Cause Analysis (RCA) report. The full report elaborates on the information
previously shared in our preliminary Post Incident Review (PIR), providing
further depth on the findings, mitigations, technical details and root cause
analysis of the incident.
Download the Root Cause Analysis PDF
Download the Executive Summary PDF
INTRODUCTION
CrowdStrike was founded with a mission to protect customers against today’s
adversaries and stop breaches. On July 19, 2024, as part of regular operations,
CrowdStrike released a content configuration update (via channel files) for the
Windows sensor that resulted in a system crash. We apologize unreservedly.
We acknowledge the incredible round-the-clock efforts of our customers and
partners who, working alongside our teams, mobilized immediately to restore
systems and bring many back online within hours. As of July 29, 2024, at 8:00
p.m. EDT, ~99% of Windows sensors were online, compared to before the content
update. We typically see a variance of ~1% week-over-week in sensor connections.
To any customers still affected, please know we will not rest until all systems
are restored.
WHAT HAPPENED
The CrowdStrike Falcon sensor delivers AI and machine learning to protect
customer systems by identifying and remediating the latest advanced threats. In
February 2024, CrowdStrike introduced a new sensor capability to enable
visibility into possible novel attack techniques that may abuse certain Windows
mechanisms. This capability pre-defined a set of fields for Rapid Response
Content to gather data. As outlined in the RCA, this new sensor capability was
developed and tested according to our standard software development processes.
On March 5, 2024, following a successful stress test, the first Rapid Response
Content for Channel File 291 was released to production as part of a content
configuration update, with three additional Rapid Response updates deployed
between April 8, 2024 and April 24, 2024. These performed as expected in
production.
On July 19, 2024, a Rapid Response Content update was delivered to certain
Windows hosts, evolving the new capability first released in February 2024. The
sensor expected 20 input fields, while the update provided 21 input fields. In
this instance, the mismatch resulted in an out-of-bounds memory read, causing a
system crash. Our analysis, together with a third-party review, confirmed this
bug is not exploitable by a threat actor.
While this scenario with Channel File 291 is now incapable of recurring, it
informs the process improvements and mitigation steps that CrowdStrike is
deploying to ensure further enhanced resilience.
WHAT WE DID AND WHAT’S NEXT
Based on the findings in the RCA, here are some of the actions CrowdStrike has
taken and will take moving forward:
* Update Content Configuration System test procedures. This work has been
completed. This includes upgraded tests for Template Type development, with
automated tests for all existing Template Types. Template Types are part of
the sensor and contain predefined fields for threat detection engineers to
leverage in Rapid Response Content.
* Add additional deployment layers and acceptance checks for the Content
Configuration System. This work has been completed with an updated deployment
ring process, ensuring Template Instances pass successive deployment rings
before rollout into production.
* Provide customers additional control over the deployment of Rapid Response
Content updates. New capabilities have been implemented and deployed to our
cloud that allow customers to control how Rapid Response Content is deployed,
with additional functionality planned for the future.
* Prevent the creation of problematic Channel 291 files. Validation for the
number of input fields has been implemented to prevent this issue from
happening.
* Implement additional checks in the Content Validator. Additional checks are
planned for release into production by August 19, 2024.
* Enhance bounds checking in the Content Interpreter for Rapid Response Content
in Channel File 291. Bounds checking was added on July 25, 2024, with general
availability expected August 9, 2024. These fixes are being backported to all
Windows sensor versions 7.11 and above through a sensor software hotfix
release.
* Engage two independent third-party software security vendors to conduct
further review of the Falcon sensor code and end-to-end quality control and
release processes. This work has begun and will be ongoing as part of our
focus on security and resilience by design.
For additional details and defined terms, please refer to the RCA.
Read more
STATEMENTS FROM OUR CEO
Sent 2024-08-06 1605 UTC
AUGUST 6, 2024 STATEMENT FROM GEORGE KURTZ, FOUNDER AND CEO, CROWDSTRIKE
Valued Customers and Partners,
I want to express my sincere gratitude for the incredible round-the-clock
efforts of our customers and partners who, working alongside our teams,
mobilized immediately to restore systems and bring many back online within
hours.
As of 8:00 p.m. EDT on July 29, 2024, ~99% of Windows sensors were back online,
compared to before the content update and using a week-over-week comparison. We
could not have accomplished so much, so quickly, without your collaboration. We
were on the ground with many of you starting in the early morning of July 19,
working side-by-side to remediate systems. To our customers that are still
affected, please know that we will not rest until all systems are restored.
We are using the lessons learned from this incident to better serve our
customers. To this end, we have already taken decisive steps to help prevent
this situation from repeating, and to help ensure that we — and you — become
even more resilient. The Root Cause Analysis (RCA) and executive summary are
available on our guidance hub and provide more detail on the Channel File 291
incident and how we are further enhancing our processes.
We are deeply sorry for the impact this had on you. Nothing is more important
than regaining your trust and confidence. Since our founding, we have always put
customer protection at the forefront. This has been our North Star, and it
continues to be our focus every single day.
I want to extend my personal thanks to each of you; your continued partnership,
and the countless expressions of support we have received over the past two
weeks, have been incredibly meaningful.
If you have questions or need additional support, please reach out to your
CrowdStrike team.
George Kurtz
CrowdStrike Founder and CEO
Read more
Sent 2024-07-19 1930 UTC
JULY 19, 2024 STATEMENT FROM GEORGE KURTZ, FOUNDER AND CEO, CROWDSTRIKE
Valued Customers and Partners,
I want to sincerely apologize directly to all of you for the outage. All of
CrowdStrike understands the gravity and impact of the situation. We quickly
identified the issue and deployed a fix, allowing us to focus diligently on
restoring customer systems as our highest priority.
The outage was caused by a defect found in a Falcon content update for Windows
hosts. Mac and Linux hosts are not impacted. This was not a cyberattack.
We are working closely with impacted customers and partners to ensure that all
systems are restored, so you can deliver the services your customers rely on.
CrowdStrike is operating normally, and this issue does not affect our Falcon
platform systems. There is no impact to any protection if the Falcon sensor is
installed. Falcon Complete and Falcon OverWatch services are not disrupted.
We will provide continuous updates through our Support Portal.
We have mobilized all of CrowdStrike to help you and your teams. If you have
questions or need additional support, please reach out to your CrowdStrike
representative or Technical Support.
We know that adversaries and bad actors will try to exploit events like this. I
encourage everyone to remain vigilant and ensure that you’re engaging with
official CrowdStrike representatives. Our blog and technical support will
continue to be the official channels for the latest updates.
Nothing is more important to me than the trust and confidence that our customers
and partners have put into CrowdStrike. As we resolve this incident, you have my
commitment to provide full transparency on how this occurred and steps we’re
taking to prevent anything like this from happening again.
George Kurtz
CrowdStrike Founder and CEO
Read more
Updated 2024-08-06 2119 UTC
FREQUENTLY ASKED QUESTIONS
We recognize the July 19, 2024, Channel File 291 incident has been challenging
for customers. We sincerely apologize and thank them for their support and
partnership. As of July 29, 2024, at 5 p.m. PT, ~99% of Windows sensors were
online, compared to before the content update and using a week-over-week
comparison. We typically see a variance of ~1% week-over-week in sensor
connections. To any customers still affected, please know we will not rest until
all systems are restored.
We have now released the Root Cause Analysis (RCA) report, which elaborates on
the information previously shared in our preliminary Post Incident Review,
providing further depth on the findings, mitigations, technical details and root
cause of the Channel File 291 incident.
The below FAQ addresses some common questions about the incident.
Q: WHAT HAPPENED?
On July 19, 2024, a Rapid Response Content update was delivered to certain
Windows hosts, evolving a new sensor capability first released in February 2024
that enabled visibility into possible novel attack techniques that may abuse
certain Windows mechanisms. The sensor expected 20 input fields, and the update
provided 21 input fields. In this instance, the mismatch resulted in an
out-of-bounds memory read, causing a system crash. Our analysis, together with a
third-party review, confirmed this bug is not exploitable by a threat actor. For
more details, please refer to the RCA or read the executive summary above.
Q: HAS THIS ISSUE BEEN FIXED?
Channel File 291 was identified and fixed 78 minutes after it was released, at
1:27 a.m. EDT on July 19. A logic error in our Content Validator (software that
performs control checks on content before deployment) has also been fixed, and
we are implementing additional enhancements that are explained in the RCA by
August 19, 2024.
Each customer environment is unique, with varying degrees of complexity and
architectural constraints that can significantly impact remediation timelines.
We can confirm that almost all systems have restored operations and are back
online. We remain ready to support any customer that continues to experience
operational issues.
Q: HOW DOES RAPID RESPONSE CONTENT MAKE CUSTOMERS MORE SECURE?
Our proven security model is built on the reality that stopping evolving cyber
threats requires: (i) effective threat intelligence and real-time information
about IT infrastructures that are augmented by the experiences of tens of
thousands of enterprises; and (ii) speed of threat identification and response
that is commensurate with the radically accelerated advanced attacks coming from
adversaries.
Rapid Response Content is separate and distinct from CrowdStrike’s on-sensor AI
prevention and detection capabilities, but it is an important part of the
dynamic protection mechanisms of the CrowdStrike Falcon® platform. It fine-tunes
and enhances the sensor’s ability to observe specific behaviors at operational
speed — without requiring changes to the sensor code. Rapid Response Content is
configuration data; it is not code or a kernel driver.
The Falcon platform — which relies on a unique combination of AI, machine
learning and real-time rapid response intelligence — protects customer systems
by identifying and remediating the latest advanced threats. That means our
customers get the highest level of protection against increasingly sophisticated
bad actors. In doing so, we take a comprehensive approach that prioritizes both
rigorous testing and rapid response to emerging threats.
Q: WHAT IS CROWDSTRIKE DOING TO HELP ENSURE THIS DOESN’T HAPPEN IN THE FUTURE?
While this scenario with Channel File 291 is now incapable of recurring, it
informs the process improvements and mitigation steps that CrowdStrike is
deploying to help ensure further enhanced resilience.
Based on the findings in the RCA, here are some of the actions CrowdStrike has
taken and will take moving forward:
* Update Content Configuration System test procedures. This work has been
completed. This includes upgraded tests for Template Type development, with
automated tests for all existing Template Types. Template Types are part of
the sensor and contain predefined fields for threat detection engineers to
leverage in Rapid Response Content.
* Add additional deployment layers and acceptance checks for the Content
Configuration System. This work has been completed with an updated deployment
ring process, ensuring Template Instances pass successive deployment rings
before rollout into production.
* Provide customers additional control over the deployment of Rapid Response
Content updates. New capabilities have been implemented and deployed to our
cloud that allow customers to control how Rapid Response Content is deployed,
with additional functionality planned for the future.
* Prevent the creation of problematic Channel 291 files. Validation for the
number of input fields has been implemented to prevent this issue from
happening.
* Implement additional checks in the Content Validator. Additional checks are
planned for release into production by August 19, 2024.
* Enhance bounds checking in the Content Interpreter for Rapid Response Content
in Channel File 291. Bounds checking was added on July 25, 2024, with general
availability expected August 9, 2024. These fixes are being backported to all
Windows sensor versions 7.11 and above through a sensor software hotfix
release.
* Engage two independent third-party software security vendors to conduct
further review of the Falcon sensor code and end-to-end quality control and
release processes. This work has begun and will be ongoing as part of our
focus on security and resilience by design.
For additional detail on the actions CrowdStrike has taken and will take moving
forward to help ensure this doesn’t happen in the future, please refer to the
RCA.
Q: HOW DID CROWDSTRIKE ENGAGE WITH CUSTOMERS AND PARTNERS TO MINIMIZE IMPACT?
CrowdStrike began working with customers and partners to bring systems online as
quickly as possible, initially through manual remediation. These efforts enabled
the systems to come back online within hours following the initial incident.
On July 22, 2024, CrowdStrike introduced automated techniques to accelerate
remediation.
To further help customers bring systems online as quickly as possible,
CrowdStrike deployed personnel and engaged with strategic partner services teams
to assist customers with recovery efforts. We also worked to provide continuous
and transparent updates to customers throughout our response. As of July 29,
2024, at 8:00 p.m. EDT, ~99% of Windows sensors were online, compared to before
the content update. We typically see a variance of ~1% week-over-week in sensor
connections.
Q: HOW SHOULD CUSTOMERS THINK ABOUT THE COMPANY’S FINANCIAL STRENGTH?
CrowdStrike has always managed the business with financial discipline, and this
is showcased by our financial strength.
As of April 30, 2024, the end of our first quarter fiscal year 2025, we had cash
and cash equivalents of $3.7 billion. The company also has a $750 million
revolving credit facility.
By delivering value through our industry-leading platform and building
resiliency, we create strong and enduring relationships with our customers. For
the trailing 12 months ending April 30, 2024, CrowdStrike generated over $1
billion in cash flow, which we believe will enable us to continue investing in
the business and cover potential legal liabilities. In addition, we maintain
insurance policies that are intended to mitigate the potential impact of certain
claims.
Our standard terms and conditions related to customer contracts, including
limitations of liability, are laid out on our website
(https://www.crowdstrike.com/terms-conditions/).
Q: DOES CROWDSTRIKE FALCON HAVE KERNEL ACCESS TO THE WINDOWS OPERATING SYSTEM?
IS THIS STANDARD FOR THE INDUSTRY, AND WILL IT CONTINUE?
Yes. CrowdStrike Falcon, like other cybersecurity products, runs parts of its
logic in the kernel of the Windows operating system. Presence in the kernel
offers rich visibility into system-wide security-relevant activities, such as
process and thread creation or files being written, deleted and modified on
disk. This access provides maximum protection against cyber threats.
CrowdStrike certifies each new sensor release, including the latest versions of
all channel files at the time of certification, through Microsoft’s Windows
Hardware Quality Labs (WHQL) program. This includes extensive testing through
Microsoft’s Windows Hardware Lab Kit (Windows HLK).
The WHQL certification process marks the end of a comprehensive internal testing
gauntlet involving functional tests, longevity tests, stress tests with fault
injection, fuzzing and performance tests. During the testing required for the
WHQL program, the sensors use the latest versions of channel files at the time
of certification. Recent reports that any kernel-related processes were bypassed
are false. For more information, please see the RCA.
We are unaware of any plans for Microsoft to remove kernel access from
CrowdStrike or any other cybersecurity company.
Q: DID NULL BYTES IN CHANNEL FILE 291 CAUSE THE INCIDENT?
No. For additional information, please read the following: Tech Analysis:
Channel File May Contain Null Bytes.
Read more
Updated 2024-07-24 0335 UTC
PRELIMINARY POST INCIDENT REVIEW
CONTENT CONFIGURATION UPDATE IMPACTING THE FALCON SENSOR AND THE WINDOWS
OPERATING SYSTEM (BSOD)
Executive Summary PDF
This is CrowdStrike’s preliminary Post Incident Review (PIR). We will be
detailing our full investigation in the forthcoming Root Cause Analysis that
will be released publicly. Throughout this PIR, we have used generalized
terminology to describe the Falcon platform for improved readability.
Terminology in other documentation may be more specific and technical.
WHAT HAPPENED?
On Friday, July 19, 2024 at 04:09 UTC, as part of regular operations,
CrowdStrike released a content configuration update for the Windows sensor to
gather telemetry on possible novel threat techniques.
These updates are a regular part of the dynamic protection mechanisms of the
Falcon platform. The problematic Rapid Response Content configuration update
resulted in a Windows system crash.
Systems in scope include Windows hosts running sensor version 7.11 and above
that were online between Friday, July 19, 2024 04:09 UTC and Friday, July 19,
2024 05:27 UTC and received the update. Mac and Linux hosts were not impacted.
The defect in the content update was reverted on Friday, July 19, 2024 at 05:27
UTC. Systems coming online after this time, or that did not connect during the
window, were not impacted.
WHAT WENT WRONG AND WHY?
CrowdStrike delivers security content configuration updates to our sensors in
two ways: Sensor Content that is shipped with our sensor directly, and Rapid
Response Content that is designed to respond to the changing threat landscape at
operational speed.
The issue on Friday involved a Rapid Response Content update with an undetected
error.
Sensor Content
Sensor Content provides a wide range of capabilities to assist in adversary
response. It is always part of a sensor release and not dynamically updated from
the cloud. Sensor Content includes on-sensor AI and machine learning models, and
comprises code written expressly to deliver longer-term, reusable capabilities
for CrowdStrike’s threat detection engineers.
These capabilities include Template Types, which have pre-defined fields for
threat detection engineers to leverage in Rapid Response Content. Template Types
are expressed in code. All Sensor Content, including Template Types, go through
an extensive QA process, which includes automated testing, manual testing,
validation and rollout steps.
The sensor release process begins with automated testing, both prior to and
after merging into our code base. This includes unit testing, integration
testing, performance testing and stress testing. This culminates in a staged
sensor rollout process that starts with dogfooding internally at CrowdStrike,
followed by early adopters. It is then made generally available to customers.
Customers then have the option of selecting which parts of their fleet should
install the latest sensor release (‘N’), or one version older (‘N-1’) or two
versions older (‘N-2’) through Sensor Update Policies.
The event of Friday, July 19, 2024 was not triggered by Sensor Content, which is
only delivered with the release of an updated Falcon sensor. Customers have
complete control over the deployment of the sensor — which includes Sensor
Content and Template Types.
Rapid Response Content
Rapid Response Content is used to perform a variety of behavioral
pattern-matching operations on the sensor using a highly optimized engine. Rapid
Response Content is a representation of fields and values, with associated
filtering. This Rapid Response Content is stored in a proprietary binary file
that contains configuration data. It is not code or a kernel driver.
Rapid Response Content is delivered as “Template Instances,” which are
instantiations of a given Template Type. Each Template Instance maps to specific
behaviors for the sensor to observe, detect or prevent. Template Instances have
a set of fields that can be configured to match the desired behavior.
In other words, Template Types represent a sensor capability that enables new
telemetry and detection, and their runtime behavior is configured dynamically by
the Template Instance (i.e., Rapid Response Content).
Rapid Response Content provides visibility and detections on the sensor without
requiring sensor code changes. This capability is used by threat detection
engineers to gather telemetry, identify indicators of adversary behavior and
perform detections and preventions. Rapid Response Content is behavioral
heuristics, separate and distinct from CrowdStrike’s on-sensor AI prevention and
detection capabilities.
Rapid Response Content Testing and Deployment
Rapid Response Content is delivered as content configuration updates to the
Falcon sensor. There are three primary systems: the Content Configuration
System, the Content Interpreter and the Sensor Detection Engine.
The Content Configuration System is part of the Falcon platform in the cloud,
while the Content Interpreter and Sensor Detection Engine are components of the
Falcon sensor. The Content Configuration System is used to create Template
Instances, which are validated and deployed to the sensor through a mechanism
called Channel Files. The sensor stores and updates its content configuration
data through Channel Files, which are written to disk on the host.
The Content Interpreter on the sensor reads the Channel File and interprets the
Rapid Response Content, enabling the Sensor Detection Engine to observe, detect
or prevent malicious activity, depending on the customer’s policy configuration.
The Content Interpreter is designed to gracefully handle exceptions from
potentially problematic content.
Newly released Template Types are stress tested across many aspects, such as
resource utilization, system performance impact and event volume. For each
Template Type, a specific Template Instance is used to stress test the Template
Type by matching against any possible value of the associated data fields to
identify adverse system interactions.
Template Instances are created and configured through the use of the Content
Configuration System, which includes the Content Validator that performs
validation checks on the content before it is published.
Timeline of Events: Testing and Rollout of the InterProcessCommunication (IPC)
Template Type
Sensor Content Release: On February 28, 2024, sensor 7.11 was made generally
available to customers, introducing a new IPC Template Type to detect novel
attack techniques that abuse Named Pipes. This release followed all Sensor
Content testing procedures outlined above in the Sensor Content section.
Template Type Stress Testing: On March 05, 2024, a stress test of the IPC
Template Type was executed in our staging environment, which consists of a
variety of operating systems and workloads. The IPC Template Type passed the
stress test and was validated for use.
Template Instance Release via Channel File 291: On March 05, 2024, following the
successful stress test, an IPC Template Instance was released to production as
part of a content configuration update. Subsequently, three additional IPC
Template Instances were deployed between April 8, 2024 and April 24, 2024. These
Template Instances performed as expected in production.
What Happened on July 19, 2024?
On July 19, 2024, two additional IPC Template Instances were deployed. Due to a
bug in the Content Validator, one of the two Template Instances passed
validation despite containing problematic content data.
Based on the testing performed before the initial deployment of the Template
Type (on March 05, 2024), trust in the checks performed in the Content
Validator, and previous successful IPC Template Instance deployments, these
instances were deployed into production.
When received by the sensor and loaded into the Content Interpreter, problematic
content in Channel File 291 resulted in an out-of-bounds memory read triggering
an exception. This unexpected exception could not be gracefully handled,
resulting in a Windows operating system crash (BSOD).
HOW DO WE PREVENT THIS FROM HAPPENING AGAIN?
SOFTWARE RESILIENCY AND TESTING
* Improve Rapid Response Content testing by using testing types such as:
* Local developer testing
* Content update and rollback testing
* Stress testing, fuzzing and fault injection
* Stability testing
* Content interface testing
* Add additional validation checks to the Content Validator for Rapid Response
Content. A new check is in process to guard against this type of problematic
content from being deployed in the future.
* Enhance existing error handling in the Content Interpreter.
RAPID RESPONSE CONTENT DEPLOYMENT
* Implement a staggered deployment strategy for Rapid Response Content in which
updates are gradually deployed to larger portions of the sensor base,
starting with a canary deployment.
* Improve monitoring for both sensor and system performance, collecting
feedback during Rapid Response Content deployment to guide a phased rollout.
* Provide customers with greater control over the delivery of Rapid Response
Content updates by allowing granular selection of when and where these
updates are deployed.
* Provide content update details via release notes, which customers can
subscribe to.
Updated 2024-07-24 2217 UTC
THIRD PARTY VALIDATION
* Conduct multiple independent third-party security code reviews.
* Conduct independent reviews of end-to-end quality processes from development
through deployment.
In addition to this preliminary Post Incident Review, CrowdStrike is committed
to publicly releasing the full Root Cause Analysis once the investigation is
complete.
Read more
Updated 2024-08-05 1254 UTC
Tech Alerts keep you updated on important events requiring timely action. To be
notified of new Tech Alerts via email, please opt-in through your profile
settings in the Support Portal.
TECHNICAL DETAILS
* Technical Details on the outage can be found here: Read the blog Published
2024-07-20 0100 UTC
* We assure our customers that CrowdStrike is operating normally and this issue
does not affect our Falcon platform systems. If your systems are operating
normally, there is no impact to their protection if the Falcon Sensor is
installed. Falcon Complete and OverWatch services are not disrupted by this
incident.
* CrowdStrike has identified the trigger for this issue as a Windows sensor
related content deployment and we have reverted those changes. The content is
a channel file located in the %WINDIR%\System32\drivers\CrowdStrike
directory.
* Channel file “C-00000291*.sys” with timestamp of 2024-07-19 0527 UTC or
later is the reverted (good) version.
* Channel file “C-00000291*.sys” with timestamp of 2024-07-19 0409 UTC is the
problematic version.
* Note: It is normal for multiple “C-00000291*.sys files to be present in
the CrowdStrike directory – as long as one of the files in the folder has
a timestamp of 05:27 UTC or later, that will be the active content.
* Symptoms include hosts experiencing a bugcheck\blue screen error related to
the Falcon Sensor.
* Windows hosts which have not been impacted do not require any action as the
problematic channel file has been reverted.
Updated 2024-07-24 0335 UTC
FILE CLASSIFICATION STATUS
The channel file responsible for system crashes on Friday, July 19, 2024
beginning at 04:09 UTC was identified and deprecated on operational systems.
When deprecation occurs, a new file is deployed, but the old file can remain in
the sensor’s directory.
Out of an abundance of caution, and to prevent Windows systems from further
disruption, the impacted version of the channel file was added to Falcon’s
known-bad list in the CrowdStrike Cloud.
No sensor updates, new channel files, or code was deployed from the CrowdStrike
Cloud.
For operational machines, this is a hygiene action. For impacted systems with
strong network connectivity, this action could also result in the automatic
recovery of systems in a boot loop.
This was configured in US-1, US-2, and EU on July 23, 2024 UTC.
Gov-1 and Gov-2 customers can request a channel file 291 known-bad
classification by contacting CrowdStrike Support.
NON-IMPACTED HOSTS
* Windows hosts which are brought online after 2024-07-19 0527 UTC will not be
impacted
* Windows hosts installed and provisioned after 2024-07-19 0527 UTC are not
impacted
Updated 2024-07-21 1435 UTC
* This issue is not impacting Mac- or Linux-based hosts
HOW DO I REMEDIATE?
Follow the steps below to learn how to remediate and identify impacted hosts.
Updated 2024-07-22 0139 UTC
HOW DO I IDENTIFY IMPACTED HOSTS VIA ADVANCED EVENT SEARCH QUERY?
The queries utilized by the dashboards are listed at the bottom of the
appropriate dashboard manuals.
Updated 2024-07-23 0217 UTC
HOW DO I IDENTIFY IMPACTED HOSTS VIA DASHBOARD?
An updated granular dashboard is available that displays the Windows hosts
impacted by the content update defect described in this Tech Alert. See Granular
status dashboards to identify Windows hosts impacted by content issue (v8.6)
(pdf) or log in to view in the support portal. Note that the queries utilized by
the dashboards are listed at the bottom of the appropriate dashboard manuals.
Watch the following walkthrough video on Identifying Possibly Impacted Hosts
with the Updated Granular Dashboard. Updated 2024-07-25 1810 UTC
HOW DO I REMEDIATE IMPACTED HOSTS?
If hosts are still crashing and unable to stay online to receive the Channel
File update, the remediation steps below can be used.
Updated 2024-07-21 0932 UTC
HOW DO I REMEDIATE INDIVIDUAL HOSTS?
* Reboot the host to give it an opportunity to download the reverted channel
file. We strongly recommend putting the host on a wired network (as opposed
to WiFi) prior to rebooting as the host will acquire internet connectivity
considerably faster via ethernet.
* If the host crashes again on reboot:
Updated 2024-07-22 1758 UTC
* Option 1 – Build automated recovery ISOs with drivers
* Follow the instructions for Building CrowdStrike Bootable Recovery Images
in this manual (PDF) or log in to view in the support portal. Updated
2024-07-26 2105 UTC
* Note: Bitlocker-encrypted hosts may require a recovery key.
* Review the following video on CrowdStrike Host Remediation with
Bootable USB Drive.
* * Option 2 – Manual process
* Review the following video on CrowdStrike Host Self-Remediation for
Remote Users. Follow the instructions contained within the video if
directed to do so by your organization’s IT department. Updated
2024-07-22 1510 UTC
* Alternatively, please see this Microsoft article for detailed steps.
* Note: Bitlocker-encrypted hosts may require a recovery key.
Updated 2024-07-23 2230 UTC
REPAIRING FALCON WINDOWS SENSORS
* Repairing Falcon Windows Sensors (PDF) or log in to view in the support
portal – guidance on repairing a broken sensor following deletion or renaming
of sensor files and/or folders.
Updated 2024-07-23 1930 UTC
* Removing locked memory.dmp files after successful remediation (PDF) or log in
to view in the support portal
Updated 2024-07-27 1803 UTC
Updated 2024-07-21 1810 UTC
HOW DO I RECOVER BITLOCKER KEYS?
Bitlocker Recovery Manuals
Resources
Updated 2024-07-21 1810 UTC
Microsoft Azure
(PDF) or log in to view in the support portal.
Updated 2024-07-21 1810 UTC
SCCM
(PDF) or log in to view in the support portal.
Updated 2024-07-21 1810 UTC
Active Directory and GPOs
(PDF) or log in to view in the support portal.
Updated 2024-07-21 1810 UTC
Ivanti Endpoint Manager
(PDF) or log in to view in the support portal.
Updated 2024-07-21 1810 UTC
ManageEngine Desktop Central
(PDF) or log in to view in the support portal.
Updated 2024-07-21 1810 UTC
BigFix
(PDF) or log in to view in the support portal.
Updated 2024-07-21 0023 UTC
Bitlocker recovery without recovery keys
(PDF) or log in to view in the support portal.
Workspace ONE Portal
Omnissa article
Tanium
Tanium article
Citrix
Citrix article
HOW DO I RECOVER CLOUD–BASED ENVIRONMENTS?
Cloud Environment
Guidance
AWS
AWS article
Azure
Microsoft article
Updated 2024-07-22 1758 UTC
GCP
* See instructions for Manual Recovery from Blue Screen on Windows Instances in
GCP (PDF) or log in to view in the support portal
* See GCP CrowdStrike File Remediation Script – provides a Python script
customers can use to remediate impacted hosts residing in the GCP.
Public Cloud/Virtual Environments
Option 1:
* Detach the operating system disk volume from the impacted virtual server
* Create a snapshot or backup of the disk volume before proceeding further as a
precaution against unintended changes
* Attach/mount the volume to to a new virtual server
* Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory
* Locate the files matching “C-00000291*.sys”, and delete them
* Detach the volume from the new virtual server
* Reattach the fixed volume to the impacted virtual server
Option 2:
* Roll back to a snapshot before 2024-07-19 0409 UTC
Updated 2024-07-20 2259 UTC
THIRD PARTY VENDOR INFORMATION
Third Party Vendor
Guidance
Intel vPro technology remediation guide
Remediate CrowdStrike Falcon® update issue on Windows systems with Intel vPro®
technology
Absolute Software Support
Steps to Repair BSOD Devices and Run Corrupted File Detection Procedure
Recovery for Rubrik customers
CrowdStrike & Rubrik Customer Content Update Recovery For Windows Hosts
Cohesity Support
Cohesity’s support for CrowdStrike’s Falcon Sensor updates
REMEDIATION VIDEOS
CROWDSTRIKE HOST SELF-REMEDIATION FOR REMOTE USERS
This video outlines the steps required to self-remediate impacted remote Windows
laptops. Follow these instructions if directed to do so by your organization’s
IT department.
IDENTIFY POSSIBLY IMPACTED HOSTS WITH CROWDSTRIKE DASHBOARD
This video is an overview of the dashboard available for CrowdStrike Insight
customers to identify possibly impacted devices related to the recent defect in
a CrowdStrike content update for Windows hosts. For more information on this
dashboard, please visit the CrowdStrike Remediation and Guidance Hub.
CROWDSTRIKE HOST REMEDIATION WITH BOOTABLE USB DRIVE
This video shows you how to use the Falcon Windows Host Recovery project to
build bootable USB drives to remediate Windows hosts impacted by the recent
Falcon Content Update.
ADDITIONAL RESOURCES
Published 2024-07-24 2207 UTC
TECH ANALYSIS: CHANNEL FILE MAY CONTAIN NULL BYTES
Read now
Published 2024-07-30 1636 UTC
INTEL: MALICIOUS INAUTHENTIC FALCON CRASH REPORTER INSTALLER DELIVERS LLVM-BASED
MYTHIC C2 AGENT NAMED CIRO
Read now
Published 2024-07-25 0200 UTC
GLOSSARY OF TERMS
Download now
Published 2024-07-25 2216 UTC
INTEL: MALICIOUS INAUTHENTIC FALCON CRASH REPORTER INSTALLER DISTRIBUTED TO
GERMAN ENTITY VIA SPEARPHISHING WEBSITE
Read now
Published 2024-07-23 2249 UTC
INTEL: THREAT ACTOR DISTRIBUTES PYTHON-BASED INFORMATION STEALER USING A FAKE
FALCON SENSOR UPDATE LURE
Read now
Published 2024-07-22 1953 UTC
INTEL: THREAT ACTOR USES FAKE RECOVERY MANUAL TO DELIVER UNIDENTIFIED STEALER
Read now
Published 2024-07-20 0145 UTC
INTEL: LIKELY ECRIME ACTOR USES FILENAMES CAPITALIZING ON FALCON SENSOR CONTENT
ISSUES IN OPERATION TARGETING LATAM-BASED CROWDSTRIKE CUSTOMERS
Read now
Published 2024-07-20 0100 UTC
TECHNICAL DETAILS: FALCON CONTENT UPDATE FOR WINDOWS HOSTS
Read now
Published 2024-07-19 2030 UTC
INTEL: FALCON SENSOR CONTENT ISSUE LIKELY USED TO TARGET CROWDSTRIKE CUSTOMERS
Read now
Contact us
CrowdStrike is the leader in next-generation endpoint protection, threat
intelligence and response services. CrowdStrike’s core technology, the Falcon
platform, stops breaches by preventing and responding to all types of attacks —
both malware and malware-free.
Reach out
*
*
*
*
*
New to CrowdStrike? About the platform Explore products Services Why choose
CrowdStrike? CrowdStrike Financial Services
Company About CrowdStrike Careers Events Newsroom Partners CrowdStrike
Marketplace
Learn with CrowdStrike 2024 Global Threat Report Cybersecurity 101 Your Threat
Landscape Tech Center View all resources
Contact us Experienced a breach?
Copyright © 2024
* Contact us
* Privacy
* Cookies
* Your Privacy Choices
* Terms of Use
* Accessibility
ABOUT COOKIES ON THIS SITE
By clicking “Accept All Cookies”, you agree to the storing of cookies on your
device to enhance site navigation, analyze site usage, and assist in our
marketing efforts. Cookie Notice
Cookie Settings Reject All Accept All Cookies
COOKIE PREFERENCE CENTER
* YOUR PRIVACY
* STRICTLY NECESSARY COOKIES
* FUNCTIONAL COOKIES
* PERFORMANCE COOKIES
* TARGETING COOKIES
YOUR PRIVACY
When you visit any website, it may store or retrieve information on your
browser, mostly in the form of cookies. This information might be about you,
your preferences or your device and is mostly used to make the site work as you
expect it to. The information does not usually directly identify you, but it can
give you a more personalized web experience. Because we respect your right to
privacy, you can choose not to allow some types of cookies. Click on the
different category headings to find out more and change our default settings.
However, blocking some types of cookies may impact your experience of the site
and the services we are able to offer.
More information.
STRICTLY NECESSARY COOKIES
Always Active
These cookies are necessary for the website to function and cannot be switched
off in our systems. This includes diagnostic functions such as identifying 404
errors and monitoring page load speed. They are usually only set in response to
actions made by you which amount to a request for services, such as setting your
privacy preferences, logging in or filling in forms. You can set your browser to
block or alert you about these cookies, but some parts of the site will not then
work. These cookies do not store any personally identifiable information.
Cookies Details
FUNCTIONAL COOKIES
Functional Cookies
These cookies enable the website to provide enhanced functionality and
personalisation. They may be set by us or by third party providers whose
services we have added to our pages. If you do not allow these cookies then some
or all of these services may not function properly.
Cookies Details
PERFORMANCE COOKIES
Performance Cookies
These cookies allow us to count visits and traffic sources so we can measure and
improve the performance of our site. They help us to know which pages are the
most and least popular and see how visitors move around the site. All
information these cookies collet is aggregated and therefore anonymous. If you
do not allow these cookies we will not know when you have visited our site, and
will not be able to monitor its performance.
Cookies Details
TARGETING COOKIES
Targeting Cookies
These cookies may be set through our site by our advertising partners. They may
be used by those companies to build a profile of your interests and show you
relevant adverts on other sites. They do not store directly personal
information, but are based on uniquely identifying your browser and internet
device. If you do not allow these cookies, you will experience less targeted
advertising.
Cookies Details
Back Button
COOKIE LIST
Filter Button
Consent Leg.Interest
checkbox label label
checkbox label label
checkbox label label
Clear
checkbox label label
Apply Cancel
Confirm My Choices
Allow All