Testing Garage: My First Hand Analysis of CrowdStrike Falcon Update Incident

I attempted to analyze the process dump of CrowdStrike shared by my friend. He said, there could be an attack which is leading to crash of Windows OS globally. This made me curious to look into the dump and learn.

I had no much context around it, but, a test engineer in me did not sit quite. I started to analyze the dump information. Here is my first hand analysis that I made on 19th July 2024 post 10:30 AM IST.

What I Saw?

It is a Windows OS's process dump.
Looks like something with C or C++ application reading how the memory offsets were in the dump.
It started to read a memory offset.
Then the process witnessed an exception.

Here the program could not read further
Why it could not read further from this offset?

My little experience of testing drivers on Windows OS for a card printer machine, refreshed and recalled what I had witnessed when testing.

Scratching and Striking My Mind

I started to ask these questions myself while I asked what could have gone wrong. I could not stop here as I was curious what led Windows machine crash. I referred to web and learn there was an update by CrowdStrike, and then this incident.

The bugs do exist in every software no matter the level and depth of testing, automation and engineering's excellence. All software do crash and OS is not an exception to it. But, what made the update to crash the Windows OS? Pointing and blaming CrowdStrike or Microsoft is not a way for the practicing test engineer. If these two organizations are serving its huge customer base, they have something working and reliable. Engineering does not eliminate problems.

By now, I had a thought that it is not an attack. It is a software bug! Where is the bug? What is the bug? Was it not experienced in pipeline?

The Open Ended Questions

I had these questions as I analyzed and spoke to my friend.

What is Falcon?
What was this update to Falcon?
How frequently the updates are rolled out?
How the updates are rolled out globally?
What pipeline do they have in testing?
Who is impacted the most in business? Is it Microsoft or CrowdStrike? Impacted in what way?
What is CrowdStrike? What they do? Who are the customers?
Where do the CrowdStrike's Falcon sit in the OS and what it does?
How CrowdStrike works in the machines and what it offers?
What do the dump say? Relook into it with different perspectives.
How this could have been prevented?
How will I prevent this if I join this team knowing this incident?

With these questions, I started to analyze the process dump which was shared.

I had more such questions, but these were the first few that I crossed as I started.

Analysis of Process Dump

My interpretation, tells me the below for today

Accept that it is an incident as any other incident which I witness in production environment.
Do not fall to the speculation happening around. Remain calm and focus to interpret and understand your exploration.
I see, if it can start to read from an offset and then ending to experience a non-existent or invalid offset, is it a NULL Pointer?

What is NULL Pointer?

A NULL Pointer is a pointer that does NOT point to any memory location and hence does not hold the address of any variables.
If I do not initialize and assign, the pointer will have NULL as its value.
For example, int *test;

When I want to access the pointer test (a location in memory) pointing to, I will not be sure what is in the pointer when I read it.

I may not set it later or set it.
In this case, the code can tell if the pointer is valid or pointing to a garbage memory

But, if I declare it like int *test = NULL;

I can check if was set and initialized

It is a better practice to assign a NULL value to a pointer during initialization so that we can check if it is NULL or as any address assigned to it.

This understanding of Pointer makes me think, is it due not initializing a pointer and so the error code c0000005 on reading a memory that is not valid.
When we assign a NULL value to pointer, it is a null pointer in C++

We assign null value for testing and asserting

If the memory is allocated to a pointer or not
If it has a return address and is a valid one or not
If a pointer is not initialized, assigning null it prevents problems to certain extent

With this understanding, I also read, it started to read from an offset 0x9c, and then failing.

What is 0x9c?

In Octal it is 234. In Decimal it is 156.
Can there be such address in a computer's memory? I don't know.

If it is a access violation, then is it a memory which is in preemption of the OS?

If so the OS can terminate the program or process which is trying to access it.
Is this killing the process and aborting the operation of Falcon's IPC and eventually Windows coming to BSOD?

This tells me it is not a NULL Pointer in first case but not initializing a pointer to NULL.

I infer, if the pointer was assigned to NULL, that is initialized, there could have been some hint in the state and event when accessing the memory.

This is my analysis; but, I have not seen the test code nor aware of the product. All this inference is based on the process dump and my experience of testing drivers.

It got something in between from update (a config or pattern?) for which it cannot find and read in the memory? Why?

This indicates me, it could be a bug, that is, a logical problem. This is my hunch for today!

Data in the dump

Exception Address
Read from Address 0x9c
Exception Code: c0000005 (Access violation)

Testing my Interpretations

CrowdStrike as an org when it caters its SAAS to such a customer base, won't it have a testing pipeline

It will have, I have no doubt in it. They test and roll out the updates, I believe in it.

Did they witness any such incidents earlier?

I searched on web for it and I did not find something similar on the Windows, earlier.

Is this a NULL Pointer? Are you sure?

No, I'm not sure. But, there is something that is leading it to address which does not exist or which is invalid? I will have to wait for their RCA to know technically what caused this. But this is my understanding reading the dump.

How do you think it is a memory access problem?

The error code 0xc0000005 says that.
I referred to driver easy website for the information because my experience of testing the drivers for Windows OS and experiencing such incidents led me there. This is what I learn:

https://www.drivereasy.com/knowledge/solved-how-to-fix-0xc0000005-error/

Do you think the programmer would not have handled the obvious Pointer and NULL initialization?

I believe there will be a check for Pointer and what it is pointing to. But is it due to no initialization? Technically this has to be analyzed which I cannot do. I will have to wait for CrowdStrike team to share the tech details.

Is this a driver problem that killed the Windows kernel?

I don't know. But, the .sys file will not have driver as per my learning. It will have information about the drivers and any configurations.
This incident is a problem, which impacted both CrowdStrike and Microsoft. Maybe, both will have their areas to look and fix it they see so. But, in this context, CrowdStrike can fix it quicker and that is much better -- is what I understand.
I'm a Windows user for long time. I see, Windows has worked well to all my contexts so far. The Engineers of Windows OS knows better than me here. I'm not well aware and informed as they are.
CrowdStrike's engineering team are skilled and they are rolling out updates often in a day. They have a better pipeline when this is being done.

But, the question I have is, how did this happen?
No one lets such problem into production when they are aware of it. Do you?
There is something that has not come to their observation and experience. What is that?
Knowing this will help to prevent this and similar incidents happening in future.

I'm waiting to know what did not come to their experience and led to this incident.

What could be in the .sys file of CrowdStrike?

I don't know! I want to learn that.
But, from my testing of .sys file and drivers on Windows OS, I learn there could be a configuration details with certain pattern or information to capture at run time, and help the installed software to run. This is my learning and awareness from my testing.
That said, testing at OS level and Anti Virus engines are not obvious. Testing of drivers is like the risky mines. What is sufficient and good enough in test coverage? It needs an expertise at OS internals level.
Windows OS having such a fragmentation in its versions, updates and patches, it is a battle field and mines for engineers building such solutions for sure!
I learn, the Windows OS stopped when an application tried to access the invalid region or non-existent memory.

The update which was rolled out, did it have a configuration or a pattern that showed a logical problem when processing it?
I have such questions and thoughts that are striking my mind as I think and build a problem model for the same.

Is this a race condition incident?

I see, it is not a race condition incident as users across globe experienced it.

Is this specific to a Falcon version, OS version and hardware?

Not all host machines would be on latest version of Falcon, is my presumption.
At least, n-1 and n-2 versions should be on host machine which experienced this behavior.

So it is not a Falcon version specific, I see.

It looks to me as it is not specific to the Windows OS version and hardware configuration.

It is an application software problem which occurred at driver level is what I see.

This is an IPC communication and process is my understanding.

The driver can receive the IPC communication in continuous mode.
At times, this can get queued based on the application and what it does.

Where is the Problem?

Well, I'm looking and pulling from my visualization by relating with my experience of testing the driver on Windows OS. I don't know the exact reason or close enough to tell what could have gone wrong.

Reading the process dump, it says accessing a memory that does not exist or corrupted. One of the high possibility is, the starting offset is seen but it is not helping when reading.

For example, Ravi has the address of India's Prime Minister house.

But, he does not know from where to start despite having the address.
He is void and null in knowing where to start and what to do when he is not initialized with the start location to begin the travel to the Prime Minister's house.
In short, he do not know where the address is pointing to and what it has, though he is given a address to start.

Can he access the Prime Minister's house premise without any access granted and authorized to do so?
If not, won't he be arrested by police or other security forces and stop him?

Do I Know the Precise Problem?

I don't know! I do not know the CrowdStrike product and platform. I'm waiting to read the technical details from Crowd Strike.

I see, it comes to the data, state and event. I would focus on how to prevent it learning which data, state and event led to this behavior. I think of figuring out the Test Design and Strategy that can help me to identify such use cases. I focus here and see can it brought into the automation so that it gets exercised and regressed consistently.

If it is due to the memory access that had a problem, I did such tests when testing driver for a hardware machine on Windows OS. I will share the tests that I did in upcoming blog.

I wrote the technical analysis from process dump to CrowdStrike and Microsoft. I did not get a response. Anyways, I'm sharing the overall information in a non-technical way so that it is consumable to most readers here.

Note: Here are another threads of me sharing my thoughts on same:

1. https://x.com/testingGarage/status/1814215089525821763?t=XSFdx69ElL0ZmBOcEFrTjg&s=19

2. https://www.linkedin.com/posts/ravisuriya_%3F%3F%3F%3F%3F%3F%3F-%3F%3F%3F%3F%3F%3F%3F%3F%3F%3F%3F-%3F-activity-7221156949445206017-oeRa