Widespread system outage due to CrowdStrike

[Krash]

Originally Posted by Blansky

I guess not.

Why do some on this site have it and were affected on their home computers.

Was is because they are tied to their company network or software which does have it?

This CrowdStrike Falcon software is for large corporations. I’d be surprised if any home consumer is impacted by this. Not every company uses it either. But it’s obviously popular in the airline industry.


Sent from my iPhone using Tapatalk

[INC]

I think we've been waiting for this kind of bug since the Y2K era, so I hope the top deceision makers will finally start seriously thinking about how to make automatic online updates secure.

After all, this was "just" a mistake. But now imagine that someone packs a ransomware into an update package...

[Boothroyd]

8.5M PC's affected worldwide. I thought it would be more, but I was one of the "lucky" ones who got it. We have roughly 2500 PC's at the company I work for (the rest are Apple), and about 100 had the issue, according to our IT team.

[Blansky]

Originally Posted by huncho

they had issues because they were either using their work laptop or remoting into work from whatever computer they have at home. crowdstrike is used for enterprises so it has to be installed on the computer you're using or trying to access for you to be impacted. a personal computer won't have the issue

Got it. Thanks.

[ltmgeller]

Seems excessive to be unintentional.

[Krash]

Originally Posted by Easy E

My understanding is that a .sys file deposited in the update was completely full of null characters. That would produce the exception pointer bsod. If that is true, this situation is much different that just a bad memory address call. Much different.

The thing that’s so bad about this is that it’s causing a catastrophic failure at the operating system level. If Falcon itself crashed, that’s one thing, but bringing down the operating system with it is why this is such a catastrophe. Everyone thought it was a Microsoft problem at first.


Sent from my iPhone using Tapatalk

[Maleg]

Originally Posted by ltmgeller

Seems excessive to be unintentional.

A hallucinating AI programmer, maybe?

[hdrazor251]

The fix is actually very simple.

1. Boot Windows in Safe Mode
2. Navigate to the C:\Windows\System32\drivers\Crownstrike directory
3. Locate and delete the file “C-00000291*.sys”
4. Boot the machine normally

Our biggest complicating factor was BitLocker as Boothroyd (hi Daniel) mentioned earlier. We had to look-up the unique 48 character recovery code for each machine before we could boot in Safe Mode. So it wasn’t as simple as walking the floor and fixing what we saw was broke. We had to get the device name, go back and look-up the code for that device, then back to the device to apply the fix.

As for what broke and what didn’t, it just depended if the device was powered on when the bad one was pushed by CrowdStrike. The bad “C-00000291*.sys” file had a timestamp of 0409 UTC. They later sent a good “C-00000291*.sys” file that had a timestamp of 0527 UTC but if you already had the bad one you were screwed.

[Nice marmot]

Originally Posted by RTG

What a mess.Hopefully CRWD shareholders are still happy with the company’s DEI commitment because you know, hiring the best and brightest is just racist. Let’s destroy all of our customers computers, and their weekends, because that’s a great idea. Bargain buyers beware, this stock will get hammered Monday.

https://www.crowdstrike.com/careers/...and-inclusion/

So something bad happens in the world and the first thing you can think of is to blame minorities? God it must suck going through life like that.

[enjoythemusic]

Wow, no one likes blue ball.

_n2b.jpg

_n2.jpg

[S’portEye]

Originally Posted by scarlet knight

Maleg, I understand your point. But my point is hire the best, but is the best the one with the highest GPA or the most relevant experience? What about nepotism? Do you want a team of ten white men or might better ideas come out of a more diverse team.

If it’s DEI instead of hiring the most qualified, that is wrong. But hiring managers should be wary of the old boy network, pro-white bias, nepotism, etc. I say keep an eye out for people of underrepresented backgrounds. They add dynamism to a monochromatic team. But they must be qualified.

1. Yes, the best is typically the candidate with the best GPA or most experience.
2. Merit based is not nepotism.
3. If it’s 10 white dudes with the best ideas, that’s what I want. Diversity doesn’t matter.
4. DEI and most qualified are mutually exclusive. This is because the most qualified is regardless of race or sex. It’s simple, meritocracy works.

[Boothroyd]

Originally Posted by hdrazor251

The fix is actually very simple.

1. Boot Windows in Safe Mode
2. Navigate to the C:\Windows\System32\drivers\Crownstrike directory
3. Locate and delete the file “C-00000291*.sys”
4. Boot the machine normally

Our biggest complicating factor was BitLocker as Boothroyd (hi Daniel) mentioned earlier. We had to look-up the unique 48 character recovery code for each machine before we could boot in Safe Mode. So it wasn’t as simple as walking the floor and fixing what we saw was broke. We had to get the device name, go back and look-up the code for that device, then back to the device to apply the fix.

As for what broke and what didn’t, it just depended if the device was powered on when the bad one was pushed by CrowdStrike. The bad “C-00000291*.sys” file had a timestamp of 0409 UTC. They later sent a good “C-00000291*.sys” file that had a timestamp of 0527 UTC but if you already had the bad one you were screwed.

Jeff, it's been a while! Go, Pack, Go! Hope you and the family are well!

[Rock]

Originally Posted by S’portEye

1. Yes, the best is typically the candidate with the best GPA or most experience.
2. Merit based is not nepotism.
3. If it’s 10 white dudes with the best ideas, that’s what I want. Diversity doesn’t matter.
4. DEI and most qualified are mutually exclusive. This is because the most qualified is regardless of race or sex. It’s simple, meritocracy works.

It's a bit peripheral to the point of the thread, but having worked for many big organisations and been involved in a lot of 'selections/recruitement', I would have to comment that "Merit-Based" is a whole can of worms on it's own.
In my experience, "merit" was often determined by Nepotism. I saw a lot of candidates with glowing reports from managers who turned out to be complete 'Duds' because they were synchophants who had sucked-up to their managers.
Determining "merit" was the hardest thing about the Selections process. Just sayin'.

[EEpro]

Originally Posted by Rock

It's a bit peripheral to the point of the thread, but having worked for many big organisations and been involved in a lot of 'selections/recruitement', I would have to comment that "Merit-Based" is a whole can of worms on it's own.
In my experience, "merit" was often determined by Nepotism. I saw a lot of candidates with glowing reports from managers who turned out to be complete 'Duds' because they were synchophants who had sucked-up to their managers.
Determining "merit" was the hardest thing about the Selections process. Just sayin'.

It requires technically competent hiring managers which is often not the case. My most recent transfer (not even a new parent company, just division change working in same building) was 3 separate interviews over 2 days with a technical fellow, a principal fellow, and an associate director.

There was no taking anyone's word for anything. It was white board sessions and real time problem solving the issues they are seeing in their product line.

[Maleg]

Originally Posted by EEpro

It requires technically competent hiring managers which is often not the case. My most recent transfer (not even a new parent company, just division change working in same building) was 3 separate interviews over 2 days with a technical fellow, a principal fellow, and an associate director.

There was no taking anyone's word for anything. It was white board sessions and real time problem solving the issues they are seeing in their product line.

Demonstrable ability to perform the work defines merit. I owned an engineering company and your experience is exactly how we hired and promoted staff.

[Brenngun]

Originally Posted by hdrazor251

The fix is actually very simple.

1. Boot Windows in Safe Mode
2. Navigate to the C:\Windows\System32\drivers\Crownstrike directory
3. Locate and delete the file “C-00000291*.sys”
4. Boot the machine normally

Our biggest complicating factor was BitLocker as Boothroyd (hi Daniel) mentioned earlier. We had to look-up the unique 48 character recovery code for each machine before we could boot in Safe Mode. So it wasn’t as simple as walking the floor and fixing what we saw was broke. We had to get the device name, go back and look-up the code for that device, then back to the device to apply the fix.

As for what broke and what didn’t, it just depended if the device was powered on when the bad one was pushed by CrowdStrike. The bad “C-00000291*.sys” file had a timestamp of 0409 UTC. They later sent a good “C-00000291*.sys” file that had a timestamp of 0527 UTC but if you already had the bad one you were screwed.

I got the bad one
Time to find the bitlocker
I hate when this happens

[Krash]

Originally Posted by Rock

It's a bit peripheral to the point of the thread, but having worked for many big organisations and been involved in a lot of 'selections/recruitement', I would have to comment that "Merit-Based" is a whole can of worms on it's own.
In my experience, "merit" was often determined by Nepotism. I saw a lot of candidates with glowing reports from managers who turned out to be complete 'Duds' because they were synchophants who had sucked-up to their managers.
Determining "merit" was the hardest thing about the Selections process. Just sayin'.

I’ve interviewed and hired and hundreds and hundreds of people through the years. I’ve also hired people that interviewed well, but turned out to be duds. Hiring and choosing the right person is never easy.

But the only thing worse than merit based hiring is NOT merit based hiring. You have to cut through the nonsense and hire the person you think is best suited for the position, period.

Plus, if you’re hiring based on other criteria, you’re potentially breaking the law and setting yourself up for major lawsuit. It’s technically illegal to not hire someone because they are a white male. The last decade or so, it’s been tolerated, but there are cracks in that foundation. Things are changing, and changing in a big way.


Sent from my iPhone using Tapatalk

[hambone1983]

Originally Posted by Krash

I have lots of friends in the industry. Some are suggesting that someone at CrowdStrike maliciously dropped this bomb on everyone. But they have to know who implemented this change (at least I would think so). I can’t imagine their processes being so lackadaisical that they have no idea who did it.

Others are suggesting it was indeed a nefarious hack despite what the CEO of CrowdStrike is saying. Some group or organization penetrated CrowdStrike and affected their update. They point to the timing of the event.

Personally, I just think it was incompetence and a failure to properly implement the change they wanted to roll out.

Regardless of what the real root cause is, I’m just not sure how CrowdStrike could ever be trusted again.


Sent from my iPhone using Tapatalk

change management is the bugaboo of the tech field.

99% of all changes are so minor that they don't even have the ability to create an issue if they fail. Also 99% of the time the resolution to failed change is a simple back out.

The amount of time IT management spends completing/reviewing change requests and sitting in change control meetings is a real drag on productive time.

Changes also have to be implemented during off hours, requiring engineers & programmers to work extended hours. No one wants to be the people that get stuck with this duty

You often don't have any vendor support when changes are made, and senior management is never present.

Any kind of substantial change that requires QA requires programmers to write scripts for the QA people (who are generally the least skilled IT people). What they test almost never finds problems.

The only surprising thing about this to me is that it seems like they had no process to restore the old environment when the change failed.

[subtona]

Originally Posted by huncho

do you have crowdstrike on your computer?

Is it part of the 365 platform?

[Easy E]

Originally Posted by subtona

Is it part of the 365 platform?

no, its a third party app/service

[Krash]

Originally Posted by hambone1983

change management is the bugaboo of the tech field.

99% of all changes are so minor that they don't even have the ability to create an issue if they fail. Also 99% of the time the resolution to failed change is a simple back out.

The amount of time IT management spends completing/reviewing change requests and sitting in change control meetings is a real drag on productive time.

Changes also have to be implemented during off hours, requiring engineers & programmers to work extended hours. No one wants to be the people that get stuck with this duty

You often don't have any vendor support when changes are made, and senior management is never present.

Any kind of substantial change that requires QA requires programmers to write scripts for the QA people (who are generally the least skilled IT people). What they test almost never finds problems.

The only surprising thing about this to me is that it seems like they had no process to restore the old environment when the change failed.

When I was still heading a team, we treated every change as if it were a surgical procedure on a human being.

We had our “out patient” procedures and then we had our major, open heart surgery procedures. Typically the difference is between huge multi-million dollar projects vs a simple configuration change that took less than a week to package. Plus, we had everything in between. Sometimes it was our out patient procedures that created more problems than anything. We had small, minor changes that took out entire lines of businesses for a day. You’re really on the hot seat when that happens.

I used to tell everyone, no matter how small the change, infection could always set in and kill us.

For me, the biggest take away is the degree of damage this change did. It took out the entire operating system. That’s f’d up. No software package should have the ability to do that.

Microsoft should be holding their feet to the fire and literally not let them deploy anything until they can prove all their changes are well contained. Not sure how feasible that is, but Microsoft could exert some pressure on them.


Sent from my iPhone using Tapatalk

[EEpro]

So who's buying the stock now?

After seeing how embedded and widespread destructive they can be perhaps they'll pull in billions in NSA contracts.

[hambone1983]

Originally Posted by Krash

When I was still heading a team, we treated every change as if it were a surgical procedure on a human being.

We had our “out patient” procedures and then we had our major, open heart surgery procedures. Typically the difference is between huge multi-million dollar projects vs a simple configuration change that took less than a week to package. Plus, we had everything in between. Sometimes it was our out patient procedures that created more problems than anything. We had small, minor changes that took out entire lines of businesses for a day. You’re really on the hot seat when that happens.

I used to tell everyone, no matter how small the change, infection could always set in and kill us.

For me, the biggest take away is the degree of damage this change did. It took out the entire operating system. That’s f’d up. No software package should have the ability to do that.

Microsoft should be holding their feet to the fire and literally not let them deploy anything until they can prove all their changes are well contained. Not sure how feasible that is, but Microsoft could exert some pressure on them.


Sent from my iPhone using Tapatalk

I've logged some saddle time in that hot seat.

I handled voice so a lot of this is above my pay grade, but my understanding is no change should touch the OS. That joint must be pretty cavalier about root access.

[EEpro]

Originally Posted by Nice marmot

So something bad happens in the world and the first thing you can think of is to blame minorities? God it must suck going through life like that.

Or like an ostrich with your head in the ground.

https://www.dailymail.co.uk/news/art...le-coders.html

[Krash]

Originally Posted by EEpro

Or like an ostrich with your head in the ground.

https://www.dailymail.co.uk/news/art...le-coders.html

They’re getting sued to high heaven. And if they discriminated against people because of race and gender like that article states, then that’s just going to make it a lot worse for them.


Sent from my iPhone using Tapatalk

[EEpro]

Originally Posted by Krash

They’re getting sued to high heaven. And if they discriminated against people because of race and gender like that article states, then that’s just going to make it a lot worse for them.


Sent from my iPhone using Tapatalk

They probably did.

My current company trashed my resume within 24 hrs so I contacted a white male recruiter on LinkedIn, set up two phone calls with him and the engineering manager and got a full price offer after the phone screens, no on site required.

The black female that canned me without a phone call is now working at a college in the diversity department. She got moved on after multiple complaints from managers and recruiters.

It is real. It happens.

[Nice marmot]

Originally Posted by EEpro

Or like an ostrich with your head in the ground.

https://www.dailymail.co.uk/news/art...le-coders.html

My head is not in the ground. It’s pretty easy to spot a bigot.

[Krash]

Originally Posted by EEpro

They probably did.

My current company trashed my resume within 24 hrs so I contacted a white male recruiter on LinkedIn, set up two phone calls with him and the engineering manager and got a full price offer after the phone screens, no on site required.

The black female that canned me without a phone call is now working at a college in the diversity department. She got moved on after multiple complaints from managers and recruiters.

It is real. It happens.

Ironically, what happened to you is discrimination. You have a strong legal case against them.


Sent from my iPhone using Tapatalk

[Maleg]

Originally Posted by Nice marmot

My head is not in the ground. It’s pretty easy to spot a bigot.

You need to squash the insults. Neither productive nor welcome in a civilized discussion

[77T]

A code writer may have fashioned the malformed file. But at two layers up in the org, someone approved its release.

Then, without putting it through a test server, or a development server, released the file to a production server as an auto-update package.

CRWD needs a new CTO, new methods and practices, and shareholders have voted on their performance already.




Sent from my iPhone using Tapatalk Pro