Crowdstrike Nukes Windows Computers Worldwide

[nullplan]

Unfortunately, without that facility security software wouldn’t be able to do its job.

So it's impossible to create a somewhat standardized interface to load virus definitions without directly accessing the file system and loading files directly into kernel space? I find that hard to believe.

You seem to be under the impression that there are no security problems with Linux.

I am fully aware that all software sucks. It is just that some software sucks less, Linux is not perfect, and indeed the state keeps getting worse, because a lot of software is written without any thought for security. There is this D-Bus monstrosity, where absolutely nothing is documented. And all sorts of data is dumped into it, with no regard for privacy. There's this systemd thing that keeps pushing its tendrils into everything, and indeed that was the crux of the xz thing I mentioned earlier.

But with Linux at least it is possible to find out about these things, and to change them. With Windows, you have no choice but to accept the state of affairs you find. You have no choice but to accept the statements from Microsoft, that Windows is insecure and must be secured with "security" products, and that there is nothing else that can be done to help. And that there is no keylogger in Windows that is phoning home, honest!

You seem to be laboring under the misconception that I see the world in black and white. I don't. I try to look at things evidence based and rational. And from a rational point of view, using closed-source software, trusting only in the words of a company that is motivated by greed and greed alone, is insane.

[iansjack]

Great rant.

But it doesn’t alter the fact that CrowdStrike were responsible for this debacle. You think it’s Microsoft’s fault; no doubt others will blame Intel for not making their processors proof against this sort of problem.

Anyway, we’re not going to agree. I think that there’s a place for diverse operating systems, which is why I use Mac OS, Linux, and Windows every day. Each has their place.

Whatever, I still cannot agree with your initial comment that this is in any way amusing.

[chase]

I don't think this is Microsoft's fault but I hope they make improvements to limit the damage a 3rd party can cause. For example automatically starting some form of startup repair that is able to query MS for needed repair procedures. With a confirmation prompt of course.

[iansjack]

I don't know how CrowdStrike works, but when I managed a network of several hundred Windows machines all security updates - from Microsoft and third parties - were tested on an isolated mini network first. If there were no problems the updates would then be served out from an internal server. No machines received updates directly from an outside source.

This isn't a foolproof process. It's not really feasible to test every software combination on every individual machine but, if CrowdStrike updates can be rolled out in this way, it would have prevented this problem.

To my mind the blame for this sort of outage rests, in order, with 1. the network administrators (and their bosses who had the purse strings), 2. the 3rd party software manufacturer, 3. the operating system vendor, 4. the hardware vendor. (But, having been in that position, my sympathies are totally with the sysadmins. Very few companies are as accommodating and generous as mine was when it comes to purchasing apparently redundant equipment.)

It's a debatable matter as to whether open-source makes such problems more or less possible. People should be able to find solutions more easily (although I would have thought the software writers were in the best position to correct errors); on the other hand, bad actors have much more information about how to circumvent security barriers and much more opportunity to do so. But then, I work from the point of view that companies like Apple and Microsoft are trying their best to produce good software that will be popular and enhance their bottom line. I don't see them as semi-criminals ripping off the public at every opportunity.

The most secure systems that I have ever worked with were System/38, AS/400, iSeries from IBM. Totally propriety from hardware to software, totally closed software and hardware, solid as a rock (and boring as a rock).

[eekee]

I don't know how CrowdStrike works, but when I managed a network of several hundred Windows machines all security updates - from Microsoft and third parties - were tested on an isolated mini network first. If there were no problems the updates would then be served out from an internal server. No machines received updates directly from an outside source.

Apparently, all professional network admins do this. Crowdstrike pulled an end run around it. Here's a comment on a Dave's Garage video on Youtube:

While this is technically what crashed machines it isn't the worst part.

CS Falcon has a way to control the staging of updates across your environment. businesses who don't want to go out of business have a N-1 or greater staging policy and only test systems get the latest updates immediately. My work for example has a test group at N staging, a small group of noncritical systems at N-1, and the rest of our computers at N-2.

This broken update IGNORED our staging policies and went to ALL machine at the same time. CS informed us after our business was brought down that this is by design and some updates bypass policies.

So in the end, CS caused untold millions of dollars in damages not just because they pushed a bad update, but because they pushed an update that ignored their customers' staging policies which would have prevented this type of widespread damage. Unbelievable.

The comment was left on CrowdStrike IT Outage Explained by a Windows Developer. Since Dave is a former Microsoft dev, here's his Lessons Learned too. I haven't watched either because I'm too unwell to deal with the subject, but I thought this particular detail ought to be known.

Ditto for the subject of Windows secure boot compromised. I have a lot of respect for Dave Plumber, he clearly knows his stuff, and would watch all his Windows videos if I were fitter.

Incidentally, I'm very sorry for what I've said in the past to promote the idea that Microsoft are criminally irresponsible. It was before I was active on here, and I guess there are others with similar mental flaws who may have reached the same conclusion independently, but I was rather vocal and ideas do spread.

And I can believe a perfectly solid system is a boring system. What interests me about 8-bit computers are the very things which make them weird and troublesome; the clever hacks for making best use of limited resources.

[nullplan]

Crowdstrike has now published a post mortem on the incident. If you subscribe to nihilistic humor, this one is a great read. One of the things they found was that it would be a good idea to check that the number of arguments is correct. Yeah... you know, the kind of thing that compilers were invented to do? More than half a century ago?

Another finding is that runtime bounds checking is good. Which would be novel, were it not for the preceding 30 years since "smashing the stack for fun and profit" came out.

[eekee]

Another finding is that runtime bounds checking is good. Which would be novel, were it not for the preceding 30 years since "smashing the stack for fun and profit" came out.

Bounds checking was a big deal long before 1994. It was a hot topic in the 80s, and that was in the microcomputer market. I'm sure it goes back to the 60s at least. But with a company name like Crowdstrike, what else could they be than kids who think they know everything when they've hardly taken the time to learn anything?