Just watching the proverbial bodies stack up
I hate you. Today I discovered a server that appeared to be in working order was actually just hanging on by a thread. The crash when the crowdstrike driver caused the blue screen of death must have corrupted some of the network configuration in the registry. When I couldnt make the failover cluster live migrate to or from that server I ejected it from the cluster and rebooted it, only it never came back up 😥
Now all my servers are suspect until I can verify that they are healthy.
Ah yes, the realisation that everything dependable and solid in the world is just a monument to hubris, built on foundations of sand
“registry” shudders
In general, I don’t think the concept of a configuration database is bad - at least not inherently moreso than every application putting its own configuration file(s) in whichever format it wants somewhere on the filesystem.
Whether the window registry is a good implementation of this concept is surely debatable.
This latest debacle is making my department move from windows to Linux. We were already planning it very slowly but then everything crashed at the same time…and all our other services worked except the ones on windows boxes. We can’t afford downtime so it was decided.
It’s not a good reason but I’ll take it haha.
I don’t know how CrowdStrike works on Linux, but it’s worth remembering that if it’s a kernel level driver like it is on windows, and they release a driver that crashes the Linux kernel, there’s a chance for the same thing to happen.
CrowdStrike absolutely fucked Linux in the past.
Thank you for mentioning that. I really hate how people on here think Linux is some panacea that will magically solve everything. It too it just another tool that depends on how it’s used. CrowdStrike exists for Linux, and it was crashing systems a few months ago.
The bigger issue is most people who use Linux know what they’re doing. There are a lot of competent Windows Administrators too who didn’t have issues or were able to recover them in a timely manner. What happens though is you have a very large set of people who just need a computer, need is secured and don’t know how to administer or manage it. Doesn’t matter if they’re running Linux, or Windows, they’ll always have the greatest problems. They just happen to use Windows because it offers better Enterprise support options and usability. One day, Linux may be that, but I guarantee it won’t fix all of those pebkac issues.
I mostly agree. As someone that’s worked with both Windows and Linux for over 15 years, I think we need to ask the question of “why do we see so many incompetent admins?”
If you aren’t paying people enough to give a shit about what they are doing, they won’t.
The answer is that companies are unwilling to allocate sufficient budget to infrastructure. So anyone competent leaves either because either there is better pay elsewhere, or they don’t want to be held responsible for the shoestring shitshow that companies are willing to pay for.
Which is sort of the reason crowdstrike is so popular in the first place. Technically inept leaders want to check a “secure” box in their infrastructure presentation to the board, and certainly don’t want to hire an actual cybersecurity team alongside what they already consider to be an expensive IT team. (Granted they can’t do the mental work of realizing that basically every one of their employee uses a computer every day for hours at a time, and connects to vast networks of computers sitting in datacenters). So to save money, and seeing the legally binding contract, they use crowdstrike.
It did happen already https://www.theregister.com/2024/07/21/crowdstrike_linux_crashes_restoration_tools/
I personally get nervous when any software wants to mess with drivers unless it’s graphic drivers.
For work we don’t plan on using cloud strike. We needed to get everything up asap and the os allowed us to do so quickly. Seemingly unrelated systems and Azure was all down for quite some time.
I was on vacation. They laid off my boss 2 weeks ago. They laid off my bosses boss 3 weeks ago. Guess who no one could contact?
I found out about all of this 2 days after the fact. Best vacation ever.
Wtf did they honestly expect from you in that situation anyway? Do they really think you would come back from vacation to deal with that? From what I can tell, fixing it is barely beyond a T1 hekpdesk amount of knowledge, just an annoying process. Unless there’s more to it, I haven’t looked heard at the details but that’s the gist I’m hearing.
But unless they’re going to triple my vacation length (in terms of paid days off in other words) and maxing out all my upgrade options for the next one I’ma have to say they can get bent.
Basically they gotta pay for the best version possible of that trip (assuming there was a trip) and give me enough time to take 2 more.
I ain’t taking a work call on vacation.
Funny story. My boss vetoed work phones as a cost savings and at first there was a squawking fest but honestly he handled all communication to us and protected us from so much bullshit. Now he’s gone and no one could figure out how to get in contact with us aside from email/teams.
The Feels Good guy has a stick figure hand sticking up out of his arm. I cannot unsee it, and I suspect now neither can you.
Why would you burden me with this?
Police! This poster right here!
Ah man, why you do this to me!! Lol
Man, I quit my IT job 2 years ago and I haven’t looked back. So glad I got out when I did.
I love working IT always been good to me…
What field did you move into?
I was laid off several several months ago, am kinda glad now
