World wide IT Outage

Can any IT-savvy explain what is going on and how this happened? Surely such an event could have been predicted and there was a plan in place ....
I have a friend flying over from Germany for a much looked-forward-to event, so I'm worried he ain't gonna make it and Mrs RR can't book a much-needed GP appointment. These are trivial concerns of course compared to what's happening world-wide, but still....
Any sensible info much appreciated.
I can't help thinking about the story of the Tower of Babel. It beggars belief we put all our ergs in one gasket

Comments

  • The Guardian (UK):

    https://www.theguardian.com/australia-news/article/2024/jul/19/microsoft-windows-pcs-outage-blue-screen-of-death

    It seems to have been caused by a software failure, and not (presumably) by Malefactors seeking to bring the world to a *grinding halt*...
  • HugalHugal Shipmate
    So a cybersecurity company sent an update that caused the shut down. Ironic much?
  • Hugal wrote: »
    So a cybersecurity company sent an update that caused the shut down. Ironic much?

    Well, quite. The question perhaps arises as to how much of an accident this was...

    IOW, could it have somehow been engineered by Malefactors? I'm old enough to remember as far back as circa 1990, when my employers first put files etc. on computers, on which occasion I was laughed out of court for suggesting that pencils and index cards might still be needed as backup...

    (I checked my online bank a/c earlier today, and all seems well...).
  • Jane RJane R Shipmate
    edited July 2024
    This is the kind of thing that happens when you bolt a piece of software onto an existing system without testing it thoroughly beforehand. The only unusual thing is that it's highlighted just how dependent we are nowadays on a handful of computer programs working correctly.

    They're saying at the moment that it wasn't a cyberattack, so just imagine the chaos if someone tried to do it deliberately...
  • RockyRogerRockyRoger Shipmate
    edited July 2024
    Many thanks for the link, which I read with much interest, not to say incredulity. Was the faulty software update not fully beta-tested in an isolated system before release?

    I'd like to add, 'heads will roll', but according to the article, they probably won't.

    But surely something needs to change? If it happened once it can happen again ...

    One nice comment on the BBC website wondered if it just happened to be Liz Truss's first day at the IT company .....
  • Unfortunately there are quite a few major vulnerabilities in the structures that support our world. The one that keeps haunting me is what would happen if we had another Carrington Event (which will happen sooner or later). But when I mentioned this to our IT people, they looked at me as if I'd lost my mind. They were in a rush to transition all our stuff to servers maintained by Somebody Else and available to us henceforth only via the internet...
  • Jane R wrote: »
    This is the kind of thing that happens when you bolt a piece of software onto an existing system without testing it thoroughly beforehand.

    Everyone has a test system, some people also have a separate production system.
  • The estimable LC wrote: 'The one that keeps haunting me is what would happen if we had another Carrington Event'. A most timely comment, this worries me too. The 'Carrington event' was a humungous solar storm in 1859 that knocked out the then-new telegraph system.
    And the chances of another equally powerful solar storm hitting the Earth? It's not 'if' ,but when. But the guys just don't want to know ....

    As I said, The story of the Tower of Babel springs to mind.

    We're doomed!

    Meanwhile, prayers for all those poor folk stranded at airports and those whose operations have been postponed.
  • Gramps49Gramps49 Shipmate
    Ancestry.com keeps crashing today. Can't blow an hour that way.
  • FirenzeFirenze Shipmate, Host Emeritus
    I am old enough to remember the world Before (internet/TV/smartphones/apps for everything) and I do wonder about those who take such things for granted.


  • KarlLBKarlLB Shipmate
    edited July 2024
    Hypothesis - the update was tested and found to have a bug. The bug was patched and then a SNAFU in the versioning system caused the buggy version to be released instead of the patched one.

    Given the short release cycle of security product updates, this is far from impossible.
  • KarlLB wrote: »
    Hypothesis - the update was tested and found to have a bug. The bug was patched and then a SNAFU in the versioning system caused the buggy version to be released instead of the patched one.

    Given the short release cycle of security product updates, this is far from impossible.

    This sounds all too plausable ....

    So this is the way the world ends!
  • ...not with a Bang, but with a Blue Screen?
  • DoublethinkDoublethink Admin, 8th Day Host
    *Hat-tip to @pease as the ship sails on where others founder*
  • NicoleMRNicoleMR Shipmate
    Starbucks online ordering is down from this. Very minor of corse but it goes to show how widespread something like this is.
  • eBay is OK, which is just as well, given the extensive use I make of that emporium!

    My sister is due to fly back to France over the w/e (not sure which day), so I hope she's not affected. They say, however, that it might be some time before the fault is fixed.
  • HuiaHuia Shipmate
    Thanks for the link @Ruth.

    I was busy swearing at my needlework, so only found out what was happening when a friend texted me. My first thought was that it was some kind of cyber attack, the news that it was due to a glitch was both reassuring and worrying.

    It did make me realise I should have more than $20 cash on hand though.
  • MoyessaMoyessa Shipmate
    My grand-daughter is in the air on her way to China (left from California several hours ago).
    Not going to think about it too much.
  • No. Though if she's already in the air, that's the greatest part of the trouble averted.
  • MoyessaMoyessa Shipmate
    Thanks
  • ...not with a Bang, but with a Blue Screen?

    Actual laugh our loud, thank you!
  • Jane R wrote: »
    This is the kind of thing that happens when you bolt a piece of software onto an existing system without testing it thoroughly beforehand.

    Everyone has a test system, some people also have a separate production system.

    Everyone in IT knows, or should know, that you NEVER roll anything out without thoroughly testing it in a test environment, and then having a rollback plan in place just in case something does go wrong. I am very, very surprised at Crowdstrike.

    As for me -- my weekend away was ruined. Got as far as the airport and that's about it. Pandemonium. And I made the mistake of booking my flight and hotel through Expedia, and not taking the cancellation insurance. And Expedia's customer support is a joke, if indeed it does exist (which I doubt).

    If Expedia were run by gentlemen they would refund everyone's canceled flights and canceled hotel bookings regardless of whether or not insurance was in place.

    But I've learned my lesson. Never again will I book through Expedia.
  • HugalHugal Shipmate
    Here Expedia are part of groups like ABTA that guarantee certain things. Is there anything like that over there?
  • HugalHugal Shipmate
    Sorry to double post here is a link to ABTA for those who don’t know.
    https://www.abta.com/
  • I don't think we have anything like that.

    I will give Hertz credit, though. I reserved a car through Hertz and had to pay a deposit up front. Their policy was that the deposit was refundable only if the reservation was canceled a certain number of days in advance. If the reservation was canceled the day of pick-up, or if pick-up was a no-show, the deposit would not be refunded.

    Well, Hertz did refund my deposit in full. Gentlemen all the way.

    Meanwhile, I see that Delta Airlines is still struggling to recover, and is still canceling flights.

    It used to be -- I don't know if it still is, as I'm probably the only person in the world who is still on Windows 7 -- that you could create "restore points" in Windows, so that Windows could be made to load to the "last known good restore point" if something went wrong. Best practice dictated that you **always** created a restore point before receiving a software upgrade of any kind. Clearly the folks at Delta did not.
  • la vie en rougela vie en rouge Purgatory Host, Circus Host
    I suspect I have been Crowdstruck. I went to withdraw some cash and typed in that I wanted €60. The following screen informed me that I had requested €270 and gave me no option to cancel or go back and I ended up taking out this whopping amount of money.

    I am waiting to check my account to see how much has been debited.
  • HugalHugal Shipmate
    I suspect I have been Crowdstruck. I went to withdraw some cash and typed in that I wanted €60. The following screen informed me that I had requested €270 and gave me no option to cancel or go back and I ended up taking out this whopping amount of money.

    I am waiting to check my account to see how much has been debited.

    Ouch
  • I suspect I have been Crowdstruck. I went to withdraw some cash and typed in that I wanted €60. The following screen informed me that I had requested €270 and gave me no option to cancel or go back and I ended up taking out this whopping amount of money.

    Spend it wisely. :wink:
  • la vie en rougela vie en rouge Purgatory Host, Circus Host
    Well it's my money at the end of the day, except it's now sitting in my purse instead of in my account. I do wonder what would have happened to someone who didn't have the funds in their account to cover it.
  • Free gift . . . ? :naughty:
  • KarlLBKarlLB Shipmate
    I don't think we have anything like that.

    I will give Hertz credit, though. I reserved a car through Hertz and had to pay a deposit up front. Their policy was that the deposit was refundable only if the reservation was canceled a certain number of days in advance. If the reservation was canceled the day of pick-up, or if pick-up was a no-show, the deposit would not be refunded.

    Well, Hertz did refund my deposit in full. Gentlemen all the way.

    Meanwhile, I see that Delta Airlines is still struggling to recover, and is still canceling flights.

    It used to be -- I don't know if it still is, as I'm probably the only person in the world who is still on Windows 7 -- that you could create "restore points" in Windows, so that Windows could be made to load to the "last known good restore point" if something went wrong. Best practice dictated that you **always** created a restore point before receiving a software upgrade of any kind. Clearly the folks at Delta did not.

    Restore points are no help if you can't actually boot the OS at all, which is the problem here.
  • chrisstileschrisstiles Hell Host
    edited July 2024
    KarlLB wrote: »
    I don't think we have anything like that.

    I will give Hertz credit, though. I reserved a car through Hertz and had to pay a deposit up front. Their policy was that the deposit was refundable only if the reservation was canceled a certain number of days in advance. If the reservation was canceled the day of pick-up, or if pick-up was a no-show, the deposit would not be refunded.

    Well, Hertz did refund my deposit in full. Gentlemen all the way.

    Meanwhile, I see that Delta Airlines is still struggling to recover, and is still canceling flights.

    It used to be -- I don't know if it still is, as I'm probably the only person in the world who is still on Windows 7 -- that you could create "restore points" in Windows, so that Windows could be made to load to the "last known good restore point" if something went wrong. Best practice dictated that you **always** created a restore point before receiving a software upgrade of any kind. Clearly the folks at Delta did not.

    Restore points are no help if you can't actually boot the OS at all, which is the problem here.

    In the case of Windows you can then recover using WinPE or via safe boot and safe mode.
  • la vie en rougela vie en rouge Purgatory Host, Circus Host
    Free gift . . . ? :naughty:

    Nope. My account was debited for the full amount. :grimace:
  • KarlLBKarlLB Shipmate
    KarlLB wrote: »
    I don't think we have anything like that.

    I will give Hertz credit, though. I reserved a car through Hertz and had to pay a deposit up front. Their policy was that the deposit was refundable only if the reservation was canceled a certain number of days in advance. If the reservation was canceled the day of pick-up, or if pick-up was a no-show, the deposit would not be refunded.

    Well, Hertz did refund my deposit in full. Gentlemen all the way.

    Meanwhile, I see that Delta Airlines is still struggling to recover, and is still canceling flights.

    It used to be -- I don't know if it still is, as I'm probably the only person in the world who is still on Windows 7 -- that you could create "restore points" in Windows, so that Windows could be made to load to the "last known good restore point" if something went wrong. Best practice dictated that you **always** created a restore point before receiving a software upgrade of any kind. Clearly the folks at Delta did not.

    Restore points are no help if you can't actually boot the OS at all, which is the problem here.

    In the case of Windows you can then recover using WinPE or via safe boot and safe mode.

    That's what the official fix is. But it requires the local admin password, and depending on your config, the BitLocker key.
  • Interestingly, some of the computers that are affected by this patch are remote, without monitors, keyboards, mice or any other devices where an IT professional has to drive out with all the gear to fix that computer. Think small computer at the top of a telephone pole in the middle of the Utah desert! Ugh.
  • Interestingly, some of the computers that are affected by this patch are remote, without monitors, keyboards, mice or any other devices where an IT professional has to drive out with all the gear to fix that computer. Think small computer at the top of a telephone pole in the middle of the Utah desert! Ugh.

    Yeah, and it underlines the usefulness of out of band access via the network to their console.
  • I am happy to report that Delta has refunded my canceled flight in full. My faith in them has been restored.
  • Good to hear Miss Amanda.
  • KarlLBKarlLB Shipmate
    Interestingly, some of the computers that are affected by this patch are remote, without monitors, keyboards, mice or any other devices where an IT professional has to drive out with all the gear to fix that computer. Think small computer at the top of a telephone pole in the middle of the Utah desert! Ugh.

    Yeah, and it underlines the usefulness of out of band access via the network to their console.

    Quite. It's not like iLO and iDRAC are new technologies.
Sign In or Register to comment.