Yup, this is a ‘create howls of deriving laughter’ on Microsoft, but not in the way you would expect it. So, this all started a few hours ago when I saw an unknown party called ARN give us ‘Microsoft blames Aussie data centre outage on staff strength, failed automation’ (at https://www.arnnet.com.au/article/708608/microsoft-blames-aussie-data-centre-outage-staff-strength-failed-automation/) where we see “Microsoft has blamed staff strength and failed automation for a data centre outage in Australia that took place on August 30, disabling users from accessing Azure, Microsoft 365, and Power Platform services for over 24 hours.” And my (first) thought was ‘Is Microsoft really THAT stupid?’ You see, to see that thought you need to be aware of a few small issues. The first is “Microsoft confirmed Monday that it’s eliminating additional jobs, a week after the start of its 2024 fiscal year. The cuts are in addition to the downsizing announced in January that resulted in 10,000 layoffs. The software maker also disclosed a small number of cuts this time last year.” With the additional “US tech giant Microsoft has axed more Australian jobs after the company made major staffing cuts across the globe earlier in the year. About 50 Australian employees are believed to have lost their jobs this month, Nine newspaper the Australian Financial Review reports.” Now, job losses happen everywhere at this time and we get it. There are all kinds of issues and Microsoft is one of many shedding jobs. But to see ‘Microsoft has blamed staff strength’ after they shed 10,000 plus jobs is just the joke of the century. I get it, one job is not another job, but when you have shortages in a place that is riddled with ageism and wannabe hires (dynamic young people) whilst your operational settings are below par just doesn’t work for me. I see the same fake jobs from providers like Hays and they will not respond and often ignore you. That is the party to be for players like Microsoft and they now claim that there is no coverage does not hold any water with me. So when ARN gives us ““Due to the size of the data centre campus, the staffing of the team at night was insufficient to restart the chillers in a timely manner. We have temporarily increased the team size from three to seven, until the underlying issues are better understood and appropriate mitigations can be put in place,” Microsoft wrote as part of the report.” I wonder if their cost cutting stages are merely a joke and what company would have trust in such a system when “Azure, Microsoft 365, and Power Platform services” were down or unreachable for over 24 hours. That point is clear, is it not?
Consider the simple math. How much traffic and how many companies rely on that data centre? How come that there are only 3 people at night? So consider “Microsoft said that the cooling units could have been restarted manually, which was not possible due to the unavailability of enough personnel at the data centre” with the added “the staffing of the team at night was insufficient to restart the chillers in a timely manner” so do you think they royally screwed that part up? And in that setting how many data centres (all over the world) are understaffed? When the coolers cannot be manually started in these places, how much revenue will Microsoft miss out on, because these affected firms might optionally have a case to sue Microsoft for damages. No matter how that report phrases it, the lack of data centre labour (especially after they sacked well over 10,000 people) will not be met with a friendly judge and for Microsoft there is an additional danger. When third parties like Evroc start getting business from companies that once held Microsoft high in its banner, the walk-out might become a lot more severe and that could spell more bad news for Azure (something Amazon AWS will love) and there is a decent chance that some will optionally switch to Google or IBM. All losses for Microsoft who thought that keeping 3 people at night in a data centre was enough, all whilst THEY THEMSELVES give us “the cooling units could have been restarted manually, which was not possible due to the unavailability of enough personnel at the data centre” and that is the stage all those using a Microsoft data centre face? It is my personal opinion that someone bungled the minimum staff at a data centre during the night and even as winter is now coming to the northern hemisphere. The southern hemisphere is going into summer. So what about the Data centres in Riyadh and the UAE? In Riyadh it is around 45 degrees Celsius and in Dubai it is only 3 degrees cooler. So what happens when they need a manual restart of the cooling units? All simple questions and we could say that Microsoft has that covered, but it seems that according to ARN they do not. A simple operational question: ‘What is the minimum required staff coverage at night in a worst case scenario?’ As far as I can tell (trusting the ARN article) they were not ready and the fact that they upped it by over 100% shows that Microsoft was simply clueless on this issue. Feel free to disagree and I expect you want to talk to the corporations that lot Office and Azure for over 24 hours, but I reckon that we will not get access to those names, and that is fair enough. But do the companies who had to go through this feel the same way? I doubt it.
Enjoy the warm Tuesday coming to you.
Lawrence van Rijn - Law Lord to be 