BA Fiasco

[Suff]

Still not passing the sniff test....

In every UPS installation I have ever seen there are two power feeds. One comes from the mains and one comes from the UPS with generator backup. There is a failover switch (or more than one in a parallel system). The power from each source comes into the switch and feeds direct to the power bus. It is a two way switch, either the power is on mains or it is on UPS/Generator.

What it says in this article, in a correctly designed data centre redundant power design, is simply not possible. You can't flip one switch and have the whole power source bomb out. In fact, in most sensitive installations, even the UPS is separated from the generators and is local to the servers. If you pull both the mains and the generator power the UPS takes up the slack and there is no way of switching it off bar pulling loads of cables out of loads of machines.

To say that someone flipped one switch and the whole data centre went dark, then it all was blasted with a surge when it came back on, simply doesn't fit what I know of data centre power design.

Never mind the question about whether the other data centre, 1km away, was able to function or not...

It sounds to me like a fabrication on top of a fabrication. The thing is some grunt at the bottom of the chain knows what happened and we'll hear eventually.

[cromwell]

If the design has been done right Suff, yes. If it has been tested, yes. When was it last tested? Etc.

We did a controlled black start to one of our buildings once. Mains power off, onto the generator - bang. Cloud of black smoke and that was that.

The generator hadn't been run since Adam was a lad, the building was using far more power due to the creation of a data input bureau on the second floor etc etc.

Not saying it was the cause here though, especially since BA keep changing their story every couple of days.

Whatever it was must be pretty embarrassing though.

[Suff]

I was in the Edinburgh RBS data centre on a Saturday in 1995 when they started a UPS/Generator test. As they switched the power switch from mains to backup, the switch jammed. As they had 5 hours of battery backed power in the computer rooms, to keep the two mainframes and sundry other servers alive, it was not seen as an issue. So they dismantled the switch, stopping any chance of switching back to mains power, to fix the issue.

About 5 minutes into the fix of the switch, the mainframe tried to shut down. It was overridden. The operators went to talk to the guys doing the switch and told them to get a move on as they were going to need some power, either generators or mains, back up shortly.

It went right down to the last 20 seconds before they got the switch reassembled and the generators on. The A machine which ran all the cashlines in the country was about to go down hard. Requiring a 2 hour startup from punched cards.

Of course there would have been no issues as the Z Machine in London would just have taken over operations, but still, not where you want to be.

In the investigation, afterwards, it was found that whoever wired up the UPS had missed one of the water pumps that cooled the mainframe CPU's and that one of the CPU's was overheating.

Even with the testing, it turned out they were not doing the testing properly. They always just flipped the system to generator and ran them for 4-5 hours. The battery UPS was never needed for more than a few milliseconds and never fully tested.

Even then, in a properly designed and tested system, even the failure of a complete data centre should leave you operating.

So we have to ask, what the hell were BA doing or what was their service provider doing???