Heathrow Closure

Heathrow had enough power to avoid shutdown, says National Grid | The Independent

National Grid CEO John Pettigrew told the Financial Times: “There was no lack of capacity from the substations. Each substation individually can provide enough power to Heathrow.”

Heathrow’s management acknowledged on Friday that: "the site was served by two other substations", but claimed that, "in order for these to be used, the power supply to all terminals needed to be re-engineered."

A Heathrow spokesperson said: "It would not have been possible for Heathrow to operate uninterrupted. Hundreds of critical systems across the airport were required to be safely powered down and then safely and systematically rebooted."

It sounds like some manual reconfiguration of the in-feeds was required to reroute the supply. I find it hard to believe that automatic transfer switches were not installed for the HV distribution system. Even if manual switching was required, I'm amazed that it would have taken longer that the UPS autonomy time from critical systems.

Lots of questions. Will we ever find out the real causes?

Will we ever find out the real causes?

Well maybe, though there seems to be enough high level interest to trigger an inquiry.

It does seem pointless to have multiple in-feeds at HV if there is no  mechanism for performing the change-over faster than the hold-up time of the emergency power.
As above it sounds like the suitably qualified folk to make the technical decisions and pull the right  levers, perhaps even literally,  were not available - which is remarkable in an organization that size, even if the battle plan for such an event had to involve losing supplies to non-essential areas.

After there are plenty of other places with dual HV feeds and HV rings with zone isolation switching and so on, - consider for example that  most large hospital campuses manage this sort of thing as a matter of routine and even manage to rehearse on a semi-regular basis as well as during planned maintenance. It ought not to be that hard to have someone who understands the systems available at short notice.

Mike.

If it were up to me, I wouldn't even allow microprocessor relays without an electromechanical relay backup for the critical feeders. Putting all your trust in a microprocessor that can be programmed to burn itself up at a moments notice (or predefined future time) seems foolhardy to me - maybe I'm just paranoid.  Then again, we're constantly being told to expect cyber attacks from state actors, so why the disparity?

mapj1 said:

remarkable in an organization that size

That's usually part of the "too big to fail" problem, until the holes in the Swiss Cheese align, not forgetting that those holes can start at the edges of the cheese slices near the top layer.

It's hard to see these hairy big failures and how other systemic aspects will undermine all the planning. It's been seen in many 'big system failures'.

Some oldie but goody reading.

1. Reason, J.: ‘Human Error’ (Cambridge Univ. Pr., 1990

2. Rasmussen, J., Pejtersen, A.M., Goodstein, L.P.: ‘Cognitive systems engineering’ (J. Wiley & Sons, 1994)

True, but that's why we do Failure Modes and Effects Analyses (FMEAs) with the corresponding FMEA proving tests in other industries. No excuses for them not doing this at Heathrow.

True, but that's why we do Failure Modes and Effects Analyses (FMEAs) with the corresponding FMEA proving tests in other industries. No excuses for them not doing this at Heathrow.

The FMEAs are rarely done at the system of systems level (the old Conway's Law).

In this case we appear to have a supra system problem where too many sub-systems (from an airport operators viewpoint) have gone down and that has compromised all their individual backup plans, and the whole thing collapsed ('safety shutdown') from the middle (local operational safety management calling the shots) independently of any thing that some distant HV switching 'maintenance man' could do.

Yes, it looks like this is the case but I don't think that is a justification. My company's FMEA guidance states that "no artificial limit shall be placed on the depth and scope of the analysis". If there is a single point of failure then it's reasonable to find out what will stop working.

The ability (or lack thereof) to switch between healthy utilitiy supplies is a concern but "Failure of utility power" seems like it should be pretty high on any business continuity risk assessment to me. 

I still feel like there must be more to this than poor planning.

Agree - and I suspect that much like the reports into the Lancaster flood power cuts, or the Hornsea incident the report will once again make embarrassing reading in terms of being entirely visible with hindsight, and possibly even with foresight..

In this case once in terms of the how the transformer was able to fail quite so spectacularly, and if inspections etc might have seen it coming, and secondly, given that only  one of the three HV supplies failed, in terms of why the airport  failed to rapidly engage the redundant supplies it had after all been designed with, to best ability.

Mike

There's always a point of failure :Big-Smiley: [Hollywood movies various]

It's the errors of omission not commission (never saw it coming) that make for the big disasters.

What we haven't seen (yet) is the 'shutdown the airport' cascade of guidance and how that got overloaded. We've had major delays at Heathrow before, but electricity is becoming an essential to life aspect.