Heathrow Closure

There's plenty of hindsight bias to go around;-)

At least one engineer knew - the one who assessed the fire risk, identifying that with the lack of physical separation, a fire on one transformer was at risk of causing damage or the fire to spread to the adjacent one, taking them both out of service, and so went to the efforts of getting a deluge fire protection system added to protect them both.

There are two cables from the North Hyde substation to the airport - the incident report shows that Heathrow Substation A was supplied by two separate circuits, fed separately at 66kV from the North Hyde substation with each circuit having its own 66kV/33kV step-down transformer, providing two separate 33kV supplies into Substation A - see Single Line Diagram on page 13, Figure 3.

Therefore a single cable fault, or other fault, on one circuit would not be expected to interrupt all supplies to substation A.

Further, the North Hyde 66kV substation had 5 separate 66kV incoming supplies - 3 x 275kV/66kV Supergrid transformers and 2 x 66kV interconnectors to/from Iver.

Given the 5 separate supplies into North Hyde, on the face of it, it doesn't seem unreasonable to assume at least one of the circuits supplying Substation A would remain energised, or if it did fail would be restored within a short time frame. As it was, the report timeline shows it took ~10 hours to re-energise the 66kV substation.

Heathrow may have assumed that two separate supplies to substation A was sufficient redundancy, that those two incoming supplies should have sufficient independence such that they should not both fail at once.

I'm not sure I could be convinced of that logic - even if the supply to the substation was multiply redundant, the cable from the substation to the airport would still provide a single point of failure. I can't see how anyone can reasonably treat any single grid supply as 100% reliable - if it hadn't happened for this reason, it could well have happened for a dozen other perfectly justifiable reasons. They had multiple supplies, but didn't seem capable of utilizing them within a reasonable timescale (i.e. co-ordinated with UPS run times for critical equipment).

   -  Andy. 

Heathrow Airport's arguement here would be that the 'black swan' event was not actually a black swan event and was readily known about by National Grid. 

I'm not sure anyone in a room full of engineers would have predcted one of the transformer would fail in such spectacular fashion and take out two other supplies at the same time.

"Of course, it's quite possible that they did understand the risks entirely, but decided that it wasn't worth the extra cost of implementing any mitigation as they calculated that they wouldn't bear the brunt of the costs of any outage."

This was my thoughts - Further up the chain i had refernced that an article suggested the outage cost Heathrow Airport £20m.  The cost for the private network reconfiguration, additional generators, UPSs etc could easiliy run to multiples of this given the airport virtually never shuts.  I could see that someone may have thought why bother with all the disruption and compensation to airlines/retailers etc when if the 'black swan' event happened they could opportunistically and necessarily reconfigure the network and it would all be someone else's fault.

TurboGen said:

Heathrow may have assumed that two separate supplies to substation A was sufficient redundancy, that those two incoming supplies should have sufficient independence such that they should not both fail at once.

Maybe, but it's simply unconscionable that Heathrow Airport Holdings Ltd hadn't carried out and acted upon a basic Failure Modes and Effect Analysis for such a critical piece of national infrastructure.  Of course, it's quite possible that they did understand the risks entirely, but decided that it wasn't worth the extra cost of implementing any mitigation as they calculated that they wouldn't bear the brunt of the costs of any outage.

Simon Barker said:

This is what happens when major national infrastructure is run by for-profit companies.  Shareholder dividends come before maintenance.

But arguably, if the government and its regulator can't apply even a basic incentive structure (carrot and stick) to improve the security of critical infrastructure then what hope is there that they can run the entire operation properly?

Personally I could live with either fully private or fully nationalised infrastructure, and anything in-between, because the problems this country faces far transcend any public vs private ownership models.

Peter Brooks said:

There has to be an independent auditing organization that has the power (without being interfered with by the government) that has the power  to force the necessary controlling organization to do the required maintenance work.

Agreed

You can see if from both sides.

The network companies knowing the intricacies of their network design, point to Heathrow having supplies from several different substations, placing the responsibility on Heathrow as a very large and well resourced customer to be able to operate from any of the incoming supplies.

Heathrow's affected substation A had two separate incoming supplies from the DNO and it is the end customer, it cannot be assumed to know the intricacies of the network company's network configuration which are beyond its control. Heathrow may have assumed that two separate supplies to substation A was sufficient redundancy, that those two incoming supplies should have sufficient independence such that they should not both fail at once. Further, the restrictions on parallel interconnection between different substations imposed by the network companies meant the use of any within-site interconnectors would be highly restricted anyway.

The topic of independence is key to the whole incident, people focus on the bushing failure which was the trigger, but it was the lack of a functioning fire deluge system (due to the failure to repair the long standing fault on the fire pumps) which allowed the incident to escalate. The deluge system was there to prevent a fire on one transformer (SGT3), compromising the independence of the adjacent transformer (SGT1) and even after the initial failure of the bushing on SGT3, SGT1 continued to function for 20 minutes as the fire raged on SGT3. SGT1 finally tripping off as the fire incinerated its marshalling kiosk. At which point, SGT2 which was physically independent of SGT1 and SGT3, at the other side of the sub, but electrically dependent on SGT1 sharing the same feed also tripped off which resulted in the loss of supply.

Looking at the incident as an outsider, clearly the bushing failure was the cause of the SGT3 failure, but it was the lack of the functioning fire deluge system combined with the lack of independent 275kV supplies to SGT2 which escalated the incident to cause the loss of supply.