Sell your telecoms shares

If you had were building a telephone exchange, particularly one of the biggest telephone exchanges in the UK, one that contained literally acres and acres of electronic and electrical equipment, would you put a lot of this gear below ground level?


Particularly, would you put the hundreds of pieces of electrical gear - you know, transformers and the like - that powered the exchange in a basement 4 stories below street level? Perhaps not.

But if you did, would you be astonished when this happened?



Yup. That's what caused an abso-fucking-lutely huge telephone and network outage yesterday, which lasted right up until this afternoon. There was over 18 inches of water where there shouldn't be any.

Many businesses, including the one for which CF is currently working, have been without 'phone lines for 24 hours or more.

In a conference call to one beleaguered Telecoms company earlier today, one angry customer raised the subject of compensation.

Rather than being told to sod off, act of god, nothing can be done, he was told that all customers will be addressed individually on the subject.

Ouch.

.

8 comments:

Catosays said...

Other considerations aside, it strikes me that BT should be suing the company which was supposed to have made it waterproof.

JohnRS said...

Ok, if you're smart you dont build exchanges, especially important ones, below the water table, low tide mark or on a flood plain.

But having acted like prats in siting the kit where you did, I always understood that the BT network backbone was multipley redundant, no single point of failure, fully meshed, resilient, self healing, backed up etc etc.

So what happened guys?
Where's the failover?
Why were 400+ local exchanges affected?
Why isnt it all rerouted and working today?

That's possibly a more important set of questions than why the exchange was sited at the bottom of the sea in the first place. After all there are many ways the network can fail other than flooding. I want to know that it wont stay broken for days the next time something bad happens.

Chuckles said...

Where's the failover? There isn't any.
Why isn't it working yet? Yawn.
I want to know this wont happen again!!! It will. Sure as night follows day.

Wake up people. These companies are run by accountants now. As long as there is one hapenny extra profit to be made by dumping all those lovely concepts like failover, redundancy, resiliency, self-healing, single point of failure, etc., they will be dumped.
Ditto for putting the primary and backup power at the lowest level 4 floors down under the water table.
And as long as it is cheaper to eventually grudgingly pay the compensation to the most persistent claimers, it is guaranteed.

Catosays said...

On the other hand, it might be a giganticApril Fool by CF.

If so, well done.

Anonymous said...

To have knocked out 400 local exchanges means this was central processor site - without it the remote units are effectively useless. There would normally be alternative routing between them, but I don't remember there being any backup CPU's when I worked for BT years ago. All modern equipment has on board PSU's and backup batteries, but these would only last for an hour or so, without mains...

selsey.steve said...

It's now 2341 and my connection to the 'Net has just come back.
Almost as glee-making as the news that the rail signaller's strike has been declared unlawful!

JohnRS said...

@anon 20:48

But surely the whole point of the design of the network (so I thought) was to eliminate things like a "central processor site" so that a problem in any one exchange didnt bring down the whole house of cards?

Confused, and still wondering about the answers to my questions from earlier.

Anyone from BT Openreach out there?

Anonymous said...

@ John R - that might seem the sensible option, but it isn't how things are done now. In the "good old days" every electromechanical exchange was completely self contained, and would continue to function even if the links to other sites were severed. This method continued with the later electronic controlled exchanges.

But when "digital" became the buzzword it allowed savings to be made by not duplicating CPU power. So now you will have one central site running many "remote" units. There is normally diverse routing between them, but knock out the brains, and you see results like this.

I also did some contract work for a telco who were replacing one main PABX for a large employer, with remotes in each of their buildings. These were all controlled from a site over 100 hundred miles away!

That, apparently, is called progress...