Service Outage report for 10th November 2010

17:32-

We've just had confirmation that the outage was from two separate faults happening in two separate geographic locations, one fault was on the providers Leeds to Sheffield connection, the other on their Warrington to Birmingham connection.



13:42-

At 10:50am this morning we lost both our west and east-bound connections from Manchester to London, this had the outcome of partitioning our core network into two. This partitioning would have caused routing issues and due to the location of name and radius servers within the network name lookup and xDSL authentication would also have failed.

Our transit feed out of Manchester was also experiencing problems which as this issue cleared at the same time our connections came back up was no doubt down to the same core root problem.

With the issue affecting multiple providers it was clear the problem was itself not within any equipment within our direct control or the outcome of any of our actions within the network.

Our main telephone system is also based out of Manchester however when the server went offline it failed over onto the backup analogue PSTN system, the number of incoming calls obviously proving a challenge.

We're currently in discussion with our network provider for the Manchester to London connections as these routes should be separate and diverse, initially they also went via separate providers however due to consolidation within the market one provider has ended up owning both networks. If it transpires that our provider has without our knowledge or authorisation joined these pathways then of course action will be taken.

At 12:40pm both connections came back up, with the exception of transit our of Manchester once the network had re-converged connections and traffic flows returned to normal. Approximately ten minutes after our connections re-established transit via our transit provider also re-established.

Our apologies for this outage and the inconvenience.