Issues at Manchester TCW

10:15 Our monitoring systems are currently alerting multiple failures at Manchester TCW.  Engineers are investigating and we will have more information shortly.  The nature of the failures do not indicate a single item failure.  We are waiting feedback from TCW remote hands before deciding our next course of action however engineers are prepping to go onsite.

10:27 Feedback from remote hands is our racks appear to be drenched in water.  Our Technical Manager is on his way to asses the damage and formulate workarounds where possible.

12:00 On-Site

12:50 Damaged equipment has been replaced.  A misconfiguration on a vendor side network has also been detected which will need rectifying - if the misconfiguration had not been present the impact on our clients would not have been so great.   Except for when we correct the configuration issue (which will not be today) we do not expect any further downtime.   Our sincere apologies for the downtime caused.

Access router in London HEX6&7

07:00 - We've lost contact with one of our routers in Harbour Exchange 6&7, while all other equipment is operational any directly connected services to this router will currently be down.  A power cycle through remote hands has been requested.

08:30 - I've put in some additional VLANs to enable some additional routing. Most clients should now be seeing connectivity.

09:05 - Telecity engineers have rebooted the affected router, the workarounds will stay in place for the remainder of the day or will be engineered further into an additional fallback option later on.  In the meantime routing will be sub-optimal.

Monitoring systems

A firewall that sits in front of our primary monitoring system is currently experiencing some troubles.  This has caused that system to raise false down alerts. Due to the number and frequency of the down alerts we have decided to disable the primary monitoring system until a solution can be put in place, the alerts have been causing considerable confusion. Key infrastructure is also monitored via a second system, this system however does not raise customer facing alerts and is generally used to gain confirmation of the primary system during outages so our core continues to be monitored.

Engineers are working to restore the primary system to operation as soon as possible.

Best Regards
Atlas Technical

DSL Network Issues

17:00 - We appear to be having issues in our DSL network, the problem itself is within our providers wholesale network.   Connections will appear to be running extremely slowly, you may find your router will drop the connection and try to re-establish the link, it is likely the reconnect will fail until the issue is resolved.  Unfortunately it is affecting connections coming into all our client facing DSL terminating routers in both Manchester and London so it is not possible for us to work around the issue at this time.    We're waiting for feedback from our provider as to a fix time/update.   Our fixed line and metro wireless services are not affected.

At risk notification Manchester 30th July 2013

One of our upstream transit providers have informed us they are upgrading their infrastructure in Manchester.

The work is taking place on 30th July 2013 between the times of 11pm and 6am the following day.  We are to expect one hour of downtime during this window.

In itself this should not be service affecting to any of our clients as traffic will flow through other pathways and our other transit partners, though as the service switches to the alternate providers there may be some routing instabilities.

Kind Regards
Atlas Technical

Fibre issues in Nantwich/Aston areas.

10:30pm 12/3/2013

Our monitoring system has picked up failures on both the North and South sets of fibre bundles feeding our Bunker Datacentre.

Traffic is however flowing via another pathway so the datacentre remains connected to the rest of the network, due to the increased load however some contention is apparent.

Customers in Nantwich taking direct Internet Access connections from us over EAD, these circuits connect via the datacentre and are unfortunately currently down, our apologies for the inconvenience.

--
10:45pm



Fibre provider is carrying out repairs to multiple sections of fibre, repairs are being carried out by multiple teams both to the North and South of the datacentre.

Completion of work is estimated by 6am, however our circuits are being given priority for reconnection.

Router issue in Manchester

12/Feb/2013 - 1:26am - Monitoring has detected a failure of a router in Manchester - Engineers are currently investigating the problem.  Services mostly affected are DSL and Leased Lines terminating in Manchester as well as some cross core activity.

12/Feb/2013 - 1:45am - Waiting response from datacentre.

12/Feb/2013 - 2:20am - Confirmation from datacentre of router reboot.

12/Feb/2013 - 2:30am - Router back online and exchanging traffic.