Issues at Manchester TCW

10:15 Our monitoring systems are currently alerting multiple failures at Manchester TCW.  Engineers are investigating and we will have more information shortly.  The nature of the failures do not indicate a single item failure.  We are waiting feedback from TCW remote hands before deciding our next course of action however engineers are prepping to go onsite.

10:27 Feedback from remote hands is our racks appear to be drenched in water.  Our Technical Manager is on his way to asses the damage and formulate workarounds where possible.

12:00 On-Site

12:50 Damaged equipment has been replaced.  A misconfiguration on a vendor side network has also been detected which will need rectifying - if the misconfiguration had not been present the impact on our clients would not have been so great.   Except for when we correct the configuration issue (which will not be today) we do not expect any further downtime.   Our sincere apologies for the downtime caused.

Access router in London HEX6&7

07:00 - We've lost contact with one of our routers in Harbour Exchange 6&7, while all other equipment is operational any directly connected services to this router will currently be down.  A power cycle through remote hands has been requested.

08:30 - I've put in some additional VLANs to enable some additional routing. Most clients should now be seeing connectivity.

09:05 - Telecity engineers have rebooted the affected router, the workarounds will stay in place for the remainder of the day or will be engineered further into an additional fallback option later on.  In the meantime routing will be sub-optimal.

Monitoring systems

A firewall that sits in front of our primary monitoring system is currently experiencing some troubles.  This has caused that system to raise false down alerts. Due to the number and frequency of the down alerts we have decided to disable the primary monitoring system until a solution can be put in place, the alerts have been causing considerable confusion. Key infrastructure is also monitored via a second system, this system however does not raise customer facing alerts and is generally used to gain confirmation of the primary system during outages so our core continues to be monitored.

Engineers are working to restore the primary system to operation as soon as possible.

Best Regards
Atlas Technical

DSL Network Issues

17:00 - We appear to be having issues in our DSL network, the problem itself is within our providers wholesale network.   Connections will appear to be running extremely slowly, you may find your router will drop the connection and try to re-establish the link, it is likely the reconnect will fail until the issue is resolved.  Unfortunately it is affecting connections coming into all our client facing DSL terminating routers in both Manchester and London so it is not possible for us to work around the issue at this time.    We're waiting for feedback from our provider as to a fix time/update.   Our fixed line and metro wireless services are not affected.