Watchfront - quality internet services for the business community

Apologies for any problems this may be causing.

As an aside, that power outage affected many other ISPs and could have caused some problems on the internet at large, however it did not directly affect our services.

To avoid this happening again, the short term solution is a subtle change to the config to avoid this particular sequencing problem. The longer term solution is to replace the routers with FB6000 routers, which we are hoping to do in the next few weeks

Apologies for any problems this may have caused

We are in the process of replacing our core routers with FireBrick FB6000s which should not suffer from this sort of problem.

As soon as we get more information from Be we will let you know. Apologies for any problems this may have caused.

We have now fixed the configuration error, re-introduced the router into the cluster, and all is working correctly.

Apologies for any problems this may have caused you.

It looked like a switch problem (on one specific VLAN broadcast packets got through but no unicast), so we moved the router to another switch. However when we brought it up it caused the 3 minute outage at 14:22, due to an OSPF flap causing the other routers to reset BGP. Odd!

We have taken the router back out while we investigate further. Apologies for any problems caused.

Apologies for the inconvenience.

We have layer 2 links between our data centres, for core routing, with backup tunnels over transit. In the early hours, the link to Amsterdam, provided by Packet Exchange, suffered a brief outage. Unfortunately, due to an error in the backup tunnel configuration, the backup tunnel started to flap. This tripped a bug in our core routers which killed OSPF (which is how our core routers negotiate routing between themselves). This resulted in our routers in London and Maidenhead announcing routes to Amsterdam (and vice-versa) that weren't actually working. The result was that if you tried to access machines in our Amsterdam data centre from source IPs that would normally route via London or Maidenhead, then you would fall into a black hole. And because OSPF had now died, routing didn't recover when the Packet Exchange link came back up.

The solution was to reboot the routers, which restored OSPF and everything returned to normal. We have now identified and rectified the config error, so it shouldn't happen again. To be absolutely sure, we intend running some intrusive tests on the live system, to make sure it now fails over correctly. Sadly this will cause some brief routing outages (which we will announce in planned maintenance). Because of the complex sequence of events that led to this failure, we believe that testing the live system is the only way to be sure we have fixed it.

If you have any further queries please contact support.

We think we can provide a solution in a matter of weeks and we will keep you informed as to when the solution is in place.

A brief update - we know that engineers replaced a master fuse in the main power board, however we don't know why it failed in the first place. We will hopefully have more information tomorrow.

We believe that our systems are back to normal. Please contact us if you are still experiencing problems.

Apologies for the problems we know this will have caused you.

Due to a very unfortunate sequence of events this didn't raise the necessary alarms, and went unnoticed until 7am. However we are now reasonably confident that things are stable, and we have identified and rectified the faults that led to these outages.

Once again I do apologise for this. It was a very unfortunate stream of events, and we are doing our best to learn from it and avoid it happening again.

When a bug causes a crash in this way, the FB6000 LNS automatically reverts to an older firmware version. It did this several times until it found a stable version that worked. This is why the lines reconnected several times. By 16:00 it was stable. I guess the only silver lining is that this FB6000 mechanism for automatically reverting to older stable firmware has proven to be effective.

There may be a further very brief outage once the main LNS is fixed and lines are transferred back over.

Once again I apologise for any inconvenience caused.

The outage appears to have been caused by our LNS (the box that terminates ADSL L2TP sessions at our end), and we believe the problem is now fixed.

Apologies for any inconvenience caused

We have traced the problem to an issue with disk drive configuration, which is now fixed and the server is back to normal.

Apologies for any inconvenience this may have caused

For your information we are "in discussions" with an alternative provider for that part of our service which BT provides. This may be of particular interest to those of our customers who have multiple bonded lines and would dearly like more resilience.

We will keep you informed.

Please accept our apologies for this outage.

All routers should re-connect automatically, however we have observed that some older equipment such Binatone routers may need to be power cycled to re-establish their connection.

This work BT is carrying out is causing disruption to all ISPs and frankly we find it appalling that their network design necessitates such outages for routine maintenance. This matter has been raised with BT at the highest level available to us.

We are sorry for the disruption but it really is beyond our control.

Re-connects for customers with multiple bonded lines will be phased to ensure previously dropped connections have re-connected before moving to the next line.

We will be monitoring all connections to make sure they re-establish.

The window for disruption is from 2300hrs, however work is not anticipated to commence before 1200hrs but is likely to take up to 4 hours

Routers should reconnect automatically.

We are sorry for issues this may cause you.

We apologise for this short outage.

We are considering putting further upgrades on hold until we are satisfied that there will not be a recurrence of these problems.

We sincerely apologise to those customers currently affected and we are doing everything we can to resolve the issue.

We decided not to replace the Cisco box at this stage, though if we see further problems this may still be an option.

Incidentally we are seeing unusually high levels of traffic, possibly due to the release of the Windows 7 Beta. This is not going to help.

As mentioned before, this only affects ADSL still on the old 20CN BT network - lines that have moved over to the new 21CN are unaffected.

We will continue to monitor the situation

We are also considering fitting a more powerful variant, as part of the planned maintenance on Sunday (see planned maintenance).

All our ADSL connections from the 20CN (ATM) BT network come into a single rack. Although they come in on two BT Centrals, all traffic goes through a single ATM box that translates between BT's ATM and our IP network.

Today we suffered a very unfortunate sequence of events. Firstly power to the entire rack was tripped. When that was restored, we diagnosed a faulty power supply. A replacement was sourced and fitted, but we then found the ATM box had failed. A replacement for that was sourced and fitted, but the engineers had trouble configuring it. Simultaneously an alternative ATM box was sourced and fitted (plan B). The plan B box finally started working about 18:00. Incidentally the plan A box also started working at 18:20.

As you may be aware, for many years we have been improving our infrastructure to use multiple redundant systems and interconnects, so as to avoid a single failure taking down the network. Sadly this ATM box was one of the last remaining single points of failure. We are in the process of moving all ADSL over to BT's new 21st Century Network, which is delivered by IP over multiple redundant links. Sadly delays by BT mean this hasn't happened yet, so we still need the ATM box.

Just to clarify, today's outage only affected 20CN ADSL connections. ADSL connections on 21CN were unaffected, as were all our other services (server colo etc.)

Not surprisingly, we have had several clients ask us how they can avoid such outages in the future. The simple answer is to have multiple redundant internet connections, with no single point of failure. For example, you could have a backup connection using 3G. If you are interested, please do talk to us about your requirements, as we are continually developing new solutions (and as always can offer bespoke solutions if needed).

We will of course carry on working this evening until it is fixed. We will also be answering support phone calls (020 7517 4902) and emails support@watchfront.net , so please don't hesitate to contact us if you need to.

Once again thank you for your patience

Thank you for your patience, and please rest assured we are doing all we can to rectify the problem.

I do apologise for this

Our other services (colo etc) were unaffected.

Apologies once again for any inconvenience this may have caused.

Apologies for any problems this may have caused.

There was an ethernet loop inadvertantly introduced into the core ADSL network, due to a supplier adding an ethernet port as requested, with spanning tree enabled, for resilience. Sadly the spanning tree didn't do its job, resulting in an ethernet loop causing very high packet loss.

This was thankfully easily rectified, and the network is now stable. Apologies for any problems this may have caused.

We are not 100% sure what IP addresses were affected, but it will have affected many of our customers. For your info the outages were from approx 11:10 to 11:22, and 11:32 to 11:36.

Once again we do apologise for any problems this may have caused.

Most of the affected lines reconnected automatically, but unfortunately some did not - if still affected, power-cycling your ADSL router should clear the problem.

Apologies for the inconvenience. If we receive an explanation from BT we will post an update.

"On October 4th, approximately between 16:03 hrs and 18:00 hrs, a large area of Amsterdam has suffered from a severe power outage. Unfortunately also Science Park Amsterdam, where one of the SARA Datacenters is located, was confronted with this failure.

The Emergency Power Supply took over the major part of the power delivery, but unfortunately one of our four UPS systems malfunctioned. As a result some customers experienced a short outage of approximately 6 seconds on one of their two powerfeeds. After the 6 seconds the generators came online and provided full power on all feeds, including the affected one. However, a small number of racks needed a manual reset and was affected longer. The failure had consequences among others for the internet traffic and some other SARA services. At this moment most services are restored.

By now the external power situation is stable and normal again. SARA is investigating the cause of the malfunction of the UPS system. If needed, the generator will preventive be put online and running.

We are sorry for the inconvenience. If you need assistance you can contact us for support by one of our onsite engineers.

Apologies for any inconvenience this may have caused."

If you are without service for a long period it could mean that your router needs to be power cycled (turned off then switched on again). We will do our very best to minimize any disruption.

The problems are clearly due to video streaming of the Olympics, which caused much higher traffic levels than expected, and if it is any consolation most other ISPs are suffering similar problems.

We are doing everything we can to expedite a faster connection with BT, so we don't have similar problems in the future.

Once again, we do apologise for the problems this must have caused.

We are hopeful that the extra capacity we are expecting should be available to us very soon which would eliminate this problem. Please be assured we are doing all we can to return service to the performance levels we take pride in providing for you.

Traffic levels are slowly returning to more acceptable levels. We expect to finally be able to start moving traffic to the new Gigabit interconnect with BT within the next 2 to 3 weeks. This will ensure the normal good level of service our customers expect of us is restored.

Again we do apologise.

The datacentre have issued the following statement:-

Unfortunately on this occasion, a configuration error resulted in an old version of firmware being loaded. This alone should not have been a problem, but unfortunately the older firmware was incompatible with the current configuration in a very subtle way, resulting in unexpected behaviour which took significant time to diagnose.

Once the cause became clear, the correct firmware and configuration were restored and service resumed. Due to the way BT's network reacts to outages in our network, the restoration of connections occurs over a period. Most lines were working again by 19:15. Some lines took longer, some customer routers also needed resetting.

Steps have been taken to prevent such a chain of events happening again. We also apologise for the time it took us to get a status message posted in our support area. This is being corrected.

Finally, some customers had difficulty contacting us by phone. Please use 020 7517 4902 - this also automatically calls a mobile number 07812 811116 as well.

This is not the first complete power outage Redbus have had recently, and we are asking serious questions as to how they will ensure it doesn't happen again, and looking at ways of relocating our servers and routers so as to minimise the risk we are exposed to.

For your information, Redbus power supplies are meant to be backed up by battery and generator, and surge protected, so this kind of failure really shouldn't happen!

Apologies for the delay in posting this status message, and of course apologies for the problems this outage undoubtedly caused you.