Main Image
Status History
This page contains old Status announcements that are no longer relevant, but could be of interest.

When Who What
2008-10-14 09:55:23kevAt 9:40 this morning we again lost all connectivity to our Amsterdam facility. It was restored at 9:50 - we are awaiting an explanation from our suppliers and will post an update when we receive one.
2008-10-07 10:29:37kevLast night between 2:00am and 3:45am, most (approximately 90%) of our ADSL connections were reset by BT. This was not notified in advance and as yet we have no explanation.

Most of the affected lines reconnected automatically, but unfortunately some did not - if still affected, power-cycling your ADSL router should clear the problem.

Apologies for the inconvenience. If we receive an explanation from BT we will post an update.

2008-10-04 20:37:23kevWe have just received the following explanation for today's outage in Amsterdam:

"On October 4th, approximately between 16:03 hrs and 18:00 hrs, a large area of Amsterdam has suffered from a severe power outage. Unfortunately also Science Park Amsterdam, where one of the SARA Datacenters is located, was confronted with this failure.

The Emergency Power Supply took over the major part of the power delivery, but unfortunately one of our four UPS systems malfunctioned. As a result some customers experienced a short outage of approximately 6 seconds on one of their two powerfeeds. After the 6 seconds the generators came online and provided full power on all feeds, including the affected one. However, a small number of racks needed a manual reset and was affected longer. The failure had consequences among others for the internet traffic and some other SARA services. At this moment most services are restored.

By now the external power situation is stable and normal again. SARA is investigating the cause of the malfunction of the UPS system. If needed, the generator will preventive be put online and running.

We are sorry for the inconvenience. If you need assistance you can contact us for support by one of our onsite engineers.

Apologies for any inconvenience this may have caused."

2008-10-04 17:53:05kevWe have just received the following statement from our supplier:-
Power appears to have restored to our equipment in SARA and all services should be back up and functioning normally at the moment.
We have, however, not yet received the formal all clear notification so services that were down are currently considered at risk.
Further updates to follow.
2008-10-04 16:13:01kevConnectivity has just been restored. If we receive any explanation from our suppliers we will post an update.
2008-10-04 15:54:15kevWe have just been notified that there is currently a power failure in another Amsterdam datacentre which is affecting various connections. We have been told that engineers are working on the fault but no ETA is yet available. We will update as we find more.
2008-10-04 15:51:32kevWe have confirmed that power to our Amsterdam equipment is OK - investigation continues.
2008-10-04 15:17:44kevAll connectivity to our Amsterdam equipment is currently down - we are investigating and will update ASAP.
2008-09-25 09:20:55philThe earlier incident with BT lead to a latent problem within our core router which has been resolved. We are sorry for the outage.
2008-09-25 09:05:50philThe current service issue is that there has been a "Serious Incident" at Stepney Green. Sorry, we do not have more details at this time, we will update you as soon as we do.
2008-09-11 11:52:39philA brief glitch occurred on a BT link for no apparent reason, affecting some of our customers. It will be necessary to carry out some traffic rebalancing as a result, this will cause a very brief outage on some lines.
2008-09-10 18:53:03philNormal e-mail service is resumed, sorry for the inconvenience.
2008-09-10 16:35:49philThere is currently an issue with our mail server which we are attending to. Normal service should resume shortly and we are sorry for the inconvenience.
2008-08-25 10:35:11philOur network is performing without any problems, however some customers may need to power cycle their routers if they are still without service. Thank you for your patience.
2008-08-24 19:27:28philThe planned change has now been made successfully, we are sorry if this essential work caused you any problems. We will keep an eye on the situation to make sure everything remains stable.
2008-08-24 15:49:47philThe planned change this morning failed. Another attempt will be made early this evening.
2008-08-22 18:05:31philWe are carrying out maintenance this Sunday morning which involves replacing our core L2TP router. This will cause an outage, which if everything goes well will last for a few seconds. If there are issues we do not expect the outage to last for more than about 20 minutes.

If you are without service for a long period it could mean that your router needs to be power cycled (turned off then switched on again). We will do our very best to minimize any disruption.

2008-08-20 12:26:47benThe ongoing ADSL packet loss problem has improved today, as traffic levels are considerably lower. We are expecting matters to improve further when the Olympics end at the weekend.

The problems are clearly due to video streaming of the Olympics, which caused much higher traffic levels than expected, and if it is any consolation most other ISPs are suffering similar problems.

We are doing everything we can to expedite a faster connection with BT, so we don't have similar problems in the future.

Once again, we do apologise for the problems this must have caused.

2008-08-18 16:53:50philWe are sorry to advise you that the ADSL packet loss being experienced last week is occurring again this week. We believe this to be due to video streaming of the Olympics from various web sites which has lead peak demand to be 20% up.

We are hopeful that the extra capacity we are expecting should be available to us very soon which would eliminate this problem. Please be assured we are doing all we can to return service to the performance levels we take pride in providing for you.

2008-08-15 17:25:28philSome of our ADSL customers have been experiencing periods of packet loss over the last few days, for which we do apologise. This has been caused by a combination of extra-ordinary traffic levels due to an exceptionally large security patch update from Microsoft, significant extra media streaming due to the Olympic Games, combined with the extra capacity we need from BT being delayed.

Traffic levels are slowly returning to more acceptable levels. We expect to finally be able to start moving traffic to the new Gigabit interconnect with BT within the next 2 to 3 weeks. This will ensure the normal good level of service our customers expect of us is restored.

Again we do apologise.

2008-08-04 09:22:32philWe have been experiencing an issue with one of our core routers. A reboot appears to have cleared the problem. We apologies for the inconvenience.
2008-06-29 17:19:58kevJust before 5:00am this morning we suffered a power failure in our Maidenhead datacentre affecting all equipment. The outage lasted approximately 30 minutes. Apologies to all customers with services provided from Maidenhead. London and Amsterdam were unaffected.

The datacentre have issued the following statement:-

Following on from the issues this morning, we are continuing to work with suppliers to find the cause of the outage.

Current investigations appear to have identified a phase detection relay which may have failed in the generator auto-changeover panel. This could have caused the panel to think there was a mains power failure when in fact there had not been.

The UPS systems worked as expected without fault, however due to the mains being forced off by the potentially failed relay, and power from the generators also locked out, the UPS batteries fully discharged after 10 minutes resulting in a critical power failure.

We have engineers collecting a new relay, along with other spares that maybe needed from the manufactures in North Yorkshire. We expect these to arrive in the early hours of tomorrow morning.

During this time we have manually disabled the mains auto changeover system in order to aviod any potential further mains cuts. In the event of a real power outage on site we will have staff perminatly onsite to operate the generator changeover procedure manually.

This should be treated as an 'at risk' period for customers in BlueSquare 2 & 3 until further notice.

Once we have further information we will update this ticket.

We apologise for any inconvenience caused.

2008-06-27 11:46:28benAn update on yesterday's 10 minute outage - this was due to a massive DoS attack that took down our L2TP connections. Over the past few years we have been improving our resilience to attacks, and we have been riding most attacks without any noticeable affects. However the magnitude of this latest attack was a first for us. The good news is that we are developing new L2TP boxes that should ride even this magnitude of attack.
2008-06-26 15:01:10benSome ADSL routers have failed to log back in after the outage. If you are still having problems, please try power cycling the ADSL router. If that doesn't help, please contact us and we will investigate further
2008-06-26 14:57:28benOur ADSL services suffered an outage at 14:15 today, of around 10 minutes duration. It appears that this was due to an attack that took down our L2TP connections with BT. We are investigating exactly what happened, so we can take steps to avoid similar problems in the future. Apologies to all those affected.
2008-06-26 14:32:04benServices are now returning to normal. I will post more details shortly. Apologies for the outage
2008-06-26 14:27:20benWe are currently suffering a major outage More news shortly
2008-06-19 09:28:58benThere was a brief problem with routing around 9am this morning, that may have affected some customers. It was due to a problem with BGP that has now been resolved. Apologies for any inconvenience this may have caused.
2008-06-11 12:00:07benWe are currently seeing low levels of packet loss on some ADSL lines. We are working on this, please bear with us.
2008-05-02 12:52:03kevWe have just received the following message from one of our transit providers:-
"Just for information the London Internet Exchange has been suffering some technical problems over the last few hours that has caused widespread disruption to connectivity in the UK. They are working to resolve the situation as soon as possible."
Apologies for any poor connectivity experienced in the meantime.
2008-03-20 13:37:00philA very limited number of customers experienced a brief outage this morning at about 10:00hrs. An error in the database of a radius server prevented some customers who had logged out, (possibly as the result of a glitch within BT) from immediately logging back in again. The fault has been rectified and we apologise for the inconvenience.
2008-03-08 20:58:56kevThe cause of the earlier problem has been fixed (it was a permission error on a configuration file). Normal routing is now restored through the main router, with seamless fallover to the backup once again working as intended.
2008-03-08 20:38:25kevWe just suffered a brief routing outage. During a routine router software update, routing failed to hand over immediately to a backup router, but instead took a couple of minutes to do so. Apologies for the outage - the handover mechanism is being investigated.
2008-01-09 12:27:07kevWe are very sorry for the long outage affecting all our ADSL customers on Monday evening. The problem arose around 16:00 when a core router detected an internal error and restarted itself. The cause of the error is being investigated, but such a restart should cause the router to load fail-safe firmware and resume normal operation within 10 seconds.

Unfortunately on this occasion, a configuration error resulted in an old version of firmware being loaded. This alone should not have been a problem, but unfortunately the older firmware was incompatible with the current configuration in a very subtle way, resulting in unexpected behaviour which took significant time to diagnose.

Once the cause became clear, the correct firmware and configuration were restored and service resumed. Due to the way BT's network reacts to outages in our network, the restoration of connections occurs over a period. Most lines were working again by 19:15. Some lines took longer, some customer routers also needed resetting.

Steps have been taken to prevent such a chain of events happening again. We also apologise for the time it took us to get a status message posted in our support area. This is being corrected.

Finally, some customers had difficulty contacting us by phone. Please use 020 7517 4902 - this also automatically calls a mobile number 07812 811116 as well.

2008-01-07 19:54:48philThe problems have been fixed and normal service has been resumed. We are sorry for the considerable inconvenience. Some changes have been made to prevent this problem happening again.
2008-01-07 18:54:22philWe are experiencing core routing problems. We are trying to solve the issue as soon as possible. We do sorry for the inconvenience.
2007-12-12 17:25:28philTransit between London and Amsterdam is now restored. We sorry if you suffered any temporary disruption.
2007-12-12 17:02:08philWe are currently investigating a possible transit issue with our data centre in Amsterdam. We are sorry for the disruption.
2007-12-11 14:23:27benWe are seeing small amounts of packet loss on our ADSL provision, due to a problem with one of the BT Centrals. We are working to resolve the problem ASAP. Apologies for any problems this may cause.
2007-10-16 15:50:36benToday's outage at our Amsterdam data centre was due to a problem with our transit provider. It appears that one of their switches locked up, and it took them an hour to diagnose and rectify the problem. Apologies again for any inconvenience this may have caused.
2007-10-16 14:34:39benOur Amsterdam data centre is now back up, and we are investigating what happened
2007-10-16 13:47:24benWe appear to have connectivity problems to our data centre in Amsterdam. We suspect there may have been a power outage, and are investigating. This will affect anyone who has a hosted server in our Amsterdam facility. Apologies for the inconvenience.
2007-08-04 07:12:25benWe are seeing intermittent packet loss on some ADSL lines, and are investigating.
2007-07-20 16:47:14benWe now have some more information - it was a particularly nasty DoS attack that took out all our ADSL between 16:12 and 16:28. Our other services were unaffected. Most lines came straight back, but some have been slow to recover. If you are still experiencing problems please contact us. Apologies again for the inconvenience.
2007-07-20 16:38:19benMost ADSL came back at 16:28, but some lines are still causing trouble. We are investigating. Apologies for the inconvenience.
2007-07-20 16:22:07benWe are suffering an ADSL outage and are investigating, more shortly
2007-07-06 17:06:30kevWe suffered a DDoS attack at 16:36 which affected all ADSL customers. Services are currently gradually returning to normal. Apologies for the outage.
2007-06-27 11:17:47benOur main transit provider (AAISP) now has a new primary core router installed, to replace the troublesome Juniper. After some thorough testing it looks as though both the primary and backup routers are working well, and this should be an end to the problems we had over the last couple of days. Thank you for your patience.
2007-06-26 11:08:56benIn view of the routing problems experienced over the last couple of days, we are working with our main transit provider to urgently replace the core routers over the next few days. Naturally we will do our utmost to prevent any disruption, though we may be running on backup routers at times. As ever we are striving to provide the best possible service.
2007-06-26 10:57:43benThere was a burst of severe packet loss around 10:40 this morning. This appears to have been another BGP related problem for our main transit provider, AAISP. We are currently working with them to replace the core routers. In the meantime we apologise for any inconvenience this has caused.
2007-06-26 10:40:49benWe are currently seeing serious packet loss on Level 3 transit, investigating
2007-06-25 14:59:21benAn update on today's two outages. The first outage was due to the failure of a core router. Our main transit provider (AAISP) runs dual redundant core routers, using VRRP to switch between them. The primary router failed in such a way that it was still announcing VRRP (so the backup router didn't take over), but was not routing traffic. This is a disastrous failure mode for a router, and they are investigating how it could happen. The result was a routing outage that would have affected most if not all of our customers. The second outage was due to a BGP failure between the L2TP box our DSL customers are logged on to, and the upstream router. This resulted in a 7 minute outage for all our DSL customers, but did not affect other services (hosted servers etc.). It was unrelated to the first outage, and we are investigating what caused it. We do apologise for any problems this may have caused, and we are planning changes to our network to improve resilience to these sorts of failures.
2007-06-25 13:56:20benOur main transit provider has suffered another routing problem between 13:42 and 13:49, which as far as we can tell was unrelated to the earlier problem. We will post more information here as it becomes available. Apologies for any problems this has caused.
2007-06-25 13:12:01benWe have suffered a routing outage from 12:49 to 13:08 today. We are investigating the cause and will post more information shortly.
2007-06-13 09:57:16philWe have been seeing some low levels of packet loss on broadband lines. We are investigating although the situation has returned to normal, as this is not the quality of service we aim for.
2007-06-12 09:16:53benThere was a DoS attack at 9:04 this morning which affected our L2TP box, thus affecting all our ADSL customers. The outage lasted about 5 minutes, and all services are now back to normal. Apologies for any inconvenience this may have caused you.
2007-05-30 14:41:31philWe have just experienced a DDoS attack, which affected our core network router. The newly installed router failed to recover automatically and we are now running on a backup system while we investigate exactly what went wrong. This is of great concern to us and we will take what ever measures are necessary to prevent a repeat of this problem. We are sorry for the problems we may have caused you.
2007-05-22 15:55:08philWe have just experienced a severe DDoS attack, which affected our core network and took a few moments for routing to be re-established. Service is now back to normal and we apologise for this short interruption.
2007-05-10 10:09:49philThere was an incident last night causing one of a pair of BT centrals to be dropped. When BT returned full service it was necessary to re balance the load over the BT centrals. When doing this some customers may have experienced a short outage. Please accept our apologies for any inconvenience this may have caused you.
2007-04-27 16:31:08kevThis Friday (after midnight) the L2TP machine is being reloaded. This should not result in any problems. If on Saturday there is no connectivity on one of your lines please reboot the appropriate router. On Sunday, one of the 2 core routers is being replaced. This should have no impact as the network is designed to operate on just one of them should the other fail. If your connection is lost for some reason during this operation *do not* reboot your router. If you have any problems please call our support number +44 20 7517 4902
2007-04-23 00:05:06kevThere has been a spate of short outages for ADSL customers this evening. An earlier DDoS attack had caused a core router to revert to running an old version of software, which was unstable under certain specific conditions which unfortunately occurred this evening. It has now been updated and service should now be stable. If any lines remain out of service, power-cycling the ADSL router should restore them. Apologies for the inconvenience.
2007-04-04 15:21:10kevWe have just experienced another DDoS attack, which once again resulted in loss of service for most customers. Filtering has been employed and service has returned to normal. Apologies for the inconvenience
2007-03-30 17:05:12kevWe have just experienced a DDoS attack, which resulted in loss of service for most customers. Filtering has been employed and service has returned to normal. Apologies for the inconvenience
2007-03-28 14:41:28benAn update on today's outages - they were caused by the failure of a machine that forms part of our transit provider's BGP routing mechanism. There is a backup machine that should mean that routing is unaffected by such a failure, but that backup mechanism failed as well. Our transit provider is taking this very seriously, and working to rectify the problems with both the primary and backup machines. We hope to have full resilience restored by the end of today.
2007-03-28 14:09:55benWe have just suffered another transit outage due to problems experienced by our transit provider. All services should now be back to normal. We apologise for any inconvenience this may have caused.
2007-03-28 13:00:40benWe have just suffered a brief transit outage at our data centre in London. We believe this was due work being carried out by our transit provider. This will have affected access to servers in our data centre (our website, email, and hosted servers), plus bonded ADSL customers. All services should now be back to normal. We apologise for any inconvenience this may have caused.
2007-03-16 15:48:11benAt 15:20 today there was a problem within BTs network which affected some of our ADSL customers. The outage was brief but some ADSL lines are not yet back - if your line is still down please try power-cycling the ADSL router. Apologies for any inconvenience this may have caused
2007-03-04 11:21:58benOur ADSL services suffered a brief outage yesterday (Saturday) afternoon, due to a problem with the L2TP router. Service was quickly restored, however we have noticed that some customers' ADSL routers have not reconnected. If your ADSL is still off, please power-cycle your ADSL router, that should reconnect you. Apologies for any inconvenience this caused.
2007-01-23 11:18:41benWe are currently seeing a small amount of packet loss on one of our BT Centrals, affecting some ADSL customers. If you are having problems, please contact us and we will move your line to another BT Central while the problem is resolved.
2007-01-22 13:13:20benAt 13:08 our DSL service suffered a 20 second outage, due to a glitch at our end. All services are now back to normal. If your DSL connection is still down, please try power-cycling your router. If you still have problems, please contact support. Apologies for the inconvenience
2007-01-21 12:02:35benOur primary mail server is now back up following planned maintenance. If you use webmail or imap, you may have to re-subscribe to your mail folders.
2007-01-21 11:03:28benAs explained in our Planned Maintenance, we are now taking our primary mail server down for maintenance. It should be back by 12 noon.
2007-01-15 14:02:09benOur main transit provider (AAISP) suffered routing problems at around 13:40 today, resulting in intermittent connectivity for all our customers for around 5 minutes. Services are all now back to normal.
2007-01-15 13:50:19benWe just suffered a brief outage, services are now restored and I'm just trying to establish what happened. Apologies for any problems this may have caused
2007-01-12 13:43:13benWe are currently seeing packet loss on around a third of our ADSL lines. It appears to be a problem within BT's network, and we are chasing them to get it sorted. Apologies for any problems this may cause.
2006-11-14 13:28:06benAn update on the outage - we had a problem with level-3 transit which caused a network storm, briefly taking out both our transit and ADSL connectivity. Our Level-3 transit is currently disconnected and all services are back to normal. If your ADSL is still off line, please try power-cycling the router.
2006-11-14 12:55:08benWe have just suffered a brief outage from 12:44 to 12:50, which will have affected all customers. We are investigating what happened, and do apologise for any inconvenience this may have caused.
2006-10-05 14:30:00JBMOur mail server has suffered a couple of brief outages today, due to hard disk problems. It is now stable again. Apologies for any inconvenience this may have caused
2006-09-25 09:50:00JBMAll customers will have experienced several outages yesterday (Sunday). This was due to a power outage at Telehouse (not our data centre), which resulted in large parts of the UK Internet failing. We were working on our ADSL connectivity at the time, following the problems on 2006-09-21 (see below), and so the outage caused some confusion! Also as power was restored at Telehouse, our main transit provider suffered some sort of network storm, adding to our problems. So generally a rather disastrous day - I do apologise for any inconveniences this may have caused.
2006-09-21 16:30:00JBMThis afternoon's outages were due to an intermittent fault with the ATM interface to BT's network. We believe it is now stable, and will be carrying out maintenance work on Sunday to ensure it doesn't happen again. This may cause a further brief outage for DSL customers. Apologies for any inconvenience this may cause.
2006-09-21 14:30:00JBMAll DSL is back up, more details shortly
2006-09-21 14:25:00JBMAll DSL is off again, many apologies, more details shortly
2006-09-21 14:03:00JBMOur DSL is back, and we are investigating! Some ADSL routers are being slow to reconnect - if yours is still down, try switching it off and back on again.
2006-09-21 13:55:00JBMThere is currently another outage of all DSL connections. We are doing our best to sort it! Apologies for the inconvenience
2006-09-21 13:31:00JBMAll DSL services are now back to normal. The ATM interface to BT's network failed, we are investigating why. Apologies for the inconvenience
2006-09-21 13:18:00JBMThere is currently an outage of all DSL connections. We are working on it and hope to have service restored very shortly. Apologies for the inconvenience
2006-09-14 16:25:00JBMWe have just experienced a DDoS attack, which has resulted in intermittent packet loss for DSL customers. Service now appears to be back to normal. Apologies for the inconvenience
2006-09-01 14:38:00JBMService should now be back to normal. It looks like a DoS attack upset the BGP routing of our main transit provider, though we are still investigating. Apologies for the inconvenience
2006-09-01 14:31:00JBMWe are currently experiencing problems with our transit connection, resulting in packet loss for all customers. We are working on it and will keep you posted. Apologies for the inconvenience
2006-07-18 18:53:00JBMFollowing the earlier DoS attack, we have had to reboot our DSL endpoint, resulting in a further outage for ADSL and SDSL customers, for which we apologise. If you are still experiencing problems please try power-cycling your DSL router.
2006-07-18 18:24:00JBMWe suffered a DoS attack just now, which caused a high level of packet loss for a few minutes. Our service is now back to normal. Apologies for the disruption.
2006-06-28 14:21:00JBMAn update on the earlier DSL outage - the outage was from 13:20 to 13:45, and was caused by a failure of our ATM interface to the BT network, resulting in loss of PPP for all ADSL and SDSL customers. It is now working correctly, but some lines are being slow to reconnect. If you are still experiencing problems, please try power-cycling your router. If this doesn't help, please call support (020 7517 4902)
2006-06-28 13:49:00JBMOur ADSL customers all experienced an outage just now. Service is now back to normal and we are investigating what happened. It appears to be unrelated to yesterday's outage, a most unfortunate coincidence to have two outages in two days. Apologies for any problems this may have caused
2006-06-27 11:05:00JBMAt 10:36am this morning, our main transit provider lost routing. This resulted in loss of connectivity until service was restored at 10:53am. They have dual redundant routers, but it appears that the redundancy mechanism failed such that routing failed. They are investigating why this happened, in order to avoid it happening again. We do apologise for any problems this may have caused.
2006-05-30 12:35:00JBMAs promised, here is a more detailed report on yesterday's DDoS attack. Our network is resilient to most DDoS attacks, and normally rides the traffic surges with no impact on users. Our normal defence is to start giving ICMP errors for the attacked IP address, announcing no route to network, and DDoS attacks normally give up at this stage as they believe they have achieved their aim of knocking out the network.Yesterday's attack was unusual in that it didn't stop at this point. This in itself shouldn't have been a problem. However our main transit provider (AAISP) had installed a new core router on Sunday, and this reacted badly to the prolonged attack, resulting in a BGP flap (BGP is how we announce our routes to the world). This resulted in repeated loss of routing for all Watchfront IP addresses, and our BGP announcements were damped by the rest of the world, slowing the return to normal routing. As always, this has been a learning experience for us, and we hope to avoid similar problems in the future. If anyone would like more information, please call support on 020 7517 4902
2006-05-30 09:23:00JBMWe suffered a severe ongoing DDoS attack yesterday afternoon, which resulted in poor to non-existent connectivity for all our customers. Things are back to normal now, and we will post more details when the dust has settled. Apologies for any inconvenience this caused.
2006-04-27 16:23:00JBMWe suffered a massive DDoS attack at 16:08, which resulted in a few minutes outage for ADSL customers. Apologies for any inconvenience this caused.
2006-04-11 23:15:00KRHEarlier this evening at approximately 21:30, we suffered an outage caused by a large scale DDoS attack. Connectivity to hosted services was restored after a few minutes, but ADSL connectivity was not restored until approximately 22:15, as a router required rebooting. Apologies for any inconvenience this caused.
2006-02-15 12:52:00JBMAn update on the outage this morning. We have dual redundant core routers in each of our data centres, using VRRP to provide seamless switching between them. This allows for routine maintenance or failure without a break in service. A routine update to one of the routers resulted in it announcing by VRRP that it was able to route, when in fact it wasn't, and so the backup router didn't take over. The length of the outage was simply due to the time it took us to gain access and manually switch to the backup router. We are looking at ways to avoid this vulnerability in the future.
2006-02-15 11:03:00JBMWe suffered an outage of our core routers at Redbus HEX, between 10:40 and 11:00am this morning. All is now back to normal and we are investigating the cause of such an outage. Apologies for any inconvenience this caused.
2005-09-21 12:08:00JBMOur transit provider suffered an outage this morning between 11:40 and 12:00, which will have affected ADSL customers and access to servers in our data centre in London. Apologies for any inconvenience this caused.
2005-08-05 13:05:00KRHBT replaced the ATM card and service was restored at 12:38. Apologies once again for the inconvenience.
2005-08-05 12:25:00KRHThe BT Central pipe linking all ADSL connections to our network is currently down. We've just heard from BT that there is a faulty ATM card in their facility in Ilford and they are working to rectify this ASAP. Apologies for the inconvenience.
2005-06-27 12:30:00JBMOur main transit provider suffered another failure today, resulting in loss of transit in our London facility between 10:30 and 11:40 this morning. There was also a 10 minute ADSL outage from around 11:10. Our apologies for any inconvenience this may have caused. We are looking at ways to improve the reliability of our main transit feed!
2005-06-16 16:40:00KRHAt around 14:25 today a core switch failed in our main London transit provider's network. This resulted in loss of connectivity for all ADSL customers and hosted servers in our London datacentre. The problem was resolved by around 15:50. Our Amsterdam datacentre was unaffected and we were still able to reach our London datacentre via our Amsterdam datacentre, as were customers who have servers in Amsterdam. Our apologies for any inconvenience this caused.
2005-05-26 17:40:00JBMWe suffered a DoS attack at 17:27, lasting a few minutes, which affected our ADSL customers' connectivity. Apologies for any inconvenience this caused.
2005-03-26 18:20:00KRHThere was a 10 minute service outage at around 18:07 today affecting our hosting customers (including tunnel bonding FireBrick customers). This was due to a configuration error entirely of my own doing - apologies for the inconvenience.
2005-03-08 16:55:00KRHThere was a brief loss of ADSL service between 16:01 and 16:13 this afternoon due once again to a relatively minor DoS attack upsetting a BT router. We are still chasing BT for a long term fix for this issue. Apologies for the inconvenience.
2005-03-01 20:45:00JBMRedbus HEX, where our data centre is located, suffered a complete power outage today, which will have affected all customers. Power went off at around 10:05am, and services started coming back around midday. There was also a power surge which damaged a number of servers.

This is not the first complete power outage Redbus have had recently, and we are asking serious questions as to how they will ensure it doesn't happen again, and looking at ways of relocating our servers and routers so as to minimise the risk we are exposed to.

For your information, Redbus power supplies are meant to be backed up by battery and generator, and surge protected, so this kind of failure really shouldn't happen!

Apologies for the delay in posting this status message, and of course apologies for the problems this outage undoubtedly caused you.

2005-01-28 14:40:00JBMToday's brief loss of ADSL service was due to a DoS attack upsetting one of BT's routers. The attack was relatively minor and we are encouraging BT to improve the resilience of the router in question. Apologies for any inconvenience.
2005-01-28 13:30:00JBMThere was a brief outage just now affecting all our ADSL customers. All appears to be back to normal now and we are investigating what happened. If you are still having difficulties please call our support number.
2004-11-24 10:30:00JBMYou may have seen packet loss early this morning, between 01:40 and 07:15. This was due to a faulty switch in our main transit provider's rack, which has now been reset. Apologies for any inconvenience.
2004-11-09 13:55:00JBMHere's a more detailed post mortem of this morning's events. First the power went off at Redbus for 15 minutes from 00:20 - this really should not happen, and we are trying to get answers from Redbus. This resulted in loss of connectivity until 01:25. Then at around 06:54 a switch in our main transit provider's rack started flooding our network with broadcast packets, causing packet loss until it was reset at around 08:30 - we are not sure whether this was related to the power outage. Then finally BGP routing died between 09:04 and 09:41, as a result of side effects of the earlier power outage. Everything is now back to normal, and we are planning changes to reduce the possibility of this happening again.
2004-11-09 09:00:00JBMWe suffered a loss of power in our data centre this morning, which will have affected all customers. All services are now back to normal, and we will be asking Redbus some serious questions. Apologies for any inconvenience.
2004-10-29 09:40:00JBMYou may have seen massive packet loss this morning up to around 09:15. This was due to a faulty switch in our main transit provider's rack, which has now been reset. Apologies for any inconvenience.
2004-10-14 17:20:00JBMWe suffered an outage of our main transit provider this afternoon, between 16:42 and 17:08. This was due to a power outage in one of their racks. It will have affected most of our customers. Apologies for any inconvenience.
2004-09-21 15:38:00JBMADSL customers may have noticed a brief outage of a few minutes at around 15:30 today. We believe it was within BT's network. All is now back to normal and we apologise for any inconvenience.
2004-09-01 19:45:00KRHThe backup transit feed is now up and running again. Sorry again for any inconvenience or poor performance.
2004-09-01 18:55:00KRHOur backup transit provider is currently suffering a DOS attack which spilled over into our network, causing packet loss. The backup feed is temporarily suspended until this is resolved. Apologies for the poor performance between 17:30 and 18:30 - more later.
2004-06-15 17:40:00JBMThere was a major outage at our data centre this afternoon, between 16:40 and 17:20, which would have affected all ADSL and colocation customers. We apologise for any inconvenience this will have caused.
2004-05-18 10:15:00JBMOur primary authoritative dns server (ns0.watchfront.net and local caching server 217.169.24.209) was down for about an hour this morning. During the outage the secondary dns server (ns1.watchfront.net and local caching server 217.169.31.244) provided backup dns as expected, so if you experienced problems please check that your machines are configured to use both servers. Apologies if anyone was affected.
2004-03-08 15:00:00JBMThere was a problem within BTs network today between 11:45 and 12:40, affecting some of our ADSL customers. Everyone is now back on-line and we are chasing BT for an explanation. Apologies to those affected.
2004-02-25 23:50:00KRHThere was a routing problem affecting netblock 213.228.226.0/24 between approximately 15:45 and 23:30 today. This would have affected backup (but not main) mail and DNS and also certain customer connections. Apologies to those affected.
2003-12-11 14:20:00JBMNot our lucky day. First we had a brief (5min) router outage at 09:20 affecting all ADSL customers, then BT had a problem within their network at 10:48, affecting some ADSL customers. All appears to be back to normal - please contact us if you are still having problems. Apologies for any inconvenience.
2003-12-04 20:15:00JBMOne of our routers failed tonight at 18:25, affecting our web, email, and dns servers for about an hour. The router has been power-cycled, and we are investigating the cause of the failure. Apologies for any inconvenience.
2003-08-27 18:00:00JBMWe suffered a power outage last night between 11pm and midnight, in our data centre at Redbus. All services were restored shortly afterwards. We are awaiting an explanation as to how an un-interruptable power supply was interrupted! Apologies for any inconvenience.
2003-07-03 13:50:00JBMOne of our core routers has had an upgrade which should make it much more tolerant of the sort of attack seen yesterday, and will hopefully prevent a reoccurrence of yesterday's problems.
2003-07-02 12:50:00JBMThere have been ongoing apparent DoS attacks this morning, resulting in very high packet loss in our core routers. We are currently trying to identify the source such that we can block future attacks. Apologies for the inconvenience.
2003-05-07 18:30:00JBMWe are still awaiting an explanation from BT of yesterday's packet loss.
2003-05-06 14:30:00JBMBT have reported a Major Service Outage between 00:25 and 02:25 today, which resulted in high packet loss for all ADSL customers. We have also seen much lower levels of packet loss during the day, which may be related - we will keep you posted.
2003-04-26 19:30:00KRHDue to a stupid configuration error entirely of my own doing, there was a short outage of 'chiswick', one of our core servers this evening. This would have caused website and mail problems for some customers. Sorry for the inconvenience.
2003-04-16 10:20:00JBMADSL customers may have seen some packet loss between 08:30 and 09:00 this morning. We are investigating the cause. Please accept our apologies.
2003-04-14 17:20:00JBMWe suffered a brief outage at 17:00 today, due to problems with our transit feed. All now appears stable and we are investigating the cause. Please accept our apologies.
2003-04-06 00:00:00KRHFTP access to customer web sites restored.
2003-04-06 00:00:00KRHFTP access to customer web sites has been temporarily suspended whilst they are moved to a new server.
2003-04-02 09:40:00JBMBT finally cleared the major service outage at 23:30. Please accept our apologies for the inconvenience.
2003-04-01 20:30:00JBMBT have a major service outage, affecting all of our ADSL customers. BT anticipate clearing it by 10pm tonight. Please accept our apologies.