Update 4:05pm – All services are now back online.  Further investigation will continue in to the cause.  Sorry for the inconvenience.

To protect the integrity of the server we have taken it offline to run some diagnostics.  We hope this will not take very long and the services should be back online shortly.  The diagnostics results will then indicate whether we need to take further action at another time.

We will update this post with further information once it is available.

Update 1:46pm - Upon investigation it appears that the fault was being caused by one of the hard disk cables.  This cable has been replaced and all services are back online.

13/03/2012 11:30am - We are looking into issues with these services again as they are currently offline.  Please bear with us and we will update this post when further information is available.

Update 11:50am - All services are back online.

Update 1:58pm - Qbic11 and mySQL5 are both back online. We are still investigating the cause of the outage and closely monitoring these services.

Please accept our apologies for any inconvenience caused.

Update 4:11pm – Mail Server 1 is now back to normal and the delivery queue has been processed.

Each of the servers below will be powered off and restarted.  The expected downtime duration will be 30 minutes which will occur between 22:00 and 00:00 on 12/10/2011 (UK Time).

The following servers are relevant to this message:

Database Server: mysql.controldns.co.uk (mySQL4)
Database Server: mysql5.controldns.co.uk (mySQL5)
Database Server: mysql5-7.controldns.co.uk (mySQL5)
Backup Mail Server: mx1.backupmailserver.co.uk
Web Server: Qbic07
Web Server: Qbic09
Web Server: Qbic11

Please accept our apologies for any inconvenience this planned maintenance causes and rest assured we will do everything we can to keep the downtime to a minimum.

Update: This maintenance was completed by 22:40 without incident.  Each server was only offline for approximately 10 minutes and everything is now running as normal.

Update 12:15pm - Webserver Qbic03 is now back online once again.   Please accept our apologies for this outage.

Update 9:55am - Mail server 1 is now responding correctly and the service is back to normal.  Please accept our apologies for this short outage of this service.

We will update this post with further information when available.

Update 1:35pm - The initial problem with a peering point that occurred earlier has led to a problem within the VSS routing cluster that services the data-centre. The data-centre network team quickly eliminated as many causes as possible. This issue was then escalated to Cisco and they are currently involved in joint investigation to try and discern the underlying problem.

As soon as we have any progress from this work we will post the information here.

Update 2:30pm – Today’s network connectivity issue should now be resolved and service should be returning to normal.

Unfortunately due to the nature of the repairs all power to our servers will be unavailable for at least four hours.  During this time all our Helm based windows hosting will be unavailable, including websites, email, databases and other services.  This will also affect the control panel and our own website.  

Our support and status websites will continue to be available as these are housed in a separate data centre, however, please do not submit tickets in relation to this maintenance during the maintenance window. 

During this outage DNS will be handled by our secondary DNS server and emails will be queued to our backup mail servers as these are also in a separate data centre.  Therefore during this time no emails will be lost.

A full breakdown of the maintenance schedule is below:-

Summary

• The data centre will be enacting procedures for a controlled power down and power restore. This will happen between the hours of 00:00 and 08:00 on Saturday 02/10/10.
• These procedures are being put into action to allow essential repairs to the main power systems. For further details on these repairs please see ‘Technical explanation of cause’.
• As part of procedures for restoration of power there will additional staff on site at the data centre. They will be checking that all services have power restored and pro-actively investigating any issues.

Sequence of events

• 00:00: We will start executing a soft shutdown on all servers  in the data centre.
• 01:00: Mains power will be systematically shut down.
• 05:00: The data centre aims to begin restoring power systematically across the site, in stages.
  Technical staff will begin pro-actively investigating and restoring service to any problematic servers.
• 08:00: The data centre aims to have all servers back online.

Technical explanation of cause

The data centre recently installed a fourth 500kva UPS into the data centre to add resilience and capacity to their power systems.  The data centre was designed with this in mind, so the appropriate connections exist on the Schneider UPS Output Panel (that feeds all servers from a common busbar) to connect this UPS. 

During the installation of the UPS, engineers discovered there was a fault with the panel which meant the additional UPS could not be connected.  To repair this fault, they need to electrically isolate the panel, as engineers cannot work on the panel while it is live for safety reasons.

The manufacturer of the panel, Schneider, is sending a team of engineers to repair the panel so it can accept the fourth 500kva UPS.

Please accept our apologies for this extended maintenance which is unfortunately out of our control.

Update 02/10/10 7:10am - The work is now complete and all servers and services are back online.  Our servers started coming online at 4:35am with the last server at 5:55am. 

All services appear to be functioning normally but if you have any issues please let us know.

We expect the upgrades to only take about 2 hours in total.   As each server is upgraded POP, SMTP and Webmail will be be unavailable on the server being upgraded.  During the upgrades our backup mail servers will store all email recieved so no email will be lost.   Once each mail server comes back online the backup mail servers will deliver all stored email to the server ready for collection.

We will keep this post updated as the upgrades are carried out to inform customers of progress.

Update 20/12/2009 22:05 - The planned upgrades are now commencing.  Please note that as each mail server is upgraded sending and receiving email, plus webmail, will be unavailable on that server.  We will post further information here once available.

Update 21/12/2009 00:08 -  The planned upgrades are now complete and all mail servers are back online.  An email will be sent out to all customers later today with news of some new email features.