Past Incidents

The outage was caused by the automated database maintenance scripts taking longer than usual (caused by a room with a massive amount of state events). This maintenance has now been completed.

The secondary server has been fixed and the earlier degraded service has resumed. There may be some minor interruption in the next few days when as we perform maintenance on our secondary service platform.

We believe the issue might be resolved but are continuing to monitor the situation.

A reboot has fixed the outage.

The steps we took yesterday have resolved the issue. For more detail we have written an overview here.

Note that some SNT services will have to be restored from backup. This includes DAS, Jitsi and the password reset portal for the vColo web interface. For a complete overview, see the current outages on the dashboard. We will restore those in the next few weeks.

Please contact us if you need any help.

All failing SSDs have been replaced. Additionally, we have identified an issue in the configuration of the hypervisors that may have caused the I/O issues.

Unfortunatly, the problems did come back in the last few days. The most recent flare-up has caused additional issues, for which the following still applies:

not all customer data has survived: due to compounding failures in our VM platform SSD data on some vColos (including our own) are lost. Affected customers will be contacted and have been given new VMs with any recovered data mounted. Contact us for unrepaired disk images. We recommend to activate your Disaster Recovery Plan using these new VMs.

Note that we can only identify data loss on the VM platform itself. The issues may have caused corruption in the VMs (either directly, or as a side-effect) which we are unable to identify directly. Contact us at cloud@snt.utwente.nl if you want a new disk for recovery, or if you want a snapshot before recovery is attempted.

We will watch the to see if all issues have been resolved, and will start restoring all affected services tomorrow if possible.

Most of the severe performance issues have been resolved. However, they may flare up again. At this time we suspect that the issues are caused by (somewhat) failing SSDs, which are being replaced.

Unfortunately, not all customer data has survived: due to compounding failures in our VM platform SSD data on some vColos (including our own) are lost. Affected customers will be contacted and have been given new VMs with any recovered data mounted. Contact us for unrepaired disk images. We recommend to activate your Disaster Recovery Plan using these new VMs.

The issues with our cloud platform are still unresolved. Troubleshooting will continue tomorrow.

The issues with our cloud platform are still unresolved. Troubleshooting will continue tomorrow.

We are continuing to manage the cloud platform outage. Many services should be available, though with degraded performance. We will update this incident as the situation develops.

SNT Cloud has resumed operation after synchronisation of all data. To prevent further downtime for virtual machine the upgrade is postponed until tomorrow. This will allow the servers to fully synchronise and stabilise over the night.

When a new attempt will start we will inform you via a new update.

Due to a failed unexpecte failover in our cloud platform all virtual machine had an unexpected interrupt. As an effect most of our services including Hornet experience downtime during the platform upgrade. We are investigating the issue and try to resolve this in the next 20 minutes.

After stabilizing the current running systems we will continue the platform upgrade.