Some systems are experiencing issues

Past Incidents

2023-05-20

Homeserver Outage Matrix

It seems to no longer be possible to send and receive messages on Matrix.

  • The outage was caused by the automated database maintenance scripts taking longer than usual (caused by a room with a massive amount of state events). This maintenance has now been completed.

  • 2023-04-13

    Mail relay Relay outage

    The mail relay is currently unavailable due to failure of our secondary hypervisor. We expect to resolve this situation tomorrow.

  • The secondary server has been fixed and the earlier degraded service has resumed. There may be some minor interruption in the next few days when as we perform maintenance on our secondary service platform.

  • 2023-03-19

    cPanel Hornet issues

    Some of our users have reported issues with Hornet. We are looking into the problem.

  • We believe the issue might be resolved but are continuing to monitor the situation.

  • 2022-11-15

    cPanel cPanel outage

    cPanel and websites hosted on Hornet were offline for some time. The exact cause of this is still under investigation.

  • A reboot has fixed the outage.

  • 2022-08-13

    Cloud Partial outage of vColo platform

    Performance issues lead to a large outage and data loss for a number of virtual machines. The core issues have been resolved, but some services will have to be restored from backup. The status page will be updated as services are restored.

    See Cloud outage of August 2022 for details.

  • The steps we took yesterday have resolved the issue. For more detail we have written an overview here.

    Note that some SNT services will have to be restored from backup. This includes DAS, Jitsi and the password reset portal for the vColo web interface. For a complete overview, see the current outages on the dashboard. We will restore those in the next few weeks.

    Please contact us if you need any help.

  • All failing SSDs have been replaced. Additionally, we have identified an issue in the configuration of the hypervisors that may have caused the I/O issues.

    Unfortunatly, the problems did come back in the last few days. The most recent flare-up has caused additional issues, for which the following still applies:

    not all customer data has survived: due to compounding failures in our VM platform SSD data on some vColos (including our own) are lost. Affected customers will be contacted and have been given new VMs with any recovered data mounted. Contact us for unrepaired disk images. We recommend to activate your Disaster Recovery Plan using these new VMs.

    Note that we can only identify data loss on the VM platform itself. The issues may have caused corruption in the VMs (either directly, or as a side-effect) which we are unable to identify directly. Contact us at cloud@snt.utwente.nl if you want a new disk for recovery, or if you want a snapshot before recovery is attempted.

    We will watch the to see if all issues have been resolved, and will start restoring all affected services tomorrow if possible.

  • Most of the severe performance issues have been resolved. However, they may flare up again. At this time we suspect that the issues are caused by (somewhat) failing SSDs, which are being replaced.

    Unfortunately, not all customer data has survived: due to compounding failures in our VM platform SSD data on some vColos (including our own) are lost. Affected customers will be contacted and have been given new VMs with any recovered data mounted. Contact us for unrepaired disk images. We recommend to activate your Disaster Recovery Plan using these new VMs.

  • The issues with our cloud platform are still unresolved. Troubleshooting will continue tomorrow.

  • The issues with our cloud platform are still unresolved. Troubleshooting will continue tomorrow.

  • We are continuing to manage the cloud platform outage. Many services should be available, though with degraded performance. We will update this incident as the situation develops.

  • Cloud Service performance issues

    The planned maintenance on the cloud platform is causing performance issues. Most services are affected. We are monitoring the situation, and will update this incident as the situation develops.

  • We're marking this incident as fixed. Updates will be provided here.

  • 2022-08-09

    Cloud vColo Outage

    We are upgrading our vColo platform to a new version. Downtime and slow responding virtual machine might occur during this time.

    We expect this to take around 2 hours

  • SNT Cloud has resumed operation after synchronisation of all data. To prevent further downtime for virtual machine the upgrade is postponed until tomorrow. This will allow the servers to fully synchronise and stabilise over the night.

    When a new attempt will start we will inform you via a new update.

  • Due to a failed unexpecte failover in our cloud platform all virtual machine had an unexpected interrupt. As an effect most of our services including Hornet experience downtime during the platform upgrade. We are investigating the issue and try to resolve this in the next 20 minutes.

    After stabilizing the current running systems we will continue the platform upgrade.

  • 2022-08-04

    Cloud vColo Outage

    We are upgrading our vColo platform to a new version. Downtime and slow responding virtual machine might occur during this time.

    2022-07-16

    Cloud vColo Outage

    We are investigating a vColo platform outage.

    2022-07-15

    Cloud vColo outage

    Fallout from the maintenance from yesterday caused another outage. Service should be coming back online now.

    2022-07-14

    Cloud vColo outage

    Our regular maintenance brought down some virtual machines. We are working on a solution.

    2022-07-04

    Cloud Cloud Outage

    SNT cloud was offline because of network issues. The exact cause of the issues is unknown.

    2022-07-03

    DDOS attack

    Unfortunately, our systems were hit by a DDOS attack. Service will resume once the attack has been mitigated.

    2022-06-25

    Cloud DDOS

    Unfortunately, our systems were hit by a DDOS attack. Service will resume once the attack has been mitigated.

    2022-06-24

    Cloud Cloud outage

    SNT cloud was offline because of network issues. The exact cause of the issues is unknown.

    Cloud DDOS

    Unfortunately, our systems were hit by a DDOS attack. Service will resume once the attack has been mitigated.