Some systems are experiencing issues

Past Incidents

2023-03-19

cPanel Hornet issues

Some of our users have reported issues with Hornet. We are looking into the problem.

  • We believe the issue might be resolved but are continuing to monitor the situation.

    • 2022-11-15

    cPanel cPanel outage

    cPanel and websites hosted on Hornet were offline for some time. The exact cause of this is still under investigation.

  • A reboot has fixed the outage.

    • 2022-08-13

    Cloud Partial outage of vColo platform

    Performance issues lead to a large outage and data loss for a number of virtual machines. The core issues have been resolved, but some services will have to be restored from backup. The status page will be updated as services are restored.

    See Cloud outage of August 2022 for details.

  • The steps we took yesterday have resolved the issue. For more detail we have written an overview here.

    Note that some SNT services will have to be restored from backup. This includes DAS, Jitsi and the password reset portal for the vColo web interface. For a complete overview, see the current outages on the dashboard. We will restore those in the next few weeks.

    Please contact us if you need any help.

  • All failing SSDs have been replaced. Additionally, we have identified an issue in the configuration of the hypervisors that may have caused the I/O issues.

    Unfortunatly, the problems did come back in the last few days. The most recent flare-up has caused additional issues, for which the following still applies:

    not all customer data has survived: due to compounding failures in our VM platform SSD data on some vColos (including our own) are lost. Affected customers will be contacted and have been given new VMs with any recovered data mounted. Contact us for unrepaired disk images. We recommend to activate your Disaster Recovery Plan using these new VMs.

    Note that we can only identify data loss on the VM platform itself. The issues may have caused corruption in the VMs (either directly, or as a side-effect) which we are unable to identify directly. Contact us at cloud@snt.utwente.nl if you want a new disk for recovery, or if you want a snapshot before recovery is attempted.

    We will watch the to see if all issues have been resolved, and will start restoring all affected services tomorrow if possible.

  • Most of the severe performance issues have been resolved. However, they may flare up again. At this time we suspect that the issues are caused by (somewhat) failing SSDs, which are being replaced.

    Unfortunately, not all customer data has survived: due to compounding failures in our VM platform SSD data on some vColos (including our own) are lost. Affected customers will be contacted and have been given new VMs with any recovered data mounted. Contact us for unrepaired disk images. We recommend to activate your Disaster Recovery Plan using these new VMs.

  • The issues with our cloud platform are still unresolved. Troubleshooting will continue tomorrow.

  • The issues with our cloud platform are still unresolved. Troubleshooting will continue tomorrow.

  • We are continuing to manage the cloud platform outage. Many services should be available, though with degraded performance. We will update this incident as the situation develops.

    • Cloud Service performance issues

    The planned maintenance on the cloud platform is causing performance issues. Most services are affected. We are monitoring the situation, and will update this incident as the situation develops.

  • We're marking this incident as fixed. Updates will be provided here.

    • 2022-08-09

    Cloud vColo Outage

    We are upgrading our vColo platform to a new version. Downtime and slow responding virtual machine might occur during this time.

    We expect this to take around 2 hours

  • SNT Cloud has resumed operation after synchronisation of all data. To prevent further downtime for virtual machine the upgrade is postponed until tomorrow. This will allow the servers to fully synchronise and stabilise over the night.

    When a new attempt will start we will inform you via a new update.

  • Due to a failed unexpecte failover in our cloud platform all virtual machine had an unexpected interrupt. As an effect most of our services including Hornet experience downtime during the platform upgrade. We are investigating the issue and try to resolve this in the next 20 minutes.

    After stabilizing the current running systems we will continue the platform upgrade.

    • 2022-08-04

    Cloud vColo Outage

    We are upgrading our vColo platform to a new version. Downtime and slow responding virtual machine might occur during this time.

    2022-07-16

    Cloud vColo Outage

    We are investigating a vColo platform outage.

    2022-07-15

    Cloud vColo outage

    Fallout from the maintenance from yesterday caused another outage. Service should be coming back online now.

    2022-07-14

    Cloud vColo outage

    Our regular maintenance brought down some virtual machines. We are working on a solution.

    2022-07-04

    Cloud Cloud Outage

    SNT cloud was offline because of network issues. The exact cause of the issues is unknown.

    2022-07-03

    DDOS attack

    Unfortunately, our systems were hit by a DDOS attack. Service will resume once the attack has been mitigated.

    2022-06-25

    Cloud DDOS

    Unfortunately, our systems were hit by a DDOS attack. Service will resume once the attack has been mitigated.

    2022-06-24

    Cloud Cloud outage

    SNT cloud was offline because of network issues. The exact cause of the issues is unknown.

    Cloud DDOS

    Unfortunately, our systems were hit by a DDOS attack. Service will resume once the attack has been mitigated.

    2022-04-30

    cPanel Hornet MySQL outage

    The earlier cloud issues have impacted MySQL availability.

  • The MySQL database was restored from a backup made yesterday before the first outage. Unfortunately, this means that MySQL data between 2022-04-29 12:00 and 2022-04-29 20:00 has been lost.

    • Cloud Cloud outage

    The SNT cloud platform is currently down. This affects all services hosted by SNT.

  • Everything seems operational once again.

  • Hornet seems to be impacted again by this cloud outage and isn't coming directly back online yet, unlike most other services which seem to be fully operational once again.

  • Everything is being brought back online

    • 2022-04-29

    cPanel Hornet MySQL outage

    The earlier cloud issues have impacted MySQL availability.

  • The MySQL database was restored from a backup made just before the outage.

    • Cloud Cloud outage

    The SNT cloud platform is currently down. This affects all services hosted by SNT.

  • Cloud service has been restored.

    • 2022-04-20

    Admin interface Hornet synchronisation issues

    We are currently experiencing issues with configuration synchronisation, resulting in association changes not being updated correctly. We are currently investigating the root cause.

  • No further issues have been detected and we will take adequate measures to ensure this issue won't recur in the future.

  • The root cause has been identified and a fix has been applied. We are looking for a permanent solution.