Some systems are experiencing issues

Stickied Incidents

2024-06-07

Backup Backup outage

Multiple consecutive disk failure in the backup resulted in major outage in the backup service

We're investigating a proper solution.

Past Incidents

2022-04-30

cPanel Hornet MySQL outage

The earlier cloud issues have impacted MySQL availability.

  • The MySQL database was restored from a backup made yesterday before the first outage. Unfortunately, this means that MySQL data between 2022-04-29 12:00 and 2022-04-29 20:00 has been lost.

    • Cloud Cloud outage

    The SNT cloud platform is currently down. This affects all services hosted by SNT.

  • Everything seems operational once again.

  • Hornet seems to be impacted again by this cloud outage and isn't coming directly back online yet, unlike most other services which seem to be fully operational once again.

  • Everything is being brought back online

    • 2022-04-29

    cPanel Hornet MySQL outage

    The earlier cloud issues have impacted MySQL availability.

  • The MySQL database was restored from a backup made just before the outage.

    • Cloud Cloud outage

    The SNT cloud platform is currently down. This affects all services hosted by SNT.

  • Cloud service has been restored.

    • 2022-04-20

    Admin interface Hornet synchronisation issues

    We are currently experiencing issues with configuration synchronisation, resulting in association changes not being updated correctly. We are currently investigating the root cause.

  • No further issues have been detected and we will take adequate measures to ensure this issue won't recur in the future.

  • The root cause has been identified and a fix has been applied. We are looking for a permanent solution.

    • 2022-01-21

    Telegram bridge Telegram bridge upgrade issues

    While upgrading the Telegram bridge to the latest version, we discovered that it does not cooperate well with our version of PostgreSQL.

  • After tailoring the bridge's database migrations to our PostgreSQL version, the bridge is starting up again.

    • 2021-12-24

    Cloud SNT cloud platform outage

    Another issue with our virtualisation platform caused most of the services running on this platform to go offline.

  • The issue has been resolved

    • 2021-12-23

    Cloud SNT's cloud platform outage

    SNT's cloud platform had a major outage after switch maintenance.

  • The issue has been resolved.

    • 2021-11-09

    Major outage effecting (nearly) all systems of SNT

    We are aware that most of our services are currently unavailable. This outage is under investigation and we will update this incident report once more information is available.

  • All services should be operational once again. The Matrix bridges might still have some catching up to do.

  • LISA has mitigated the root cause and we are resuming operations of all our services.

  • After everything was back in working order, the systems have been hit just now once again.

  • All services should be operational once again. The Matrix bridges might still have some catching up to do.

    • 2021-11-06

    Major outage effecting (nearly) all systems of SNT

    We are aware that all our services are currently unavailable. This outage is under investigation and we will update this incident report once more information is available.

  • All services should be operational once again. The Matrix bridges might still have some catching up to do.

    • 2021-11-04

    Major outage effecting (nearly) all systems of SNT

    We are aware that all our services are currently unavailable. This outage is under investigation and we will update this incident report once more information is available.

  • Everything has been restored.

  • All services should be operational again, except the Matrix bridges.

  • Most services are operational again, others are being started.

  • Disruptions in the network are the root cause. These disruptions have been resolved by LISA. We are now investigating our systems to bring all services back online.