Some systems are experiencing issues

Stickied Incidents

2024-06-07

Backup Backup outage

Multiple consecutive disk failure in the backup resulted in major outage in the backup service

We're investigating a proper solution.

Past Incidents

2025-06-04

GitLab runners GitLab CI runners unavailable

During maintenance an issue occurred with the GitLab CI Runner server.

Cloud Cloud node unresponsive

One of the cloud nodes has become unresponsive to commands. We're trying to resolve this issue with as little downtime as possible.

  • The cloud node is responsive again, but will require an update to be fully operational. Customers with a VM on ‘kletsmajoor’ are asked to fully shut down their VM and start it again at their earliest convenience.

    • 2025-03-17

    Telegram bridge Matrix Telegram bridge

    The Matrix Telegram bridge is experiencing a major outage.

    2025-02-20

    cPanel cPanel outage

    At this time the website and mail component of Hornet (cPanel) is experiencing a major outage.

  • cPanel has been restarted and has resumed its normal operation.

    • 2025-02-09

    Homeserver Matrix

    The Matrix homeserver is experiencing issues loading media.

  • The issues with the media repository have been fixed

    • 2025-01-26

    Telegram bridge Telegram bridge maintenance

    The Telegram bridge's homeserver database is being moved to different storage. This is expected to take up to two hours.

  • The maintenance has been completed.

    • 2024-12-29

    IRC bridges Matrix-IRCnet outage

    The bridge stopped relaying messages around 21:45. The bridge was restarted in order to resolve the problem. It may take a while before all users have rejoined every channel.

  • The bridge has stopped relaying messages again, a temporary fix has been applied but further investigation will be conducted. It may take a while before all users have rejoined every channel.

    • 2024-10-30

    Cloud Cloud platform outage

    An issue with the cloud platform is causing outages on vColo machines and internal services.

  • An issue with the cloud platform is causing outages on vColo machines and internal services.

    • 2024-06-25

    cPanel Wordpress update failures

    An automatic Wordpress update has caused a couple of of sites hosted on cPanel to break. While this issue is not related to Hornet, we post this here as notification.

    For more information, see https://core.trac.wordpress.org/ticket/61489. If your site is affected, you can fix it yourself by replacing wp-includes/default-filters.php with the correct one from the Github mirror (note the version!): https://github.com/WordPress/wordpress-develop/blob/6.5.5/src/wp-includes/default-filters.php. You can use the wordpress CLI tool to verify the file checksums in your installation.

  • This issue should no longer occur.

  • It appears that an automated security patch applied by Patchman caused this issue, which is related to the use of an older Wordpress version. While they are working on a fix, we recommend all Wordpress users to upgrade their site to the latest available version.

    • 2024-06-07

    Backup Backup outage

    Multiple consecutive disk failure in the backup resulted in major outage in the backup service

    We're investigating a proper solution.