All failing SSDs have been replaced. Additionally, we have identified an issue in the configuration of the hypervisors that may have caused the I/O issues.
Unfortunatly, the problems did come back in the last few days. The most recent flare-up has caused additional issues, for which the following still applies:
not all customer data has survived: due to compounding failures in our VM platform SSD data on some vColos (including our own) are lost. Affected customers will be contacted and have been given new VMs with any recovered data mounted. Contact us for unrepaired disk images. We recommend to activate your Disaster Recovery Plan using these new VMs.
Note that we can only identify data loss on the VM platform itself. The issues may have caused corruption in the VMs (either directly, or as a side-effect) which we are unable to identify directly. Contact us at cloud@snt.utwente.nl if you want a new disk for recovery, or if you want a snapshot before recovery is attempted.
We will watch the to see if all issues have been resolved, and will start restoring all affected services tomorrow if possible.
