Multiple Servers at EYG1 location unreachable

Timeline

1. Postmortem23 Apr 2026 · 08:19:55 UTC

   On the morning of April 23, a brief utility power failure occurred at the EYG1 data center. One of the Uninterruptible Power Supply (UPS) units failed to engage, causing a total loss of power on one of the power feeds. Servers equipped with only a single Power Supply Unit (PSU) connected to this feed experienced an immediate powerloss. While most systems recovered automatically, the host RYZEN-07-VHOST-EYG1 suffered a secondary hardware issue that required manual intervention.

   Root Cause Analysis

   1. Power Feed Failure: An external power dip was not mitigated by a specific UPS system, leading to a localized blackout on one power feed.
   2. Lack of Redundancy: Impacted Ryzen-based servers utilized non-redundant power configurations. Consequently, the loss of a single feed resulted in an immediate power loss for these machines.
   3. Hardware Damage: The abrupt shutdown triggered a hardware-level fault on host RYZEN-07-VHOST-EYG1, preventing it from rebooting autonomously.

   Incident Timeline

   • 09:33 AM: Monitoring alerts triggered for multiple host systems at EYG1. Investigation initiated.
   • 09:41 AM: Issue identified as a power feed failure. Most affected servers were already successfully restarted and entered monitoring status.
   • 09:56 AM: All systems recovered except for RYZEN-07-VHOST-EYG1. An on-site technician was dispatched to address the hardware failure.
   • 10:11 AM: Hardware repair completed. RYZEN-07-VHOST-EYG1 initiated startup, and all hosted VPS returned to online status shortly thereafter.
2. Resolved23 Apr 2026 · 08:11:27 UTC

   RYZEN-07-VHOST-EYG1 is now starting up. All VPS on it should be online within the next 5 minutes.

3. Identified23 Apr 2026 · 07:56:39 UTC

   RYZEN-07-VHOST-EYG1 is currently still unreachable. A technitian has already been informed about this incident.

4. Monitoring23 Apr 2026 · 07:41:00 UTC

   All affected servers have been started. It seems to be a power-related issue on one of the power feeds, which affected half the servers with non-redundant PSUs. Therefore only AMD Ryzen VPS in Netherlands were partially affected.

   If your server is still unreachable, please create a support ticket so we can address the issue.

5. Investigating23 Apr 2026 · 07:33:00 UTC

   Multiple of our Hostsystems at the EYG1 location are currently unreachable. We are investigating the issue.