Postmortem01 Sept 2025 · 23:14:58 UTC

Postmortem: AMD Ryzen vHosts Network Connectivity Issues

Incident Start: 31 August 2025, 9:00 PM CEST
Incident End: 31 August 2025, 9:22 PM CEST
Duration: Approximately 22 minutes
Impact: Major outage on Ryzen vHosts, particularly RYZEN-05 and RYZEN-08

Summary

On 31 August 2025, a router misconfiguration caused a significant network outage affecting multiple Ryzen-based vHosts and services. The issue began at 9:00 PM CEST and was fully resolved by 9:22 PM CEST.

Impact

RYZEN-05 and RYZEN-08 experienced full network loss for about 15 minutes.
All other Ryzen vHosts were briefly disconnected for approximately 10 seconds during the network recovery process.
Affected services included:
- DB01 - Game-Cloud Infrastructure
- Ryzen Game-Server Nodes (EYG1)
- AMD Ryzen/High-End Root servers

All customers with services on RYZEN-05 and RYZEN-08 are eligible to receive 2 days of additional runtime credit. Please open a support ticket to request this credit.

Root Cause

A router misconfiguration combined with an automation error caused connectivity issues on RYZEN-05 and RYZEN-08. Full restoration required reapplying routing configuration across all nodes, resulting in a brief interruption to all Ryzen vHosts.

Resolution

A temporary automation fix was deployed to prevent recurrence of this specific issue.
Manual configuration changes restored full network connectivity by 9:22 PM CEST.

Preventive Actions

Implement per-node VM connectivity monitoring for faster detection and resolution.
Improve automation workflows to avoid similar misconfigurations.
Add validation checks to prevent faulty network configurations from being deployed.

We apologize for the disruption and appreciate your patience as we implement additional safeguards to strengthen our network stability.

Resolved31 Aug 2025 · 19:22:00 UTC

This incident has been resolved.

Identified31 Aug 2025 · 19:00:00 UTC

We are continuing to work on a fix for this incident.