Previous incidents
All services unavailable
Resolved Mar 26 at 09:51am CET
From 9:50 AM CET to 11:27 AM CET our Mercury service experienced a major incident due to failing health checks on Hetzner Cloud load balancers. The incident response team began mitigation at 10:02 AM CET.
Timeline of Events:
9:50 AM: First alert received through our incident management platform.
10:02 AM: Failing load balancer identified; mitigation efforts began.
10:14 AM: A new load balancer was created to reroute traffic, while work continued to restore the original load balancer.
10:...
Mercury and other application down for 1h37m due to cloud provider incident
Resolved Mar 19 at 04:16pm CET
The application became unavailable at 1:59 PM, after LoadBalancing failed due to an incidient that occurred at our cloud provider, Hetzner, which led to config changes negatively affecting LoadBalancer performance. From our investigation, we deduced that an autoscaling event in the cluster that happened at 1:59 PM led to a config change propagated to the LoadBalancer which then led to the failure.
https://status.hetzner.com/incident/b8246859-8fdd-4c44-b35c-d3c9a6cad3a7
We then redeployed th...