Previous incidents

We experienced a brief outage in our production environment today due to the load balancer failover.

We are actively working to improve the redundancy and stability of our load balancers.

We experienced a brief outage in our production environment today due to maintenance work by our infrastructure provider on their load balancers.

We are actively working to improve our system resiliency to mitigate potential outages in the future.

In order to try and mitigate problems with intermittent downtimes in the range of a few minutes per week, we are upscaling a LoadBalancer.

Maintenance was started at 10:10 PM and finished at 10:41 PM.

All systems are back up and running.

Thursday, Jun 27, 2024 - Incident post mortem

By the Mercury Infrastructure Team

Today we experienced an outage in our Kubernetes infrastructure. We’re providing a preliminary incident report that details the nature of the outage and our response.
The following is the incident report for the Kubernetes outage that occurred on Jun 27, 2024. We understand this issue has severely impacted our users and apologize sincerely to everyone who was affected.

Issue Summary

From...