Incident Report: Service Interruption on November 24th

On November 24th, our platform experienced an unexpected service disruption that impacted several core functionalities. We would like to explain what happened, how it affected our services, and what actions we have taken to prevent similar incidents in the future.

What Triggered the Incident?

The root cause was technical maintenance performed by our data center. During this maintenance, a portion of our servers was partially disconnected and automatically switched to power-saving mode. As the system attempted to handle regular user activity with reduced resources, CPU load increased significantly. This resulted in noticeable service degradation.

Unfortunately, this behavior could not be anticipated based on the notification we received from the data center prior to the maintenance.

Which Services Were Affected?

The incident impacted multiple services across our platform, including:

• Images Distribution
• Live HTML
• REST APIs, which also affected the Icecat Brand Cloud

How We Responded

Our team acted immediately upon detecting the degradation. Thanks to the rapid response, we managed to restore Live HTML services in a very short time. The performance of other services temporarily improved through operational optimizations, but full recovery was only possible once the data center completed its maintenance window.

Preparing for the Upcoming Data Center Maintenance

The data center has clearly announced another scheduled maintenance on December 15. This time, we are proactively prepared.

For the upcoming data center maintenance, we have prepared two stability strategies to ensure our services remain reliable and uninterrupted during this specific event.