The Attack That Never Stopped and the Sites That Stopped Noticing

Icecat continues to receive 7.4 million bot requests per hour, while our application now runs on just one-tenth of the computing power it needed at the attack’s peak.

Since October 16, Icecat websites have experienced a flood of automated crawler traffic. At its peak, we recorded up to 15 million different IP addresses over two days, many disguised as regular browsers. For an open catalog used daily by brands, retailers, and marketplaces, this presents more than a traffic management challenge. When our websites slow down, the businesses that rely on our services are affected, too. During the night of October 20-21, pressure became particularly severe, causing several hours of service disruption and failed requests.

The Night We Stayed

Between 20:00 on October 20 and 04:00 the following morning, our websites experienced significant performance issues. At the worst point, 74% of requests failed to complete. The servers responsible for generating product pages had been operating close to capacity for four consecutive days, and the additional traffic pushed them past their limits.

Teams responsible for different parts of the infrastructure worked throughout the night, sharing findings and coordinating responses in real time to address the disruption as quickly as possible.

Then PHP2Shell Landed

On the afternoon of Wednesday, October 21, the PHP project released emergency patches addressing a critical vulnerability. Within an hour of the announcement, the first scanner reached our systems.

At that point, the team had already been managing the traffic flood for five days, including a night of widespread request failures. The newly disclosed vulnerability added another security concern that required immediate assessment, while automated traffic continued.

Managing both incidents simultaneously placed additional demands on the team. Despite the ongoing traffic pressure, Icecat Brand Cloud was upgraded to the fixed release on the same day.

The flood and the PHP2Shell flaw overlapped for about a day, with the team working on both.

The Fix That Hurt the Wrong People

On October 21, we implemented emergency protection measures, including per-visitor and per-site rate limits, along with blocks targeting previously identified crawlers. These measures helped restore website availability, but they also affected some legitimate visitors.

On the morning of October 22, one day after the vulnerability disclosure, a significantly larger traffic wave arrived. Within six hours, we recorded 2.7 million IP addresses, and all page servers operated at full capacity. The emergency limits rejected 48% of requests, including some from legitimate users.

We found no evidence linking the traffic flood to the PHP2Shell vulnerability. However, the timing meant our team had to address both incidents at the same time.

Rate limits alone cannot reliably distinguish automated traffic from legitimate visitors.

The Browser Check

To improve traffic filtering, we shifted from limiting requests to identifying which requests should reach the application.

Between 15:00 and 17:00 on October 22, we enabled a browser check across all 32 public Icecat domains, alongside connection fingerprinting designed to recognize automated tools. These checks operate at the edge, before incoming requests reach the application.

This approach let us identify and filter traffic earlier, reducing the resources needed to handle unwanted requests.

Where the requests end up: before protection, with emergency limits, and now.

What Changed

The attack continues at approximately 7.4 million requests per hour. However, the main difference is how we handle this traffic.

Approximately 88% of incoming traffic is now either stopped or directed to the browser check at the edge, allowing only real browsers to continue to the application. As a result, failed requests dropped from 74% during the worst night to 0.2%.

Performance has also improved significantly. The slowest 5% of product page loads decreased from 7.6 seconds during the October 22 traffic wave to 0.27 seconds.

At the same time, application resource consumption has fallen considerably. The application now uses approximately 35 processor cores during a typical hour, compared with 348 cores at the peak. Consequently, we need only a fraction of the previous number of page servers.

Incoming traffic stays at the attack level, while the application’s computing needs drop from 348 cores to a few dozen. Times in UTC.

Before and now: failed requests, page speed and computing power used.

The new protection measures have also minimized disruption to legitimate traffic. Googlebot continues to crawl at approximately 362,000 requests per hour, maintaining the same activity level as before the protections were introduced. Only 0.5% of its requests are now rate-limited, compared with 9.9% under the emergency limits.

For legitimate visitors, the process involves a brief Icecat security check lasting approximately one second, typically appearing once every few days. Only about 1.5% of initiated checks are completed, meaning most of this traffic does not pass verification.

The Part That Is Easy to Skip

After implementing the browser checks, we continued monitoring their impact to identify and resolve unintended disruptions.

We found that visitors using certain corporate security proxies were encountering more demanding verification checks and, in some cases, rate limits.

At 10:30 on October 23, we adjusted the configuration so that these proxies would be treated like regular browsers. Following this change, the rate of limited responses for these visitors dropped from approximately 60% to zero.

We also added Icecat branding to the security check page to ensure visitors can clearly identify the source of the verification request.

Some improvements are still required. Simple scripts that do not attempt to imitate browsers can still pass the check and are currently controlled only through per-visitor limits. Addressing this remaining gap is the next step.

We will continue using the dashboards and alerts established during the incident to monitor attack traffic, browser check success rates, and server load, as the automated traffic remains ongoing.

We would also like to thank everyone involved in restoring and maintaining service availability. From the engineers who worked throughout the night to the colleagues who adjusted the protection measures during the second traffic wave and resolved the corporate proxy issue, the coordinated response helped minimize disruption for our customers.

For most customers, these efforts went largely unnoticed, reflecting the team’s work to restore normal service despite the ongoing attack.