The Night We Were Ready to Switch Icecat Off

A 9.2 severity PHP flaw went public on Wednesday; the first scanner reached us the same afternoon, and Icecat stayed online throughout.

On 21 October, the PHP project shipped emergency releases for two critical flaws in the PHP engine, together known as PHP2Shell. In practical terms, an attacker with no account and no prior access could send one crafted request to a vulnerable server and, in the worst case, take control of it. PHP runs a large share of the web, so automated scanners started looking for exposed sites within hours, and public exploit code is now circulating. For a catalog that brands, retailers and marketplaces pull product data from every day, a successful attack would have been more than a technical incident: it would have put the data, and the trust built around it over twenty years, at risk.

The decision we made first

When a flaw like this lands, the first question is not how to patch it. It is whether the sites can keep running while you do, and for an open catalog so many businesses rely on, that is no small question.

We were ready to switch the websites off if that was what it took. We decided before we knew whether we would need it, because a plan that only exists once you are in trouble isn’t a plan, and because the alternative was to wait and find out which way the exploit would go.

Then the traffic changed

While we were still mapping our exposure, traffic from Russia rose to roughly ten times its usual level. That is the sort of number that makes a room go quiet.

A critical vulnerability and a sudden traffic surge on the same evening is a coincidence, and nobody on the team believed in it for a second. We couldn’t prove the two were connected, and we couldn’t afford to treat them as unconnected, so we watched both at once and assumed the worst until the data said otherwise.

The second thing we noticed was that much of this traffic looked borrowed. It arrived through Russian infrastructure, but it was being used as a proxy, which says more about whoever was sending it than about anyone behind those addresses. At ten times the volume, at least it was easy to spot. Someone clearly wanted us to feel popular.

Who was in the room

What made the evening manageable was not a clever tool. The people looking at the problem had spent years inside this platform and knew which service talks to which, where the edge layers sit, and what normal looks like on an ordinary Tuesday.

That is what lets you tell a scanner from a symptom in minutes instead of days. Nobody had to ask who owns Icecat Brand Cloud, or how icecat.biz is deployed, or where the logs live, because the answers were already in the room. Experience on a system doesn’t show up on a quiet day, and on a night like this, it is most of what you have.

The long hours

So the team stayed late, and then later, worked through the exposure system by system. People from different parts of the stack handed over what they found as they found it, instead of waiting for a tidy version, which is why the picture came together so quickly.

Icecat Brand Cloud was upgraded to the fixed release on the day the patch was published. icecat.biz is protected by the vendor-recommended interim mitigation, and its full upgrade is scheduled for Monday, after the required code changes and a complete regression run.

The part that is easy to skip

Then came checking, the part most people rush. We went through logs from every edge layer and the runtime metrics from every service, and we followed every request from every scanner that showed up, under any name and any address. It is slow, unglamorous work, and it is done properly only by people who know what they would expect to see if something had gone wrong.

About ten external scanners probed us, starting on Wednesday afternoon. None of them got through. We found no worker crashes, no memory exhaustion, no container restarts, and no unusual error spikes, which are the signs this exploit leaves behind. For the next two weeks, we kept watching those same signals because the scanning wouldn’t stop.

We never had to use the off switch.

It wasn’t one person’s night; it was a shared one, and the difference shows in the outcome: several people looked at the same problem from different layers, each making the others’ picture more complete.

Thank you to the people who made this a non-event for everyone outside the building. To the engineers who stayed past midnight, to the ones who handed over half-finished findings without waiting to look tidy, and to the colleague who wrote it all up so the rest of us could see what had happened and, just as important, what had not. Our customers will never notice that night, and that is the clearest sign of what this team did.

So here is the question I would put to any team running a catalog, a shop or a platform: when a critical flaw lands, which is ready first, the patch or the decision about when to switch things off?

Not the comfortable answer. The honest one.