How to Optimize Your Icecat Connection? My advice.

By far, most users have an efficient Icecat connection, but every month some heavy users spew out millions or even tens of millions of product data-sheet requests. Our fair use policy allows for a standard service level per Icecat user of up to one million product data-sheet (PDS) requests per month. Typically, more than sufficient for the average user, focusing on just a few Verticals and just a few countries. Because per month, less than one million product data-sheets have meaningful changes.

So what goes wrong then, and what are the five most common mistakes regarding optimizing the Icecat connection?

1. Don’t download every day the whole Icecat database

The most common mistake is to download the whole Icecat database every day, or at least again and again all PDSs available to a user. For example, 30 * 100K PDS requests/day = 3M PDS request/month!

This is a total waste of CPU power, data transmission capacity, and application time. Both on the Icecat server-side and the client-side. It is really not necessary, as Icecat timestamps each changed PDS. And, every change in taxonomy files is timestamped as well.

What are quick fixes? Do only a full download once a week or once a month, and rely on the daily indexes that only push PDSs that have changed. Further, look at the timestamps in taxonomy files to dynamically apply changed taxonomy structures to your PIM.

2. Do not over-proxy Icecat LIVE

Icecat LIVE is a real-time service for providing PDSs which are live embedded in the client’s website. The number of PDS requests equals then the number of consumer visits to a product page. Sometimes proxy servers are used to download and store an Icecat LIVE PDS on the client-side. Depending on the update frequency, the number of PDS requests can skyrocket.

What are quick fixes? Reduce the update frequency of the proxy server. Or better: only apply smart updates, taking into account changed PDSs only based on their timestamps.

3. Do not download irrelevant PDSs

A classic mistake is that a user downloads any accessible PDS, whether through Open Icecat indexes or Full Icecat indexes. For example, if there are 10M indexed products in Full Icecat, and these are downloaded once a month, it leads to 10M PDS requests/month. In reality, a typical reseller only has 100K to 1M of products in its catalog. So, 9 to 9.9M PDS requests are an utter waste. Why download a PDS for Nike shoes if you only sell electronics?

The solution is straightforward. Only download PDSs that match a user’s product catalog. If the index matching is too much effort, we have automated that. We call this the Personal Index File and Personal Catalog File. Please read the manual, and set it up via your user account.

Of course, we have recently reduced this issue somewhat by applying a hard limit on Full Icecat Vertical access outside subscriptions.

4. Do multi-locale optimization

In case that users need product data for multiple locales or languages, there might be a multiplication of the number of PDS requests. For example, 1M monthly PDS requests/locale * 10 locales = 10M monthly PDS requests.

Not for every locale, Icecat has unique marketing texts or other local materials. In that case, it’s mainly the translated taxonomy structures that are relevant.

Therefore, a smart optimization is to look into the taxonomy files for taxonomy updates based on timestamp changes. And, further, only download PDSs for the locales that the respective brand actively supports. In the future, Icecat will investigate to create a single real-time XML or JSON product file that covers the locales in the user’s subscription.

5. Act on “errors” Icecat connection

Another mistake is to continuously try to download a PDS if the PDS request produced an error. All the errors we return have a number and a descriptive message. It can be the case that a PDS has been merged with another one, that a user is not authorized to access a certain PDS.

Therefore, better limit the number of tries, store the reason for the error and act on these reasons if relevant for the user.