How to Optimize Your Icecat Connection? My advice.


By far, most users have an efficient Icecat connection, but every month some heavy users spew out millions or even tens of millions of product data-sheet requests. Our fair use policy allows for a standard service level per Icecat user of up to one million product data-sheet (PDS) requests per month. Typically, more than sufficient for the average user, focusing on just a few Verticals and just a few countries. Because per month, less than one million product data-sheets have meaningful changes.

So what goes wrong then, and what are the five most common mistakes regarding optimizing the Icecat connection?

1. Don’t download every day the whole Icecat database

The most common mistake is to download the whole Icecat database every day, or at least again and again all PDSs available to a user. For example, 30 * 100K PDS requests/day = 3M PDS request/month!

This is a total waste of CPU power, data transmission capacity, and application time. Both on the Icecat server-side and the client-side. It is really not necessary, as Icecat timestamps each changed PDS. And, every change in taxonomy files is timestamped as well.

What are quick fixes? Do only a full download once a week or once a month, and rely on the daily indexes that only push PDSs that have changed. Further, look at the timestamps in taxonomy files to dynamically apply changed taxonomy structures to your PIM.

2. Do not over-proxy Icecat LIVE

Icecat LIVE is a real-time service for providing PDSs which are live embedded in the client’s website. The number of PDS requests equals then the number of consumer visits to a product page. Sometimes proxy servers are used to download and store an Icecat LIVE PDS on the client-side. Depending on the update frequency, the number of PDS requests can skyrocket.

What are quick fixes? Reduce the update frequency of the proxy server. Or better: only apply smart updates, taking into account changed PDSs only based on their timestamps.

3. Do not download irrelevant PDSs

A classic mistake is that a user downloads any accessible PDS, whether through Open Icecat indexes or Full Icecat indexes. For example, if there are 10M indexed products in Full Icecat, and these are downloaded once a month, it leads to 10M PDS requests/month. In reality, a typical reseller only has 100K to 1M of products in its catalog. So, 9 to 9.9M PDS requests are an utter waste. Why download a PDS for Nike shoes if you only sell electronics?

The solution is straightforward. Only download PDSs that match a user’s product catalog. If the index matching is too much effort, we have automated that. We call this the Personal Index File and Personal Catalog File. Please read the manual, and set it up via your user account.

Of course, we have recently reduced this issue somewhat by applying a hard limit on Full Icecat Vertical access outside subscriptions.

4. Do multi-locale optimization

In case that users need product data for multiple locales or languages, there might be a multiplication of the number of PDS requests. For example, 1M monthly PDS requests/locale * 10 locales = 10M monthly PDS requests.

Not for every locale, Icecat has unique marketing texts or other local materials. In that case, it’s mainly the translated taxonomy structures that are relevant.

Therefore, a smart optimization is to look into the taxonomy files for taxonomy updates based on timestamp changes. And, further, only download PDSs for the locales that the respective brand actively supports. In the future, Icecat will investigate to create a single real-time XML or JSON product file that covers the locales in the user’s subscription.

5. Act on “errors” Icecat connection

Another mistake is to continuously try to download a PDS if the PDS request produced an error. All the errors we return have a number and a descriptive message. It can be the case that a PDS has been merged with another one, that a user is not authorized to access a certain PDS.

Therefore, better limit the number of tries, store the reason for the error and act on these reasons if relevant for the user.

Founder and CEO of Icecat NV. Investor. Ph.D.

Leave a Reply

Your email address will not be published. Required fields are marked *

Icecat xml

Open Catalog Interface (OCI): Manual for Open Icecat XML and Full Icecat XML

This document describes the Icecat XML method of Icecat's Open Catalog Inte...
 November 3, 2019

Manual for Icecat Live: Real-Time Product Data in Your App

Icecat Live is a (free) service that enables you to insert real-time produc...
 June 10, 2022
Manual for Icecat CSV Interface

Manual for Icecat CSV Interface

This document describes the manual for Icecat CSV interface (Comma-Separate...
 September 28, 2016
 October 4, 2018

How to Create a Button that Opens Video in a Modal Window

Recently, our Icecat Live JavaScript interface was updated with two new fun...
 November 3, 2021
Addons plugins

Icecat Add-Ons Overview. NEW: Red Technology

Icecat has a huge list of integration partners, making it easy for clients ...
 October 27, 2023

Manual for Open Icecat JSON Product Requests

JSON (JavaScript Object Notation) is an increasingly popular means of trans...
 September 17, 2018
 January 20, 2020
New Standard video thumbnail

Autheos video acquisition completed

July 21, Icecat and Autheos jointly a...
 September 7, 2021

Manual Personalized Interface File and Catalog from Icecat

With Icecat, you can generate personalized or customized CSV or Excel files...
 May 3, 2022