This document describes the Icecat XML method of Icecat’s Open Catalog Interface (OCI): a set of standards and file formats used in the Icecat XML repository. In summary, it gives the basics of how to use the Icecat repository or how to publish data to Icecat conforming this OCI.
If you want to integrate product data without frames, conforming your CSS, then use Icecat Live. If you want to have full control over the data, then make use of the XML (read further below), JSON, or CSV download methods. See the full overview of methods and manuals.
Further, check out the Add-On page for popular platforms already integrating Icecat data, which we regularly update.
Are you interested in adding product reviews? Then check out our reviews manual.
Brand owners that look into pushing data into Icecat, please, look into our Push-API or for manual data-entry Icecat’s Free Vendor Central.
After a rip of our general site, an IP address may be temporarily blocked. A rip is not acceptable as it may hurt site performance and can be a copyright infringement.
The JSON interface may be accessed by all Icecat Data (XML, CSV or JSON) users, with at least an Open Icecat account. Open Icecat users can access the data of the sponsoring brands. Full Icecat users can access the data of all brands in the Icecat database, and get a free Vendor Central (PIM) service included.
However attractive the free Open Icecat database is, the Full-Icecat database still contains many, many more brands: 10,000s. If you want specific brands to be included into free Open Icecat, please, recommend Icecat to your manufacturer contacts.
Read more about the main advantages of Full Icecat over free Open Icecat and see the Icecat Susbcription Plans.
In case you want to upgrade to Full Icecat, please contact us via the contact form, or click on the “request Full Icecat access” link after the User Login.
We require any user of Open Icecat product data – in a free Open Icecat subscription or as part of the Full Icecat data – to respect the Open Icecat Fair Use Policy.
Explicitly relevant for website development is:
Apart from XML, Icecat is compliant with an increasing number of international standards. See here a quick overview of such supported standards.
To decide to make use of Full Icecat or Open Icecat it may be important to analyze the coverage of your portfolio or the portfolio of your supplier. You can always ask us to give you an overview of the coverage of (stock) portfolio of your suppliers (i.e., distributors) at any given moment. We strive to cover at least 90% of the stock portfolio (stock > 1) of mainstream distributors and work together with partners to monitor this coverage daily.
If you want to analyze coverage, you can do two things:
1) Register and upload your catalog to Icecat via the Login. In the match analysis, you get the match percentages back for Full Icecat and Open Icecat. And you can get a list of unrecognized products. An advantage is that you can also optionally display your offers -using this method- on the Icecat.biz price comparison sites. NOTE: if you follow this method, Icecat also provides you automatically with your private Icecat index.
2) If you do in-depth coverage analyses, please use the following matching file, which is a plain dump of everything in our catalog: https://data.Icecat.biz/prodid/prodid_d.txt.gz (a very large file with all known products in the Icecat database, described and standardized or not in gzipped archive). NOTE: Market presence=Y in the matching file indicates that we notice that the product is currently present in at least one distributor’s catalog. Quality=Icecat means that Icecat editors have standardized the data-sheet. Quality=SUPPLIER means that the data-sheet comes from a manufacturer’s CMS (or a brand editor created a product data-sheet manually via Icecat PIM). The respective data-sheet might be queued for standardization. Quality=NOEDITOR means that our editors did not create a data-sheet yet, nor did we auto-import manufacturer data; so, it is only the raw data as we received it from channel partners.
TIP: discuss with the Icecat team a service level for your portfolio.
The Full Icecat XML repository is a set of product data files and an index file, which includes all pointers to individual product files and gives some basic information as timestamp updated, content quality level, etc. The Icecat XML repository contains product information for products of all brands, and categories that subscribers want us to support. The current content can be best reviewed on the public Icecat website
The Open Icecat repository is the part of the Icecat XML repository that is sponsored by manufacturers and is provided for free to the channel. Open Icecat is distributed under the Open Content License Agreement. Which brands are included in Open Icecat? The most recent list is here. Open Icecat is also useful during development and testing.
To obtain access to the repository, you will need a login/password, provided to you by Icecat. If you don’t have these, please register online for free Open Icecat. NOTE: During the registration you need to choose for the Live (JS) or Data (xml, json, csv) version. If you want to make use of XML, make sure that you choose the Open Icecat Data subscription. You can always change this afterwards or create a second subscription for the Live version method. For Full Icecat access, a separate contract is needed. You can request this contract here.
Take care that you provide us with the right IP addresses of your content servers, as the access is secure. You can test the login/password combination by entering it by hand in the BASIC HTTP authorization pop-up screen, which appears automatically when the directory URL is entered. In this case, we need to have added the IP address of your workplace as well to your profile. If you don’t know this IP address, just type in your browser https://www.whatismyip.com/ or https://www.getip.com/ and fill it in the IP addresses field after Login via the Icecat website (enter multiple IPs space separated).
Of course, for professional use, server access needs to be set up. In case of little experience in this field, we advise you to make use of the Live version sor ask us for assistance.
For smaller catalogs and for testing XML with small data-sets, it may be handy to make use of our realtime interface (URL2XML) for retrieving Product XML from the (Open) Icecat repositories.
The general format of the call is:
Requesting a product XML by Brand and Product code:
https://data.Icecat.biz/xml_s3/xml_server3.cgi?prod_id=<prod_id>;vendor=<vendor_name>;lang=<lang>;output=<output>
or
Requesting a product XML by GTIN (EAN or UPC):
https://data.Icecat.biz/xml_s3/xml_server3.cgi?ean_upc=<EAN or UPC>;lang=<lang>;output=<output>
Where:
prod_id – the manufacturer product code of the requested product, vendor_name – name of the manufacturer of the requested product, ean_upc – the EAN or UPC code to identify a product,
lang – the language code (INT – for the international/standardized repository, EN, NL, FR – for local ones),
output – the type of the response. It can be productxml (the product xml file) and metaxml (the index or meta information on the Brand + ProdID/M_Prod_ID or EAN/UPC).
For example, if you need to get an xml file of product ‘WM527 mouse Bluetooth Laser 1600 DPI Right-hand’. Manufactured by DELL, with product number WM527-BK, in English, the call will be:
https://data.Icecat.biz/xml_s3/xml_server3.cgi?prod_id=WM527-BK;vendor=dell;lang=en;output=productxml
In case you want to retrieve data on the basis of an EAN or UPC code:
https://data.Icecat.biz/xml_s3/xml_server3.cgi?ean_upc=5397063929863;lang=en;output=productxml
The meta data:
https://data.Icecat.biz/xml_s3/xml_server3.cgi?prod_id=WM527-BK;vendor=dell;lang=en;output=metaxml
XML Server 3 access works via the basic HTTP authorization, also used for other HTTP requests.
TIP 1: Use the Presentation_Value parameter in localized data-sheets for display of a data-sheet, and use the standardized Value parameter for search & compare on standardized values.
TIP 2: LocalValue is a transformation of the international Value according to the local dictionary and the applicable Metric/Imperial value system. Especially, for local applications,we advise to use LocalValue for search & compare functionalities
Open Icecat users have access to:
https://data.Icecat.biz/export/freexml/<lang>
for access to the standardized data files (QUALITY=ICECAT). Where <lang> stands for language code
TIP: use the standardized files for searching & comparing.
Please, note, where QUALITY = SUPPLIER only the original product data as imported from a manufacturer’s CMS is available, the respective data-sheet is not yet standardized by our editors. When standardized, the parameter changes to QUALITY = Icecat.
The language-specific product data-files are here:
https://data.Icecat.biz/export/freexml/<lang>/<product_id>.xml
where <lang> stands for language code and <product_id> stands for Icecat product ID
Please, note that this language-specific content contains the standardized content (same as in INT), which is entered by our editors. Or, if no standardized data is yet available, it contains just the original data (Quality=Supplier) imported from the CMS of a manufacturer or entered manually via the Icecat PIM by the manufacturer.
In general, we follow the ISO 639-1 two-character country codes, except when a code can not be unambiguously tied to a language. So we use for example EN_SG, and EN_IN to identify Singapore English and Indian English respectively
TIP: we advise the language-specific product content (Presentation_Value) purely for the display of product data-sheets, not for search & comparison. For search & comparison, we advise the standardized product content of the Value parameter from the /INT or local language directory. We advise LocalValue for country-specific search & compare applications.
TIP: Open Icecat is also useful for testing purposes before you upgrade to Full Icecat. The technical structures are 100% compatible with those of Full Icecat.
Full Icecat contains all product data of all 10,000s of brands supported. Often provided with service levels and coverage guarantees. For the Full Icecat subscribers, a separate directory structure is in place:
https://data.Icecat.biz/export/level4/<lang>
Where <lang> stands for language code.
Please, note, where QUALITY = SUPPLIER only the original product data as imported from a manufacturer’s PIM is available, the respective data-sheet is not yet standardized by our editors. When standardized, the parameter changes to QUALITY = ICECAT.
https://data.Icecat.biz/export/level4/<lang>/<product_id>.xml
where <lang> stands for language code and <product_id> stands for Icecat product ID.
TIP: We advise the language-specific product content (Presentation_Value parameter) purely for the display of product data-sheets, not for search & compare. For that, we advise the standardized product content (the Value parameter) in both the /INT and the respective local language directories. For country-specific applications we advise LocalValue in stead of the international Value.
Currently, we support most world languages and European languages: in total more than 55 languages or locales (country-language combinations). Our intention is to support eventually all languages. See our overview of supported languages and locales. If you feel that your language or the language that you need is missing, feel free to contact us. Probably we can work something out.
In general, we follow the language codes as given by the ISO 639-1 standard. However, in some cases, we choose the simpler two-character country codes. For example, when it’s obvious to which language code it relates (e.g., BR for Brazilian-Portuguese, or US for US English).
As files become bigger and bigger, and the number of files is growing fast, we support the gzipping of all interface files. Instead of doing this per file individually, we applied gzipping to all interface files automatically (mod_deflate). To benefit from it, you have to add one parameter to your calls.
About mod_deflate: mod_deflate transports our interface files as compressed gzip data, but will show it transparantly in its original format in modern browsers.
To benefit in your server scripts from gzip, you must use in any HTTP request this additional parameter:
Accept-Encoding: gzip
It will enforce gzipping!
Example:
GET /export/level4/EN/10.xml HTTP/1.1
Host: prf.Icecat.biz
User-Agent: Firefox/1.0
We often get questions like can we use FTP instead. Actually, this is really not necessary as http has very similar capabilities to FTP.
Here is example code in C# to download files. You can use it also in ASP.NET code. We tested it and it works ok:
public static Byte[] DownloadIcecatFile() { string strDownloadURL ="https://data.Icecat.biz/export/freexml/EN/480237.xml"; string strUser = "Your Login to Icecat Repository"; string strPWD = "Your Password to Icecat Repository"; // Creating an instance of a WebClient WebClient req = new WebClient(); // Creating an instance of a credential cache, // and passing the username and password to it CredentialCache cache = new CredentialCache(); cache.Add(new Uri(strDownloadURL), "Basic", new NetworkCredential(strUser, strPWD)); req.Credentials = cache; Byte[] fileData = req.DownloadData(strDownloadURL); return fileData;
In Classic ASP, MSXML2.ServerXMLhttp can be used Some details are present here
The purpose of the index files is to find the right product XML files for a given product. The matching between any portfolio and Icecat catalog has to be based on unique product identifiers and Icecat support few options:
There are four types of index files in our XML-OCI:
files.index.xml or .csv – An index file with references to product data-sheets in Open Icecat and with references to all described product data-sheets in Full Icecat
Path: https://data.icecat.biz/export/level4/<lang>/files.index.xml.gz or .csv.gz
daily.index.xml or .csv– a smaller index file with only references to the new or changed product data-sheets of the respective day
Path: https://data.icecat.biz/export/level4/<lang>/daily.index.xml.gz or csv.gz
on_market.index.xml or .csv – an index file with only the products that are currently on the market, as far as we can see that based on 100s of distributor and reseller price files
Path: https://data.icecat.biz/export/level4/<lang>/on_market.index.xml.gz or csv.gz
Tip: use the on_market index file, in case that you think the full index file is too big for you to process or too inefficient. DISCLAIMER: if our monitor for your market is incomplete, the on_market index file is probably too incomplete for your purposes.
nobody.index.xml or .csv – an index file with the products that are or were on the market for which we only have basic market data, but no complete data-sheet.
Path: https://data.icecat.biz/export/level4/<lang>/nobody.index.xml.gz or csv.gz
Tip: In case that you want to recategorize your products, for which we have no data-sheet (QUALITY=NOEDITOR), make use of the nobody index file
vertical.files.index.xml An index file with references to products that are in files.index.xml but limited to specific vertical. More information regarding vertical index files can be found here
Path: https://data.icecat.biz/export/level4/<lang>/<verticalID>.vertical.files.index.xml.gz
Important note:
Files include the index information for each product and gives the path to the XML product file. Below is an example of a file tag in the index file:
The On_Market attribute indicates that a product is still on the market (On_Market=“1”) or not seen on the market anymore (On_Market=“0”).
Per product, Icecat provides also the GTIN (EAN or UPC) codes, used for logistical purposes. It is possible that there are multiple GTINs (EANs or UPCs), as this varies with packaging or importers.
Further, we provide the Product code variants, used in the market, in a sub-list (M_Prod_ID).
Finally, the <Country_Market Value=”US”/> indicates that we see the product at the moment in the catalogs of US channel partners. This tag can be useful to limit your view on the index file, in case you don’t have distributor imports to do that.
The Model_Name attribute contains the product name. The Product_View attribute indicates how many times the current product is requested.
We refresh files.index.xml and daily.index.xml every day. You can use daily.index.xml to update only the data in your local database that has changed or is new in the Icecat database. files.index.xml, you can use the first time for a full import of needed product information of active data-sheets for Open Icecat users and all existing data-sheets for Full Icecat users, and for the regular analysis of coverage of your product database.
Historic and obsolete products can be downloaded via the real-time interface.
TIP 1: download ONLY the xml files that you actually need for your shop or application by using a daily catalog export from your ERP or shop (or actually the daily imports from your distributors/suppliers) as a filter. It is useless to import all the global product data files if you only operate in one geographic area or have an assortment limited to a certain category.
TIP 2: check the timestamps to only download files that have changed or check only for new files, to improve update performance cycles.
TIP 3: the daily index file also includes the list of product codes that are removed the day before (QUALITY=REMOVED). We remove product data-sheets, for example, when editors map them away. You can use it for cleansing your catalog.
The latest version of the DTD is here
Each product has its own data file in XML format. Its location is defined by the attribute “path” in the *.index.xml.gz. The file content includes all the available product information for the respective product.
The “Prod_ID” attribute is a manufacturer’s unique identifier for a product, often called Brand Product Code (Manufacturer Part Number, code de fabricant, Produkt-Kode, artikelnummer, etc). In combination with the “supplier_id” (i.e., the brand identifier) it is the unique key to select a product’s XML file via Icecat’s internal identifier (“Product_id”).
M_Prod_ID consists of one or more manufacturer product code(s) that are present in distributor or partner feeds but are mapped to the original (correct) Prod_ID.
The EAN_UPC sub-list lists the GTIN codes (EAN or UPC) that refer to this product. There can be multiple codes, as these are logistical codes depending on packaging types and geography.
On_Market (1=Yes, 0=No) indicates whether Icecat sees a product somewhere on the market or not. The Country_Market lists the individual markets (countries), where Icecat sees that the product is present. This indication can be used to limit the view on our index file.
DISCLAIMER: Our market view is always imperfect as we depend on the data of connected channel partners and distributors.
Catid is Icecat’s internal category identifier. For example Catid=”151” refers to Notebooks. For every category there is a second, external, category identifier based on UNSPSC. In the reference files or the product XML the meaning of categories can be found.
“Quality” attribute values:
SUPPLIER – This content is received from a brand’s PIM or entered manually by a brand editor via the Icecat PIM, and is not yet standardized by an Icecat editor. The language-specific directories are likely to contain the full (unstandardized) data-sheet.
ICECAT – This content is entered and/or standardized by Icecat editors. Such standardized data is present in the INT directory and the language-specific directories.
NOEDITOR – The content is received from a merchant (in most cases one of the 100s of distributors we are daily “polling”) and may be parsed. Editors haven’t described this product yet. Icecat doesn’t export the NOEDITOR data in XML to 3rd parties.
To help you with matching distributor data to the Icecat rich content, we are publishing the mappings of used variants of manufacturer names to the Icecat Supplier_ID
You can find this file in your repository, named supplier_mapping.xml.
Where <lang> stands for language code
Within this file, you’ll find the match between different manufacturer name variations and the original manufacturer name. The format is self-explanatory. The DTD for the message format is available here.
Different distributors often produce different manufacturer part numbers (MPN) for the same product.
To help you correct the MPNs from different distributors and to help you to get the best coverage, we are publishing the available MPN mappings to the official MPN. In every repository directory, the file with product mappings is published in the index files (files.index.xml, etc.).
OBSOLETE METHOD product_mapping.xml:
INT/product_mapping.xml
EN/product_mapping.xml
etc…
Note: We advise not to use product_mapping.xml files anymore as it has an obsolete structure. All product mapping data is already present in the XML index files.
In product XML feeds, you will find a lot of structures. We included these as references in the product data files, e.g., categories, features, measures (units), etc.
The reference files are present in the /refs/ subdirectory. For example, for Full Icecat, at https://data.Icecat.biz/export/level4/refs/ and for Open Icecat at https://data.Icecat.biz/export/freexml/refs/
The included files (entities) are:
Almost all files are present in Gzipped format. The typical use of the files is in combination with data from the INT (international) directory, not the localized directories as there the language-specific values are already automatically applied to the product XML files.
Brand products can have campaigns for specific period. CampaignsList.xml has the list of all assigned campaigns.
This chunk provides information on Icecat categories. Each category may be either searchable or not. Searchable categories may be used for product lookup by product feature values, as Icecat editors systematically maintain data integrity to allow smart searches.
Searchable=”1″: Use this category for product lookup in product list lookup requests.
Searchable=”0″: This category is not searchable (in our own product finder tools).
“Score” attribute in the response reflects the category usage statistic. The higher number means the higher the usage level. For example:
This chunk provides information about features that are assigned to every single category and are used for describing products in the category.
Each searchable feature will have LimitDirection attribute defined, which will give advice for finding a “better” feature value, either < or >. E.g. for feature “Hard disk capacity” it would be > relation, and for the feature “Write seek” it would be < relation.
LimitDirection="0": The relation is undefined LimitDirection="1": The relation is < LimitDirection="2": The relation is > LimitDirection="3": The relation is = Class="0": Key feature Class="1": Extra feature – tech details
Sometimes the possible feature values are restricted to a limited set of values. In that case, we list the possible values for the combination Category-Feature in the container RestrictedValue.
Icecat has dynamic relations (x-sells). They based on the product sets, manually defined. One product set relates to another one.
This chunk provides the list of supplier product families. Each product family may have information about it’s Name, Description in different languages, and category.
From Supplier list users can identify if brand belongs to Full or Open Icecat.
We log all Icecat product requests (real-time requests by end-users and xml requests). This data gives a product popularity indication, useful for sorting products and analyzing market trends. We publish this popularity data is daily. On brand level, we show this popularity data online.
In response to questions from different clients, we have created a Standardized Product Summary Description. The local versions are here:
Where <lang> stands for the language code. Column 1 is the Icecat ID, and column 2 and 3 give a short and a long standardized product summary description.
The short summary of column 2 is set-up as follows:
The extended summary of column 3 is set-up as follows:
TIP: Think about the guarantees that you get from your suppliers (distributors), based on errors made in their product descriptions. It may still be smart to make use of these supplier (distributor) texts in the ordering process for this reason.
The XML repository files comply to the standard Icecat XML response DTD, which is located here:
Please, note that the tag “No=” (example: No=”100090″) is a Priority indicator. The higher the number the more important the feature or feature group is for buyer orientation. Thus, we advise you to sort features and feature groups by priority, to get a more customer-friendly user presentation.
Product code is a deprecated field, earlier it used as product response status. Now, product code always equals 1.
Each Icecat XML has its XSD definition, you can see it as 2 mandatory attributes in the root XML tag.
You can find the basic information about XML Schema here. Icecat XSD sources are located at https://data.Icecat.biz/xsd/. Use these for Icecat XML validation.
You can check Icecat product XML example here for EN local
Notes:
Icecat supports 3D objects and Product Variants.
Bullet points are available through a separate tag.
See for a full overview: Digital Asset Types as Represented in Icecat’s Open Catalog Interfaces.
You can see all the error messages that you can get while using XML files from Icecat via the following link
Read further: Manuals, oci, PDS, PIM, XML
Chief Operating Officer at Icecat
None of the above dtd links returns a full dtd file.
Do I assume are unique to each product?
Can you explain how is related to products.
Thanks
Dear Alan
We have different DTD links for different export file types. They are mentioned in the manual. For the product data sheets we have thus only one DTD: https://data.icecat.biz/dtd/ICECAT-interface_response.dtd. It is valid for all products in the Icecat database. Please, let us know if you have further questions.
Is there any way of getting an index file per vertical, i.e. 2833 for Computers and Peripherals on it's own, rather than the complete list?
Yes, the most efficient and tailored way is to generate a personalized index based on your own catalog/price list https://iceclog.com/manuals-personalized-interface-files-and-catalogs/
Is there a notification process for any changes in the categories, if they ever change?
Yes, read https://iceclog.com/icecat-release-notes-111/ We're busy with rolling out timestamps in Reference files (per Taxonomy item).
Hi there, I would to ask for a Sony STR-DN610 SERVICE MANUAL, your help will be much appreciated thank you.
Thx. Will ask SONY to provide it.
Thanks for this.
I'm still struggling to see how I would provide a category id and in what format to get the relevant products.
Can you please provide and example for me?
Thanks Martin...
I ask a colleague to contact you directly
How can we have the name and id of all the categories for a given vertical? Where are verticals in the .xml files that you provided here? Thank you
Dear Max We store detailed information per category in the CategoriesList.xml.gz that you can find via the following link: https://data.Icecat.biz/export/freexml/refs/ By tag "ParentCategory ID" You can identify the hierarchical connection between categories.
Hi, is there a limit to how many queries I can make to the server at any one time? I'm getting a "GO AWAY" from the server when my program is downloading XML data sheets now.
Dear Martin
For XML the limit is 100 connections from one IP, however, I suggest sticking to the number of 50-75.
Kind regards
Vazha Abramishvili
Hi,
can I filter products only on BRAND ?
Thanks Milan
Hi Milan, Thanks, our country manager will contact you. Regards, Wouter
Hi icecat,
How to get data in multiple languages in a single call?
Thanks & Regards, Nikita
Hi Dev, thanks for reaching out. Your country manager Gio Abramishvilli will contact you asap. Or in the meantime you can send a request to gio at icecat dot com. Thanks, Wouter
How can I filter CategoriesList by an specific Language like English?
Hi apiazyro, the category list is multilingual. You can filter on "langid=XX". Regards, Wouter
Hello,
Is it possible to filter products based solely on the brand?
Thanks, Milan
Hello Milan, thanks for your questions. You can filter on requesting a product XML by Brand and Product Code or by GTIN (EAN or UPC). See paragraph 2.2 in the manual. Regards, Wouter
Hello, is there a restriction on the number of queries I can send to the server simultaneously? I'm currently receiving a "GO AWAY" response from the server while my program is downloading XML data sheets.
Thank you.
Hello, the max number of queries is 100 connections per second per IP. Regards, Wouter
Your email address will not be published. Required fields are marked *
Comment *
Name *
Email *
Website
Δ
