Scanner data in inflation measurement: From raw data to price indices

Abstract

Scanner data offer new opportunities for CPI or HICP calculation. They can be obtained from a wide variety of retailers (supermarkets, home electronics, Internet shops, etc.) and provide information at the level of the barcode. One of advantages of using scanner data is the fact that they contain complete transaction information, i.e. prices and quantities for every sold item. After clearing data and unifying product names, products should be carefully classified (e.g. into COICOP 5 or below), matched, filtered and aggregated. One of new challenges connected with scanner data is the appropriate choice of the index formula. In this article we present a proposal for the implementation of individual stages of handling scanner data. We also point out potential problems during scanner data processing and their solutions. We compare a large number of price index methods based on real scanner data sets and we verify their sensitivity on adopted data filtering and aggregating methods. One of the aims is also to compare calculations of multilateral indices in terms of how time-consuming they are. Finally, the paper investigates distances between these indices and the theoretical, expected value of the price share when prices are log-normally distributed. It is a new approach to providing an additional criterion in the price index selection.

Improving quality of the scanner CPI: proposition of new multilateral methods

“…It seems that this observation confirms the separateness of indices that meet the identity test. The Geary-Khamis index is a good proxy for the Time Product Dummy (TPD) index, which confirms some previous results (Chessa et al 2017;Białek and Berȩsewicz 2021), but it always seems to be the most distant from the GEKS-L index.…”

“…On the other hand, the GEKS and CCDI indices seem to be more sensitive to the so called "dump prices" (dumping occurs when a product is sold at reduced prices before removing from the market). Białek and Berȩsewicz (2021) draw attention to the differences in the calculation time when determining price indices. In particular they suggest that compiling the Geary-Khamis (GK) and TPD indices is more time consuming than calculation of the GEKS or CCDI indices.…”

“…the data_filtering() function from the cited package was used. The extreme price filter (Białek and Berȩsewicz 2021) was applied to eliminate products with more than three-fold price increase or more than double price drop from month to month. The low sale filter (van Loon and Roels 2018) was used to eliminate products with relatively low sales from the sample (almost 30% of products were removed).…”

“…automatic product classifying into COICOP groups, product matching, data filtering, data aggregating, etc. (Białek and Berȩsewicz 2021), since a typical supermarket uses 10,000-25,000 item codes. One of main challenges is the proper choice of the price index formula, which should take into account seasonal products together with new and disappearing goods.…”

Assessing the effect of new data sources on the consumer price index: a deterministic approach to uncertainty and sensitivity

“…The list of barcodes most commonly used by retail chains includes GTIN (Global Trade Item Number) or its European version -EAN (European Article Number), PLU (Price Look-Up) and SKU (Stock Keeping Unit). Product codes, including the code assigned by a given chain store, together with the product label, are used for classifying products using 5-digit ECOICOP codes and below this level of aggregation (Chessa, 2015;2016, Białek andBeręsewicz, 2021). One of the benefits of using scanner data is that they contain information about the level of consumption even at the lowest level of aggregation.…”