Information theoretic-based sampling of observations

Abstract

Due to the surge in the amount of data that are being collected, analysts are increasingly faced with very large data sets. Estimation of sophisticated discrete choice models (such as Mixed Logit models) based on these typically large data sets can be computationally burdensome, or even infeasible. Hitherto, analysts tried to overcome these computational burdens by reverting to less computationally demanding choice models or by taking advantage of the increase in computational resources. In this paper we take a different approach: we develop a new method called Sampling of Observations (SoO) which scales down the size of the choice data set, prior to the estimation. More specifically, based on information-theoretic principles this method extracts a subset of observations from the data which is much smaller in volume than the original data set, yet produces statistically nearly identical results. We show that this method can be used to estimate sophisticated discrete choice models based on data sets that were originally too large to conduct sophisticated choice analysis.

Choice modelling in the age of machine learning - Discussion paper

“…Consequently, computational times are determined by both the size of the dataset and the complexity of the likelihood function; as either increases, so do computational times (e.g. Van Cranenburgh and Bliemer 2019). This practice becomes increasingly unwieldy when faced with ever larger data sets.…”

New software tools for creating stated choice experimental designs efficient for regret minimisation and utility maximisation decision rules

“…The Fedorov algorithm starts by taking a randomly drawn design (this is termed 'an iteration'), and continues by replacing choice tasks in the design with those from the candidate set and evaluating its impact (see e.g. Van Cranenburgh and Bliemer, 2018). After having exhausted the first series of replacements, a first efficient design is found.…”