Data acquisition for improving machine learning models

Abstract

The vast advances in Machine Learning (ML) over the last ten years have been powered by the availability of suitably prepared data for training purposes. The future of ML-enabled enterprise hinges on data. As such, there is already a vibrant market offering data annotation services to tailor sophisticated ML models. In this paper, inspired by the recent vision of online data markets and associated market designs, we present research on the practical problem of obtaining data in order to improve the accuracy of ML models. We consider an environment in which consumers query for data to enhance the accuracy of their models and data providers who possess data make them available for training purposes. We first formalize this interaction process laying out the suitable framework and associated parameters for data exchange. We then propose two data acquisition strategies that consider a trade-off between exploration during which we obtain data to learn about the distribution of a provider's data and exploitation during which we optimize our data inquiries utilizing the gained knowledge. In the first strategy, Estimation and Allocation (EA), we utilize queries to estimate the utilities of various predicates while learning about the distribution of the provider's data; then we proceed to the allocation stage in which we utilize those learned utility estimates to inform our data acquisition decisions. The second algorithmic proposal, named Sequential Predicate Selection (SPS), utilizes a sampling strategy to explore the distribution of the provider's data, adaptively investing more resources to parts of the data space that are statistically more promising to improve overall model accuracy. We present a detailed experimental evaluation of our proposals utilizing a variety of ML models and associated real data sets exploring all applicable parameters of interest. Our results demonstrate the relative benefits of the proposed algorithms. Depending on the models trained and the associated learning tasks we identify trade-offs and highlight the relative benefits of each algorithm to further optimize model accuracy.

Forecasting Conversion Rate for Real Time CPC Bidding With Target ROAS

“…There is a heuristic called Thompson sampling [9], [10]. It was used in revenue management [11], web site optimization [12], in online advertising [13], in selective data acquisition for improving machine learning models [14], and more interestingly in designing multi-armed bandits (MABs) [15]. The MABs were shown to be useful in a wide range of applications, which require continuous improvement, ranging from online service economy [16], to portfolio selection in finance [17], [18], and to real time bid prediction in online advertising [19].…”

Representative Routes Discovery from Massive Trajectories

“…Tuple discovery [11,14,18,27,34,38,58] selects tuples from a pool of available datasets that are the most useful for a pre-defined target (model training [11], causal inference in question answering [18]), and the usefulness of tuples is checked for a single target. In contrast, our work focuses on acquiring entire datasets rather than tuples.…”

AI-Based Digital Rocks Augmentation and Assessment Metrics

“…Training data acquisition has been a challenge for AI model training, because of the general lack of training data, the increasing demand for training data with the wider application of AI, and the high cost to acquire data (Li et al, 2021;Roh et al, 2021). While this is the case for many types of AI modeling, one important example relevant to digital rock physics is high-resolution imaging of porous media, such as X-ray Micro-computed tomography (Micro-CT, see review in Wildenschild & Sheppard, 2013), which is time-consuming especially for high-resolution scans and results in limited data acquisition with scanning one sample at a time.…”