Targeted Feedback Collection Applied to Multi-Criteria Source Selection

Abstract

Abstract. A multi-criteria source selection (MCSS) scenario identifies, from a set of candidate data sources, the subset that best meets a user's needs. These needs are expressed using several criteria, which are used to evaluate the candidate data sources. A MCSS problem can be solved using multi-dimensional optimisation techniques that trade-off the different objectives. Sometimes we may have uncertain knowledge regarding how well the candidate data sources meet the criteria. In order to overcome this uncertainty, we may rely on end users or crowds to annotate the data items produced by the sources in relation to the selection criteria. In this paper, we introduce an approach called Targeted Feedback Collection (TFC), which aims to identify those data items on which feedback should be collected, thereby providing evidence on how the sources satisfy the required criteria. TFC targets feedback by considering the confidence intervals around the estimated criteria values. The TFC strategy has been evaluated, with promising results, against other approaches to feedback collection, including active learning, using real-world data sets.

Incorporating Data Context to Cost-Effectively Automate End-to-End Data Wrangling

“…We will investigate the effect of additional data context types on the wrangling pipeline, and on other wrangling stages such as Web data extraction [49]. To further address time-varying variety and veracity problems in data wrangling, we will investigate feedback-based learning and model refinement techniques such as presented in [42] or [50]. Furthermore, we are exploring how to combine evidence gained from data context with user preferences, as shown in [23], to elaborate the possibilities in tailoring a data product for users with different requirements.…”

“…Source instances can be exploited by mapping selection approaches to calculate mapping-specific criteria and mapping overlap estimations. In general, mapping statistics such as the number of nulls, the standard deviation of attributes or the number of tuples in the mapping can be used as input for multi-criteria based optimisation techniques [42], [23].…”

“…Target instances can be exploited by source or mapping selection approaches to calculate quality criteria [43] to improve the selection process. In [42], feedback instances are used to estimate different quality dimensions such as accuracy and freshness of sources. In general, these estimates can be used as input to source or mapping selection approaches.…”