Time-completeness trade-offs in record linkage using adaptive query processing

Abstract

Applications that involve data integration among multiple sources often require a preliminary step of data reconciliation in order to ensure that tuples match correctly across the sources. In dynamic settings such as data mashups, however, traditional offline data reconciliation techniques that require prior availability of the data may not be applicable. The alternative, performing similarity joins at query time, is computationally expensive, while ignoring the mismatch problem altogether leads to an incomplete integration. In this paper we make the assumption that, in some dynamic integration scenarios, users may agree to trade the completeness of a join result in return for a faster computation. We explore the consequences of this assumption by proposing a novel, hybrid join algorithm that involves a combination of exact and approximate join operators, managed using adaptive query processing techniques. The algorithm is optimistic: it can switch between physical join operators multiple times throughout query processing, but it only resorts to approximate join operators when there is statistical evidence that result completeness is compromised. Our experiments show that sensible savings in join execution time can be achieved in practice, at the expense of a modest reduction in result completeness.

Approximate Queries with Adaptive Processing

“…In [77], the use of adaptive techniques for combining exact (fast) and approximate (accurate) joins when performing dynamic integration has been proposed. The adaptive algorithm uses an a-priori expectation of the join result size combined with the monitoring of join progress to statistically determine, at various points during query execution, which join operator should be used.…”

“…However, the motivating and illustrating examples discussed in Section 10.2, that would fill the gap in the QoD-QoD cell of the table, show that QoD 2 techniques could be very relevant for various data management applications. Some preliminary efforts towards the definition of a reference framework for QoD 2 techniques have been presented in [29], where they are called ASAP (Approximate Search with Adaptive Processing) techniques, and some specific concrete techniques have already been proposed [77]. Much more work is required in order to determine their concrete impact on stored data and data stream management.…”

“…It may be the case, therefore, that for at least a portion of the tables a fast exact join could be used instead, without loss of joined tuples. In [77], the monitor component of the adaptive strategy is based on the assumption that a parent-child relationship is expected between the two input tables, a common case exemplified by the car accidents scenario. Under this assumption, join cardinality can be estimated easily and deviations from the expected cardinality reveal that mismatches are occurring and thus some approximation should be introduced.…”

“…We recall that a pipelined plan executes all operators in the query plan in parallel, by sending the output data of an operator directly to the next operator in the pipeline, as opposed to materialized plans in which operators are applied in sequence, computing (and materializing if needed) whole intermediate results. An example of operator that can be elaborated in pipeline is Symmetric Set Hash Join, exploited in [77], which is a pipelined symmetric hash implementation of the q-gram based set join of [33]. Another example is a pipelined implementation of Trie-Join [123], that allows to assess tuple similarity as edit-distance constraints rather than as number of common q-grams.…”

Frameworks for entity matching: A comparison

“…Some recent work regarding scalability has focused on computational aspects of string similarity computation [50,1,30,61] and time-completeness trade-offs [39]. Furthermore, we see a strong need for comparative performance evaluations of different frameworks and EM strategies.…”