Join BookitisSave favorites, build lists, and follow creators.

An introduction to duplicate detection

Work detail

Bookitis Pick
Cover for An introduction to duplicate detection
AI
Image source: Open Library
Melanie HerschelFelix Naumann2 editions

With the ever increasing volume of data, data quality problems abound. Multiple, yet different representations of the same real-world objects in data, duplicates, are one of the most intriguing data quality problems. The effects of such duplicates are detrimental; for instance, bank customers can obtain duplicate identities, inventory levels are monitored incorrectly, catalogs are mailed multiple times to the same household, etc. Automatically detecting duplicates is difficult: First, duplicate representations are usually not identical but slightly differ in their values. Second, in principle all pairs of records should be compared, which is infeasible for large volumes of data. This lecture examines closely the two main components to overcome these difficulties: (i) Similarity measures are used to automatically identify duplicates when comparing two records.Well-chosen similarity measures improve the effectiveness of duplicate detection. (ii) Algorithms are developed to perform on very large volumes of data in search for duplicates.Well-designed algorithms improve the efficiency of duplicate detection. Finally, we discuss methods to evaluate the success of duplicate detection.

Overview

Shared work-level identity and catalog context.

2 credited authorsSearch language english

Bookitis keeps work pages focused on the shared book identity and the editions that actually belong to it. Unrelated books should not appear here as primary content.

Contributors

People credited with this work in the active catalog.

  • Melanie Herschel

    Author profile in the active Bookitis catalog

    Open Author
  • Felix Naumann

    Author profile in the active Bookitis catalog

    Open Author

Editions

Publication-specific versions linked to this work only.