A de-duplication tool: Implementation of duplicate detection algorithms

Data quality is an important issue in any data store especially when it is used for analysis. Poor quality of data would drastically affect the decision making process. Cleansing is an activity performed before data is loaded into the warehouse to enhance the quality and consistency of the data. This paper addresses one of the key data quality problem namely detection of duplicates. A duplicate detection tool is built that implements two algorithms namely Sorted Neighborhood method and Token based cleansing method. The tool was tested using student dataset of 118 records and showed an accuracy of 88.09%. This paper is a primitive analysis of how duplicates can be detected in a dataset.