A novel data cleaning method to improve the data quality in a data warehouse using WordNet 3.1

Data cleaning is an important aspect of data warehousing as the quality of data is directly proportional to the kind of data that goes into the data warehouse. It is necessary to clean data and make it suitable for user requirements before it can be used for analytics. Data Cleansing includes removing various inconsistencies or ambiguities from data like incorrect data entries, inconsistent database selections and different schema arrangements. Data cleaning algorithms are generally used for cleaning the data by filling the missing values and removing the noisy data, by replacing or modifying the dirty data. Data cleaning is deeply domain-specific. There is no international common standard for reference. So the process of data cleaning varies from domain to domain but basically a process used to determine inaccurate, incomplete, or unreasonable data and then improving the quality through correction of detected errors and missing values. The principle of data cleaning is to find and rectify the errors and inconsistencies. Data cleaning approaches derive observations of data from the values of independent attributes and records.

In this paper we have explored the importance of data cleaning for improving the quality of data in a data warehouse and have also proposed a method of data cleaning using a string detection algorithm by scanning similar words using WordNet version 3.1. If a given database contains words with similar meaning and thus create redundancy, then the words are replaced by each other so that the data gets cleaned.