Impacts of Dirty Data on Classification and Clustering Models: An Experimental Evaluation

• 作者: Zhi-Xin Qi Hong-Zhi Wang An-Jie Wang
• 来源期刊: 计算机科学技术学报（英文版）
• 年，卷（期）: 2021,（4）
• 所属期刊栏目: Special Section on AI4DB and DB4AI
• 页码范围: 806-821
• 页数: 16页
• 语种: 英文

摘要

Data quality issues have attracted widespread attentions due to the negative impacts of dirty data on data mining and machine learning results. The relationship between data quality and the accuracy of results could be applied on the selection of the appropriate model with the consideration of data quality and the determination of the data share to clean. However, rare research has focused on exploring such relationship. Motivated by this, this paper conducts an experimental comparison for the effects of missing, inconsistent, and conflicting data on classification and clustering models. From the experimental results, we observe that dirty-data impacts are related to the error type, the error rate, and the data size. Based on the findings, we suggest users leverage our proposed metrics, sensibility and data quality inflection point, for model selection and data cleaning.