An Empirical Study of Good-Turing Smoothing for Language Models on Different Size Corpora of Chinese

• 作者: Chien-Yo Hwang Feng-Long Huang Ming-Shing Yu
• 来源期刊: 电脑和通信(英文)
• 学科: 医学
• 关键词: Good-Turing Methods SMOOTHING LANGUAGE Models PERPLEXITY
• 年，卷（期）: 2013,（5）
• 页码范围: 14-19
• 页数: 6页
• 分类号: R73

摘要

Data sparseness has been an inherited issue of statistical language models and smoothing method is usually used to resolve the zero count problems. In this paper, we studied empirically and analyzed the well-known smoothing methods of Good-Turing and advanced Good-Turing for language models on large sizes Chinese corpus. In the paper, ten models are generated sequentially on various size of corpus, from 30 M to 300 M Chinese words of CGW corpus. In our experiments, the smoothing methods;Good-Turing and Advanced Good-Turing smoothing are evaluated on inside testing and outside testing. Based on experiments results, we analyzed further the trends of perplexity of smoothing methods, which are useful for employing the effective smoothing methods to alleviate the issue of data sparseness on various sizes of language models. Finally, some helpful observations are described in detail.