Construction of Word Segmentation Model Based on HMM+BI-LSTM

• 作者: Bin Wen Hang Zhang
• 来源期刊: 国际计算机前沿大会会议论文集
• 学科: 工学
• 关键词: Chinese word segmentation HMM BI-LSTM Sequence tagging
• 年，卷（期）: 2020,（2）
• 页码范围: 47-61
• 页数: 15页
• 分类号: TP3

摘要

Chinese word segmentation plays an important role in search engine,artificial intelligence,machine translation and so on.There are currently three main word segmentation algorithms:dictionary-based word segmentation algorithms,statistics-based word segmentation algorithms,and understandingbased word segmentation algorithms.However,few people combine these three methods or two of them.Therefore,a Chinese word segmentation model is proposed based on a combination of statistical word segmentation algorithm and understanding-based word segmentation algorithm.It combines Hidden Markov Model(HMM)word segmentation and Bi-LSTM word segmentation to improve accuracy.The main method is to make lexical statistics on the results of the two participles,and to choose the best results based on the statistical results,and then to combine them into the final word segmentation results.This combined word segmentation model is applied to perform experiments on the MSRA corpus provided by Bakeoff.Experiments show that the accuracy of word segmentation results is 12.52%higher than that of traditional HMM model and 0.19%higher than that of BI-LSTM model.