Intelligent Web Robot for Content Extraction

• 作者: Jie LI Weiwei WANG Wenxing HONG Yang WENG
• 来源期刊: 仪器仪表学报:英文版
• 学科: 工学
• 关键词: Content Extraction DOM Machine Learning Xgboost
• 年，卷（期）: 2019,（3）
• 页码范围: 52-58
• 页数: 7页
• 分类号: TP242.6

摘要

The main content of a news web page is a source of data for Natural Language Processing(NLP)and Information Retrieval(IR),which contains large quantities of valuable information.This paper proposes a method that formulates the main content extraction problem as a DOM tree node classification problem.In terms of feature extraction,we use the DOM tree node to represent HTML document and then develop multiple features by using the DOM tree node properties,such as text length,tag path,tag properties and so on.In consideration that the essence of the problem is the classification model,we use Xgboost to help select nodes.Experimental results show that the proposed approach is effective and efficient in extracting main content of new web pages,and achieves about 98%accuracy over 1083 news pages from 10 different new sites,and the average processing time per page is within 10 ms.