Multi-GPU Based Recurrent Neural Network Language Model Training

• 作者: Hong Ye Naijie Gu Xiaoci Zhang
• 来源期刊: 国际计算机前沿大会会议论文集
• 学科: 社会科学
• 关键词: RECURRENT NEURAL network LANGUAGE MODELS (RNNLMs)
• 年，卷（期）: 2016,（1）
• 页码范围: 124-126
• 页数: 3页
• 分类号: C5

摘要

Recurrent neural network language models (RNNLMs) have been applied in a wide range of research fields, including nature language processing and speech recognition. One challenge in training RNNLMs is the heavy computational cost of the crucial back-propagation (BP) algorithm. This paper presents an effective approach to train recurrent neural network on multiple GPUs, where parallelized stochastic gradient descent (SGD) is applied. Results on text-based experiments show that the proposed approach achieves 3.4× speedup on 4 GPUs than the single one, without any performance loss in language model perplexity.