Increasing Momentum-Like Factors:A Method for Reducing Training Errors on Multiple GPUs

• 作者: Yu Tang Zhigang Kan Lujia Yin Zhiquan Lai Zhaoning Zhang Linbo Qiao Dongsheng Li
• 来源期刊: 清华大学学报自然科学版（英文版）
• 年，卷（期）: 2022,（1）
• 所属期刊栏目: REGULAR ARTICLES
• 页码范围: 114-126
• 页数: 13页
• 语种: 英文
• DOI: 10.26599/TST.2020.9010023

摘要

In distributed training,increasing batch size can improve parallelism,but it can also bring many difficulties to the training process and cause training errors.In this work,we investigate the occurrence of training errors in theory and train ResNet-50 on CIFAR-10 by using Stochastic Gradient Descent (SGD) and Adaptive moment estimation(Adam) while keeping the total batch size in the parameter server constant and lowering the batch size on each Graphics Processing Unit (GPU).A new method that considers momentum to eliminate training errors in distributed training is proposed.We define a Momentum-like Factor (MF) to represent the influence of former gradients on parameter updates in each iteration.Then,we modify the MF values and conduct experiments to explore how different MF values influence the training performance based on SGD,Adam,and Nesterov accelerated gradient.Experimental results reveal that increasing MFs is a reliable method for reducing training errors in distributed training.The analysis of convergent conditions in distributed training with consideration of a large batch size and multiple GPUs is presented in this paper.