Knowledge transfer in multi-agent reinforcement learning with incremental number of agents

• 作者: LIU Wenzhang DONG Lu LIU Jian SUN Changyin
• 来源期刊: 系统工程与电子技术（英文版）
• 年，卷（期）: 2022,（2）
• 所属期刊栏目: CONTROL THEORY AND APPLICATION
• 页码范围: 447-460
• 页数: 14页
• 语种: 英文
• DOI: 10.23919/JSEE.2022.000045

摘要

In this paper, the reinforcement learning method for cooperative multi-agent systems (MAS) with incremental num-ber of agents is studied. The existing multi-agent reinforcement learning approaches deal with the MAS with a specific number of agents, and can learn well-performed policies. However, if there is an increasing number of agents, the previously learned in may not perform well in the current scenario. The new agents need to learn from scratch to find optimal policies with others, which may slow down the learning speed of the whole team. To solve that problem, in this paper, we propose a new algorithm to take full advantage of the historical knowledge which was learned before, and transfer it from the previous agents to the new agents. Since the previous agents have been trained well in the source environment, they are treated as teacher agents in the target environment. Correspondingly, the new agents are called student agents. To enable the student agents to learn from the teacher agents, we first modify the input nodes of the networks for teacher agents to adapt to the current envi-ronment. Then, the teacher agents take the observations of the student agents as input, and output the advised actions and va-lues as supervising information. Finally, the student agents com-bine the reward from the environment and the supervising in-formation from the teacher agents, and learn the optimal policies with modified loss functions. By taking full advantage of the knowledge of teacher agents, the search space for the student agents will be reduced significantly, which can accelerate the learning speed of the holistic system. The proposed algorithm is verified in some multi-agent simulation environments, and its ef-ficiency has been demonstrated by the experiment results.