Choice of discount rate in reinforcement learning with long-delay rewards

• 作者: LIN Xiangyang XING Qinghua LIU Fuxian
• 来源期刊: 系统工程与电子技术（英文版）
• 年，卷（期）: 2022,（2）
• 所属期刊栏目: SYSTEMS ENGINEERING
• 页码范围: 381-392
• 页数: 12页
• 语种: 英文
• DOI: 10.23919/JSEE.2022.000040

摘要

In the world, most of the successes are results of long-term efforts. The reward of success is extremely high, but be-fore that, a long-term investment process is required. People who are "myopic" only value short-term rewards and are unwill-ing to make early-stage investments, so they hardly get the ulti-mate success and the corresponding high rewards. Similarly, for a reinforcement learning (RL) model with long-delay rewards, the discount rate determines the strength of agent's "farsightedness". In order to enable the trained agent to make a chain of correct choices and succeed finally, the feasible region of the discount rate is obtained through mathematical derivation in this paper firstly. It satisfies the "farsightedness" requirement of agent. Af-terwards, in order to avoid the complicated problem of solving implicit equations in the process of choosing feasible solutions, a simple method is explored and verified by theoreti cal demon-stration and mathematical experiments. Then, a series of RL ex-periments are designed and implemented to verify the validity of theory. Finally, the model is extended from the finite process to the infinite process. The validity of the extended model is veri-fied by theories and experiments. The whole research not only reveals the significance of the discount rate, but also provides a theoretical basis as well as a practical method for the choice of discount rate in future researches.