This article gives an overview of the general situation and regional differences of rural poverty in China, and points out that in the rural areas of central and western regions with serious soil erosion, the poverty is particularly prominent. Based on previous studies, we take agricultural GDP as the dependent variable, farmland area, agricultural labor, agricultural capital stock, irrigation area, and consumption of chemical fertilizer as the independent variables, to establish the C-D production function reflecting impact of soil erosion on rural poverty for regression analysis. The results show that farmland, labor, capital, irrigation, chemical fertilizer and other production factors have a positive effect on agricultural GDP; soil erosion has a significant negative effect on agricultural production; in western China, the total factor productivity is the lowest and soil erosion is the most serious. In order to resolve the dilemma of soil erosion and rural poverty, it is necessary to change the extensive mode of agricultural development, protect the ecological environment, and take the road of intensive development of agroecology.