In this paper, we study a type of susceptible-exposed-infected(SEI) epidemic model with varying population size and introduce the random perturbation of the constant contact rate into the SEI epidemic model due to the universal existence of fluctuations. Under some moderate conditions, the density of the exposed and the infected individuals exponentially approaches zero almost surely are derived. Furthermore, the stochastic SEI epidemic model admits a stationary distribution around the endemic equilibrium, and the solution is ergodic. Some numerical simulations are carried out to demonstrate the efficiency of the main results.