This paper reports results from an investigation of the interaction of displaced Si-self atoms (I) and their vacancies (V), with impurities in crystalline silicon (Si), as induced by micro-second pulse duration irradiation with electrons at different energies: 3.5, 14, 25 and 50 MeV and pico-second pulse duration with energy 3.5 MeV. V-V, I-impurity atom and V-impurity atom interactions are analyzed both experimentally and as modeled using computer simulations. A process of divacancy (V2) accumulation in the dose-dependent linear region is investigated. The effect of impurities on recombination of correlated divacancies, and I-atoms that had become displaced from regular lattice points is estimated by computer modeling of an appropriate diffusion-controlled process. It is concluded that the experimental results can be interpreted quantitatively in terms of a strongly anisotropic quasi-one-dimensional diffusion of displaced I-atoms. In addition, a significant difference is found between the effects of pico-second duration electron beam irradiation, which causes the formation of A-centre (V + Oxygen) clusters, while when the beam is applied on a micro-second timescale, divacancies are created instead, although the electrons have the same energy in both cases.