The precise positioning of dopant atoms within bulk crystal lattices could enable novel applications in areas including solid-state sensing and quantum computation. Established scanning probe techniques are capable tools for the manipulation of surface atoms, but at a disadvantage due to their need to bring a physical tip into contact with the sample. This has prompted interest in electron-beam techniques, followed by the first proof-of-principle experiment of bismuth dopant manipulation in crystalline silicon. Here, we use first-principles modeling to discover a novel indirect exchange mechanism that allows electron impacts to non-destructively move dopants with atomic precision within the silicon lattice. However, this mechanism only works for the two heaviest group V donors with split-vacancy configurations, Bi and Sb. We verify our model by directly imaging these configurations for Bi and by demonstrating that the promising nuclear spin qubit Sb can be manipulated using a focused electron beam.