Exploration of stage-specific effects of maternal exposure to trace elements and toxic metals on infancy continuous growth and trajectories is critical for early-life health management. Within a Chinese prospective cohort in 2014–2015, a total of 919 mother–infant pairs were included, and the urinary levels of 17 elements including vanadium (V), chromium (Cr), manganese, iron, cobalt, nickel, copper, zinc, arsenic, molybdenum, palladium, cadmium, tin, gold, mercury, thallium, and lead in early (mean: 11.9 weeks), and late pregnancy (mean: 32.4 weeks) were assessed. Standardized anthropometric assessments of infants were conducted at 1, 3, 6, 8, and 12 months of age. A three-step longitudinal and high-dimensional data analysis procedure was carried out to estimate the impacts of exposome on dynamic growth. Early-pregnancy exposures to V and Cr were positively associated with repeated measurements of length-for-age z-scores (LAZ). Six trajectories were identified based on LAZ. Maternal single exposure to V and Cr as well as mixed exposure to trace elements in early pregnancy were associated with raised odds for the high-stable group. Our results suggested positive associations between maternal trace element exposome and infancy dynamic growth. V and Cr were the key elements and the early pregnancy might be the critical window.