Concerns are rising about the capacity of species to adapt quickly enough to climate change. In long‐lived organisms such as trees, genetic adaptation is slow, and how much phenotypic plasticity can help them cope with climate change remains largely unknown. Here, we assess whether, where and when phenological plasticity is and will be adaptive in three major European tree species. We use a process‐based species distribution model, parameterized with extensive ecological data, and manipulate plasticity to suppress phenological variations due to interannual, geographical and trend climate variability, under current and projected climatic conditions. We show that phenological plasticity is not always adaptive and mostly affects fitness at the margins of the species' distribution and climatic niche. Under current climatic conditions, phenological plasticity constrains the northern range limit of oak and beech and the southern range limit of pine. Under future climatic conditions, phenological plasticity becomes strongly adaptive towards the trailing edges of beech and oak, but severely constrains the range and niche of pine. Our results call for caution when interpreting geographical variation in trait means as adaptive, and strongly point towards species distribution models explicitly taking phenotypic plasticity into account when forecasting species distribution under climate change scenarios.