This study aims to examine the association between change in physical activity over time and accumulation of cardiometabolic risk factors. Four consecutive surveys (Time 1 to 4) were conducted with 4-year intervals in 1997–2013 (the Finnish Public Sector study). Physical activity of 15,634 cardio-metabolically healthy participants (mean age 43.3 (SD 8.7) years, 85% women) was assessed using four-item survey measure and was expressed as weekly metabolic equivalent (MET) hours in Time 1, 2, and 3. At each time point, participants were categorised into low (<14 MET-h/week), moderate (≥14 to <30 MET-h/week), or high (≥30 MET-h/week) activity level and change in physical activity levels between Time 1 and 3 (over 8 years) was determined. The outcome was the number of incident cardiometabolic risk factors (hypertension, dyslipidemia, diabetes, and obesity) at Time 4. Cumulative logistic regression was used for data analysis. Compared to maintenance of low physical activity, increase in physical activity from low baseline activity level was associated with decreased accumulation of cardiometabolic risk factors in a dose-response manner (cumulative odds ratio [cOR] = 0.73, 95% CI 0.59–0.90 for low-to-moderate and cOR = 0.67, 95% CI 0.49–0.89 for low-to-high, P for trend 0.0007). Decrease in physical activity level from high to low was associated with increased accumulation of cardiometabolic risk factors (cOR = 1.60, 95% CI 1.27–2.01) compared to those who remained at high activity level. Thus even a modest long-term increase in physical activity was associated with reduction in cardiometabolic risk whereas decrease in physical activity was related to increased risk.