This study investigated the longitudinal association between changes in older adults’ physical activity and neighborhood walkability measured by geographic information systems (GISs, (ArcGIS, ESRI Inc., Redlands, CA, USA)). A mail survey was conducted for Japanese older adults who were randomly selected from three different settlement types. Data on walking, total moderate to vigorous physical activity (MVPA), and sociodemographic characteristics were collected at baseline (in 2010) and follow-up (in 2015). Multiple linear regression analyses were employed to assess the association between MVPA change and neighborhood walkability, adjusted for potential confounders. Effect sizes for independent variables on MVPA change were estimated. Data from 731 community-dwelling older adults (43.7% women) were analyzed. During the follow-up, older adults’ MVPA was reduced by 94.4 min/week (−14.2%) on average (675.5 and 579.9 min/week in 2010 and 2015, respectively). Overall, older adults living in highly walkable areas showed a smaller reduction than those in low walkable areas (beta: 99.7 min/week, 95% confidence interval: 28.5–171.0). Similar associations were observed among those in the urban and suburban area, but not in the rural area. Walkability had larger effect sizes for explaining MVPA change than demographic characteristics. In addition, the findings for walking were similar to MVPA. Neighborhood walkability mitigated the 5-year reduction of walking and total MVPA among older adults, especially in urban areas.