Abstraction-Based Controller Synthesis (ABCS) is an emerging field for automatic synthesis of correct-by-design controllers for non-linear dynamical systems in the presence of bounded disturbances. A major drawback of existing ABCS techniques is the lack of flexibility against changes in the disturbance model; any change in the model results in a complete re-computation of the abstraction and the controller. This flexibility is relevant to situations when disturbances are learned or estimated during operation in an environment which is previously not known precisely. As time passes, the disturbance model is progressively refined. The monolithic nature and high computational cost of existing algorithms make ABCS unsuited for such scenarios.In this paper, we present an incremental algorithm to locally adapt abstractions to changes in the disturbance model. Only the parts of the space which are affected by the changes are updated and the rest of the abstraction is reused. Our new abstraction method allows to apply existing incremental techniques to update the discrete controller locally for the changed abstraction. This results in an incremental ABCS algorithm. We empirically show the benefit of dynamic abstraction adaptation on two large examples: a 5-dimensional vehicle model and a 12-dimensional quadrotor model. In both cases, the speed-up over complete re-computation is significant.