The well-known cooperative stereo uses two dimensional rectangular window for a local block matching, and three dimensional box-shaped volume for a global optimization procedure. In many cases, appropriate selections of these matching regions can provide satisfactory matching results. This paper presents a new method for iteratively modifying sizes and shapes of matching regions based on color and depth information. This algorithm computes the aggregated matching costs with two ideas. The first idea is to select matching regions based on object boundaries to avoid projective distortion. This provides the reliable matching scores as well as the prevention of the foreground fattening phenomenon. The second idea is to iteratively modify the segmentation map by merging the regions where the disparities are likely to be the same. Experimental results show that the proposed algorithm provides more accurate disparity map than other algorithms. Especially, the computed disparity map shows the advantage of our algorithm in disparity discontinuity regions.