This paper presents a scheme for building detection and reconstruction by merging LIDAR data and aerial imagery. In the building detection part, a region-based segmentation and object-based classification are integrated. In the building reconstruction, we analyze the coplanarity of the LIDAR point clouds to shape roofs. The accurate positions of the building walls are then determined by integrating the edges extracted from aerial imagery and the plane derived from LIDAR point clouds. The three dimensional building edges are thus used to reconstruct the building models. In the reconstruction, a patented method SMS (Split-Merge-Shape) is incorporated. Having the advantages of high reliability and flexibility, the SMS method provides stable solution even when those 3D building lines are broken. LIDAR data acquired by Leica ALS 40 and aerial images were used in the validation. Experimental results indicate that the successful rate for building detecition is higher that 81%. The positioning for buildings may reach sub-meter accuracy.