In this work we present a human pose estimation method based on the skeleton fusion and tracking using multiple RGB-D sensors. The proposed method considers the skeletons provided by each RGB-D device and constructs an improved skeleton, taking into account the quality measures provided by the sensors at two different levels: the whole skeleton and each joint individually. Then, each joint is tracked by a Kalman filter, resulting in a smooth tracking performance. We have also developed a new dataset consisting of six subjects performing seven different gestures, recorded with four Kinect devices simultaneously. Experimental results performed on this dataset show that the system obtains better smoothness results than the most representative methods found in the literature. The proposed system operates at a processing rate of 25 frames per second (including the whole algorithm loop, i.e., data acquisition and processing) without the explicit use of the multithreading capabilities of the system.