Magnetic localization is an appropriate method for tracing an intra-body capsule object because of its satisfactory accuracy and efficiency. In this method, the capsule is enclosed in a ring magnet, which establishes a magnetic field around the human body. By using a sensor array system with a number of triaxial magnetic sensors, the magnetic flux densities can be measured, and the magnet can be localized by an appropriate algorithm. However, a problem for such a system is that the movements of the human body have interferences on the localization. Therefore, in order to compensate the interferences, we propose a three-magnet localization method. Here, in addition to the capsule object magnet, two other magnets are fixed on the surface of the human body to serve as reference objects. The position and orientation parameters of all the three magnets are determined by applying the optimal algorithm on the sensing data from the sensor array. Then, a reference coordinate system is built based on the two reference objects, and the capsule magnet is relatively tracked with respect to this reference system in human body. The experimental results show that the average localization error caused by the body movement interference is reduced from 30.1 mm (in the original coordinate system) to 3.82 mm (in reference coordinate system) and the average direction error reduced from 17.7° to 2.2°.