Rotors of permanent magnet synchronous motors (PMSM) used in hybrid electric vehicles are an electromechanical-coupled dynamic system. Magnetic field-induced mechanical vibration has an important effect on the performance of these high-speed and high-power-density PMSMs. In this paper, the model of an unbalanced magnetic pull (UMP) resulting from a non-uniform magnetic field is investigated theoretically and numerically. Equations of motion of an unbalanced Jeffcott rotor are established. Approximate solution to the equation of nonlinear vibration under UMP is obtained using the averaging method, and stability of the steady response is discussed using eigenvalue analysis. Nonlinear phenomenon, which is an effect of the electromagnetic stiffness coefficient, mass imbalance parameter and effective damping on overall responses, was studied in detail. It is demonstrated that response curves manifest soft characteristics, with a jump phenomenon and unstable areas pointed out. The response obtained using analytic method was compared with the numerically results. Conclusions from this work can be adopted to identify instability locus in rotors with mechanical and magnetic coupling effects taken into consideration. In addition, outcomes from this work provide theoretical and practical ideas to control the systems and optimize their operation.