Registration that aligns different views of one interested organ together is an essential technique and outstanding problem in medical robotics and image-guided surgery (IGS). This work introduces a novel rigid point set registration (PSR) approach that aims to accurately map the pre-operative space with the intra-operative space to enable successful image guidance for computer-assisted orthopaedic surgery (CAOS). The normal vectors and tangent vectors are first extracted from the pre-operative and intra-operative point sets (PSs) respectively, and are further utilized to enhance the registration accuracy and robustness. The contributions of this article are three-folds. First, we propose and formulate a novel distribution that describes the error between one normal vector and the corresponding tangent vector based on the von-Mises Fisher (vMF) distribution. Second, by modelling the anisotropic position localization error with the multi-variate Gaussian distribution, we formulate the PSR considering anisotropic localization error as a maximum likelihood estimation (MLE) problem and then solve it under the expectation maximization (EM) framework. Third, to facilitate the optimization process, the gradients of the objective function with respect to the desired parameters are computed and presented. Extensive experimental results on the human femur and pelvis models verify that the proposed approach outperforms the state-of-the-art methods, and demonstrate potential clinical values for relevant surgical navigation applications.