In this paper, we present a novel algorithm called manifold ordinal regression (MOR) for image ranking. By modeling the manifold information in the objective function, MOR is capable of uncovering the intrinsically nonlinear structure held by the image data sets. By optimizing the ranking information of the training data sets, the proposed algorithm provides faithful rating to the new coming images. To offer more general solution for the real-word tasks, we further provide the semi-supervised manifold ordinal regression (SS-MOR). Experiments on various data sets validate the effectiveness of the proposed algorithms.