Deep learning has shown great potentials for medical image analysis and computer-aided diagnosis of some diseases such as MRI brain tumor segmentation, mammogram classification, and diabetic macular edema classification. In this paper, we explore deep learning approaches for ovarian tumor classification based on ultrasound images. First, considering the lack of public ultrasound images, we annotate an ultrasound image dataset consisting of 988 image samples of three types of ovarian tumors. Second, we evaluate the generalization ability of different convolutional neural network (CNN) models on ultrasound images. Our experiments show that deep learning approaches achieve considerably high accuracies on the classification of ovarian tumors which are competitive with professional medical staffs.