In recent years, convolutional neural networks (CNNs) have found many applications in medical image analysis. Having enough labeled data, CNNs could be trained to learn image features and used for object localization, classification, and segmentation. Although there are many interests in building and improving automated systems for medical image analysis, lack of reliable and publicly available biomedical datasets makes such a task difficult. In this work, the effectiveness of CNNs for the classification of breast lesions in ultrasound (US) images will be studied. First, due to a limited number of training data, we use a custom-built CNN with a few hidden layers and apply regularization techniques to improve the performance. Second, we use transfer learning and adapt some pre-trained models for our dataset. The dataset used in this work consists of a limited number of cases, 641 in total, histopathologically categorized (413 benign and 228 malignant lesions). To assess how the results of our classifier generalize on our data set, a 5-fold cross-validation were employed, where in each fold 80% of data were used for training and the 20% for testing. Accuracy and the area under the ROC curve (AUC) were used as the main performance metrics. Before applying any regularizations techniques, we achieved an overall accuracy of 85.98% for tumor classification, and the AUC equal to 0.94. After applying image augmentation and regularization, the accuracy and the AUC increased to 92.05% and 0.97, respectively. Using a pre-trained model, we achieved an overall accuracy of 87.07% and an AUC equal to 0.96. The obtained results demonstrated the effectiveness of our custom architecture for classification of tumors in this small US imaging dataset, surpassing some traditional learning algorithm based on manual feature selection.