Objective To investigate the value of differential diagnosis of benign and malignant breast masses based on transfer learning model of ultrasound images. Methods A total of 582 ultrasound images from 300 patients with breast masses confirmed by surgery or biopsy in our hospital from May 2018 to March 2021 were collected as an ultrasound data set (482 for training set and 100 for test set). Three deep convolutional neural network models (VGG-16, Inception-v3, ResNet-50) pre-trained on the ImageNet dataset are trained and tested using a transfer learning method. For the first transfer learning, three models were used to identify and learn the X-ray images of benign and malignant breast masses in the public database CBIS-DDSM (Curated Breast Imaging Subset of DDSM) dataset and fine-tune the models. For the second transfer learning, three models using a randomly selected training set of ultrasound images from the ultrasound dataset were fine-tuned. The diagnostic performance of three models using different times of transfer learning for benign and malignant breast masses was compared. Results All evaluation indicators of VGG-16, Inception-v3 and ResNet50 models were improved after transfer learning, and the model established based on ResNet50 had a better performance on the identification of benign and malignant breast masses - accuracy was 88.0%, sensitivity was 82.7%, specificity was 93.8%，and AUC was 0.915. Conclusion The ResNet50 transfer learning model based on ultrasound images has a high accuracy in the differential diagnosis of benign and malignant breast masses and can provide decision support for primary physicians in accurate diagnosis.