Purpose To verify the effectiveness of deep learning in ultrasonic diagnosis, we construct the ultrasound image dataset of the carotid and test 15 different classification models for carotid plaque diagnosis. Methods Firstly, 254 patients and 354 normal persons" carotid ultrasound images were collected to construct a carotid ultrasound image dataset. Two images were collected for each case, so the constructed dataset contains 1216 images in total. Then, traditional method HOG+SVM and 14 different deep neural network models were trained based on our proposed dataset. Finally, the best network model is determined based on three evaluation indexes: precision, recall and F1 score. Results By comparing 15 different ultrasonic image classification methods for carotid plaque diagnosis, it is concluded that the depth residual network model ResNet50 has the best performance, and its accuracy, recall and F1 values are 97.36%, 97.32% and 97.34%, respectively. Conclusion ResNet50 can classify and recognize carotid plaque images with the highest accuracy, which verifies the effectiveness of deep learning in the application of automatic diagnosis by using ultrasound images.