Deep learning has been widely used in automatic target recognition (ATR) for synthetic aperture radar (SAR) recently. However, most of the studies are based on the network structure in optical images and lack full consideration of the inherent characteristics of SAR targets, which limits the improvement of recognition accuracy and makes poor generalization ability. In addition, due to the black-box characteristics, it is difficult to effectively interpret SAR ATR results. To conquer these problems, we propose an electromagnetic scattering feature (ESF) module embedded network based on attributed scattering center (ASC) model to incorporate the SAR targets’ characteristics into the deep learning framework. First, a novel convolutional neural network (CNN)-based algorithm for extracting ASC parameters is proposed, which makes the network focus on target features under the guidance of physical model. Then, the ESF module is designed based on a well-trained ASC parameters extractor to inject the learned target features into the classification network for more robust and interpretable results. Besides, two structures are proposed combined with the ESF module for single-view and multiview SAR target classification, which further illustrates the portability of the module. Experiments on the Moving and Stationary Target Acquisition and Recognition (MSTAR) dataset show the validity of the proposed CNN-based ASC extractor and the ESF module embedded classification network. Compared with ordinary networks, our method can achieve higher classification accuracy under complex conditions, which reflects the better generalization performance of the algorithm. Furthermore, through visualization analysis of the classification results, we show the interpretability of the network combined with the electromagnetic scattering characteristics.