The arrhythmia is an important group of cardiovascular disease. Electrocardiogram (ECG) is commonly used for detecting arrhythmias. Computer-aided diagnosis system can diagnose ECG automatically without the limitations of visual inspection. In order to improve the performance of ECG heartbeat classification, this paper proposes a novel automatic classification system. Based on convolutional neural network (CNN) and long short-term memory (LSTM) network, a deep structure with multiple input layers is proposed. Four input layers are constructed based on different regions of a heartbeat and RR interval features. The first three inputs are convolved using different strides. The three outputs of CNN are then concatenated and go through an LSTM network. Two fully-connected layers follow and the output is concatenated with the fourth input. Eventually, the last fully-connected layer outputs the predicted label. The proposed system was evaluated by two division schemes of the MIT-BIH arrhythmia database. Class-oriented scheme achieved an overall accuracy of 99.26% and subject-oriented scheme obtained an accuracy of 94.20%. The comparison with previous works showed the excellent performance of the novel network. The combination of automatic features and handcraft features was demonstrated to be helpful in heartbeat classification. Hence, the system can be used for clinical application.