Arrhythmia detection based on ECG classification has been a hot topic in the health informatics community, where each heartbeat is assigned to one of five classes: N, S, V, F and Q. However, arrhythmia detection under the inter-patient paradigm remains a challenging task. In particular, the detection of the S class is especially hard as it is morphologically similar to the N class. Recently, an LSTM-based sequence-to-sequence (seq2seq) model with CNN-based embedding has achieved the state-of the-art (SOTA) performance (as far as we know) by capturing both intra- and inter-heartbeat information, the latter of which can be crucial to S detection. However, its performance is still limited as it lacks the ability to highlight more discriminative features. In this work, based on the current SOTA, we propose a seq2seq model with a novel two-level attentional structure for accurate inter-patient arrhythmia detection. Specifically, a local channel-wise attention is used to weight intra-heartbeat morphological features, while a contextual Bahdanau's attention is used to weight inter-heartbeat semantics. A combination of the two attentions leads to a model that can utilize highly discriminative features and strike a balance between intra-and inter-heartbeat information. So far as we are concerned, our method has the best overall performance in the literature. Our source code is available at https://github.com/hierarchyJK/ Two-level-Attention-for-Arrhythmia-Detection.