Recently, human activity recognition (HAR) that uses wearable sensors has become a research hotspot due to its wide applications in a large variety of real-world scenarios including fitness, health-care, and sports tracking. In essence, HAR can be treated as multi-channel time series classification problem, where different channels may come from heterogeneous sensor modalities. Deep learning, especially convolutional neural networks (CNNs) have made major breakthroughs in ubiquitous HAR computing scenario. Various normalization methods have played an indispensable role in prior HAR works, which enable every layer of the network to do learning more independently by normalizing hybrid sensor features. However, normalization tends to produce a ‘channel collapse’ phenomenon, where a large fraction of channels only generates very small values. Most channels are inhibited and contribute very little to activity recognition. As a result, the network has to rely on only a few valid channels, which inevitably impair the generality ability of a network. In this paper, we provide an alternative called Channel Equalization to reactivate these inhibited channels by performing whitening or decorrelation operation, which compels all channels to contribute more or less to feature representation. Experiments conducted on several benchmarks including UCI-HAR, OPPORTUNITY, UniMiB-SHAR, WISDM, PAMAP2, and USC-HAD show that the proposed Channel Equalization module is an impressive alternative of convolution layers, and achieve higher recognition performance to baseline models with simliar computational cost, which significantly surpasses recent state-of-the-arts for activity recongition. To the best of our knowledge, the Channel Equalization is for the first time to be applied in multi-modal HAR scenario. Finally, the actual operation is evaluated on an embedded Raspberry Pi Model 3 B plus platform.