Online portfolio selection is a fundamental research problem, which has drawn extensive investigations in both machine learning and computational finance communities. The evolution of electronic trading has contributed to the growing prevalence of High-Frequency Trading (HFT) in recent years. Generally, HFT requires trading strategies to be fast in execution. However, the existing online portfolio selection strategies fail to either satisfy the demand for high execution speed or make effective utilization of historical data. In response, we propose a framework named Exponential Gradient with Momentum (EGM) which integrates EG with an acknowledged optimization method in stochastic learning, i.e., momentum. Specifically, momentum boosts the performance of EG by making full use of historical information. Most essentially, EGM can execute with only constant memory and running time in the number of assets per trading period, thus overcoming the drawback of most online strategies. The theoretical analysis reveals that EGM bounds the regret sublinearly. The extensive experiments conducted on four real-world datasets demonstrate that EGM outperforms relevant strategies with respect to comprehensive evaluation metrics.