Accurate load forecasting can maintain the safety and stability of power grids. The mainstream models are based on complex recurrent or convolutional neural networks (RNNs, CNNs), and in recent years they are often used in combination with the attention mechanism. The shortcomings of these models are that they cannot get rid of sequential computation and fail to capture long-term dependence. For better application in day-ahead load forecasting (DALF), we propose a new network architecture, i.e., Forwardformer, which is implemented by imposing some effective improvements on the Transformer (a pioneering network model with applications in Natural Language Processing (NLP)). The core of Forwardformer is the multi-scale forward self-attention (MSFSA) and the correction structure of the encoder-dual decoder, which confer better computational efficiency and forecasting accuracy. Meanwhile, to improve forecasting accuracy on special days (weekends, holidays, etc.), the MSFSA configures dilated attention and global attention for them, respectively. Experiments performed on datasets from China and America demonstrated that the Forwardformer requires less runtime while being superior in forecasting accuracy. Especially in terms of weekends and holidays, it has outstanding advantages and provides a new solution to the DALF problem.