Visual classification of pollen grains is crucial for various agricultural applications, particularly for the protection, monitoring, and tracking of flora to preserve the biome and maintain the quality of honey-based products. Traditionally, pollen grain classification has been performed by trained palynologists using a light microscope. Despite their wide range of applications, still tiresome and time-consuming methods are used. Artificial intelligence (AI) can be used to automate the pollen grain classification process. Recently, numerous AI-based techniques for classifying pollen grains have been proposed. However, there is still possibility for performance enhancement including processing time, memory size, and accuracy. In this study, an attention-guided pollen feature aggregation network (APFA-Net) based on deep feature aggregation and channel-wise attention is proposed. Three publicly available datasets, POLLEN73S, POLLEN23E, and Cretan pollen, having a total of 7362 images from 116 distinct pollen types are used for experiments. The proposed method shows F-measure values of 97.37 %, 97.66 %, and 98.39 % with POLLEN73S, POLLEN23E, and Cretan Pollen datasets, respectively. We confirm that our method outperforms existing state-of-the-art methods.