As the world is still trying to rebuild from the destruction caused by the widespread reach of the COVID-19 virus, and the recent alarming surge of human monkeypox outbreaks in numerous countries threatens to become a new global pandemic too. Human monkeypox disease symptoms are quite similar to chickenpox, and measles classic symptoms, with very intricate differences such as skin blisters, which come in diverse forms. Various deep-learning methods have shown promising performance in the image-based diagnosis of Covid-19, tumor cell, and skin disease classification tasks. In this paper, we try to integrate deep transfer learning-based methods, along with a convolutional block attention module (CBAM) to focus on the relevant portion of feature maps to conduct an image-based classification of human monkeypox disease. We implement five deep learning models—VGG19, Xception, DenseNet121, EfficientNetB3, and MobileNetV2 along with integrated channel and spatial attention mechanisms and perform a comparative analysis among them. An architecture consisting of Xception-CBAM-Dense layers performed better than the other models at classifying monkeypox and other diseases with a validation accuracy of 83.89%.