Smoke removal has been a challenging problem which refers to the process of retrieving clear images from the smoky ones. It has an extensive demand while the performances of existing image desmoking methods have limitations owing to the homogeneous medium based model, hand-crafted features, and insufficient datasets. Unlike the existing desmoking methods using an atmospheric scattering model, the proposed method utilizes a general deblurring model. Inspired by Generative Adversarial Network (GAN), we propose an end-to-end attentive DesmokeGAN which implements the visual attention into the generative network to effectively learn the smoke features and their surroundings. The architecture of critic network is identical to PatchGAN by adding multi-component loss function to the image patch. Due to the scarce of various and quality smoke datasets, a graphics rendering engine is used to synthesize the smoky images. The quantitative and qualitative results show that the designed framework performs better than the recent state-of-the-art desmoking approaches on both synthetic and real images in indoor and outdoor scenes, especially for the thick smoky images.