Medical images and chest X-rays, in particular, are primarily the most widely used radiological tests in clinical practice for diagnosis and treatment. Reading and interpreting a chest x-ray can be time-consuming for an experienced radiologist, more difficult for the less experienced, and almost impossible for an average person. An X-ray computer-assisted reporting system can ease the physician’s reporting task and provide decision support for radiologists. Also, it will accelerate the deployment of computer-assisted medical decision-making systems. This paper proposes an automatic chest x-ray generating report framework based on combining the transfer learning technique and the transformer approach to generate reports. We apply a pretrained convolutional neural network model from the ImageNet database for feature extraction to exploit deep neural networks’ power. Then, we use a modified transformer encoder-decoder. We test our network on Open I, Indiana University’s Chest X-ray Collection (IU X-Ray), one of the most comprehensive public data sets currently used for chest X-ray captioning. We show that our model produced promising results. Our model performed better than the state-of-the-art results in terms of performance.