The area of interpretable deep neural networks has received increased attention in recent years due to the need for transparency in various fields, including medicine, healthcare, stock market analysis, compliance with legislation, and law. Layer-wise Relevance Propagation (LRP) and Gradient-weighted Class Activation Mapping (Grad-CAM) are two widely used algorithms to interpret deep neural networks. In this work, we investigated the applicability of these two algorithms in the sensitive application area of interpreting chest radiography images. In order to get a more nuanced and balanced outcome, we use a multi-label classification-based dataset and analyze the model prediction by visualizing the outcome of LRP and Grad-CAM on the chest radiography images. The results show that LRP provides more granular heatmaps than Grad-CAM when applied to the CheXpert dataset classification model. We posit that this is due to the inherent construction difference of these algorithms (LRP is layer-wise accumulation, whereas Grad-CAM focuses primarily on the final sections in the model's architecture). Both can be useful for understanding the classification from a micro or macro level to get a superior and interpretable clinical decision support system.