Artificial intelligence-based medical image analysis promises an efficient and reliable diagnosis in today's healthcare. Traditional approaches for cataract screening by medical practitioners often results in subjectivity due to their varying levels of knowledge and expertise. Using transfer learning, ensembles of pre-trained convolutional neural networks, and stacked long short-term memory networks, we developed a non-invasive and streamlined pipeline for automatic cataract severity classification. Empirical results show that our proposed combined models of AlexNet, InceptionV3, Xception, and InceptionResNetV2 using a weighted average algorithm produces 99.20% (normal vs. cataract) and 97.76% (normal to severe) accuracies compared to standalone models. Furthermore, the ensemble model reduces classification error rates by an average of 2.17%. This study has the potential to help doctors to specify the magnitude of cataract stages with highly acceptable precision.