Both genetic and environmental factors influence the etiology of age-related macular degeneration (AMD), a leading cause of blindness. AMD severity is primarily measured by images of the fundus of the retina and recently developed machine learning methods can successfully predict AMD progression using image data. However, none of these methods have used both genetic and image data for predicting AMD progression. Here we used both genotypes and fundus images to predict whether an eye had progressed to late AMD with a modified deep convolutional neural network. In total, we used 31,262 fundus images and 52 AMD-associated genetic variants from 1,351 subjects from the Age-Related Eye Disease Study, which provided disease severity phenotypes and fundus images available at baseline and follow-up visits over a period of 12 years. Our results showed that fundus images coupled with genotypes could predict late AMD progression with an averaged area-under-the-curve value of 0.85 (95% confidence interval 0.83–0.86). The results using fundus images alone showed an averaged area under the receiver operating characteristic curve value of 0.81 (95% confidence interval 0.80–0.83). We implemented our model in a cloud-based application for individual risk assessment.