The presence and configurations of defects are primary components determining materials functionality. Their population and distribution are often nonergodic and dependent on synthesis history, and therefore rarely amenable to direct theoretical prediction. Here, dynamic electron beam–induced transformations in Si deposited on a graphene monolayer are used to create libraries of possible Si and carbon vacancy defects. Deep learning networks are developed for automated image analysis and recognition of the defects, creating a library of (meta) stable defect configurations. Density functional theory is used to estimate atomically resolved scanning tunneling microscopy (STM) signatures of the classified defects from the created library, allowing identification of several defect types across imaging platforms. This approach allows automatic creation of defect libraries in solids, exploring the metastable configurations always present in real materials, and correlative studies with other atomically resolved techniques, providing comprehensive insight into defect functionalities.