Due to inherent irregularities in recyclable materials, sorting valuable metals (e.g., aluminum and copper) via mechanical means is a difficult task resisting full automation. A particularly hard challenge in the domain is the separation of scrap metal pieces with physically attached impurities, which is further complicated by variations in different batches of recyclable materials. In this work, leveraging the latest development in machine learning and robot learning, we develop an image-based sorting system for tackling this challenging task. In addition to delivering a highly accurate deep learning model for reliably distinguishing pure scrap pieces from pieces containing impurities with over 95% precision/recall, we further automate the process of sample preparation, data acquisition/labeling/analysis, and machine learning model training.