This paper investigates the interface property between rubber asphalt and aggregates using molecular dynamics simulations to address the adhesion and debonding properties. The effect of aggregate types and rubber contents on rubber asphalt-aggregate systems' adhesion and debonding properties were investigated. The work of adhesion characterized the adhesion property while debonding characteristics were evaluated by the work of debonding and energy ratio (ER). The results show that the impact of aggregate types on the adhesion and debonding between rubber asphalt and aggregates is much more significant than rubber contents. The work of adhesion between four aggregates and rubber asphalt is evaluated as follows: microcline > albite > quartz > calcite. The reason is that the electrostatic energy can considerably contribute to the adhesion at the interface for strong alkali aggregates (microcline, albite) and rubber asphalt, while the van der Waals energy can only contribute nearly 100 mJ/m² energy for the adhesion. The work of debonding for the four aggregates with rubber asphalt follows the order: microcline > quartz > albite > calcite. Therefore, microcline presents extraordinary performance, while calcite shows the poorest response in moisture resistance.