Optimizing the cutter changing process for tunnel boring machines (TBMs) is crucial for minimizing maintenance costs and maximizing excavation efficiency. This paper introduces TunnRL-CC, a computational framework that utilizes reinforcement learning to autonomously determine cutter-changing strategies. TunnRL-CC's realistic simulation models cutter wear under varying rock conditions, including hard rock and blockyness. A reinforcement learning agent is trained to learn optimal cutter-changing policies based on a reward function that balances cutter conditions and operational costs. The agent demonstrates innovative decision-making, adapting to changing excavation conditions. TunnRL-CC's proposed methodology significantly differs from traditional cutter changing practices, which rely heavily on operator experience. Although TunnRL-CC has not been applied in practical projects, its theoretical basis and comprehensive computational experiments demonstrate its capability to significantly improve TBM cutter maintenance procedures.