Cement-stabilized soil retaining wall structures are becoming increasingly popular for high-speed railway lines. However, the widely used Portland cement is heavily associated with carbon dioxide (CO₂) emissions. Additionally, the rising cost and scarcity of sand traditionally used as fill material is becoming a concern. Besides, the geomechanical performances of the cement-stabilized soil retaining walls using three-dimensional (3-D) finite element method (FEM) numerical analysis have seldom been investigated. To address these issues, this study investigates the geomechanical performances of geocell-reinforced soil (GRS) backfilled with magnesia-based cement-stabilized marine fill using the nonlinear stress-strain relationship and 3-D FEM models as an alternative to the traditional earth GRS walls. The study finds satisfactory results regarding stability, deformation, and settlement. Only nominal facing reinforcements may be required as the magnesia cement mechanically stabilizes the fill. Moreover, the strength and stiffness properties improve with increased curing duration. The safety factors and observed displacements were within acceptable limits. However, the critical region for the cement-stabilized marine fill GRS wall showing a rupture zone was identified at the toe of the structure. Finally, the study developed a load-displacement model for settlement predictions and uncertainty interpretations, and explained the strength development and deformation mechanisms. The results suggest that cement-stabilized soil retaining walls using magnesia-based cement could be a viable alternative to traditional earth GRS walls, while mitigating environmental concerns and reducing costs.