P-cresyl sulfate and indoxyl sulfate are strongly associated with cardiovascular events and all-cause mortality in chronic kidney disease (CKD). This randomized controlled trial was conducted to compare the effects between sevelamer and calcium carbonate on protein-bound uremic toxins in pre-dialysis CKD patients with hyperphosphatemia. Forty pre-dialysis CKD patients with persistent hyperphosphatemia were randomly assigned to receive either 2400 mg of sevelamer daily or 1500 mg of calcium carbonate daily for 24 weeks. A significant decrease of total serum p-cresyl sulfate was observed in sevelamer therapy compared to calcium carbonate therapy (mean difference between two groups −5.61 mg/L; 95% CI −11.01 to −0.27 mg/L; p = 0.04). There was no significant difference in serum indoxyl sulfate levels (p = 0.36). Sevelamer had effects in terms of lowering fibroblast growth factor 23 (p = 0.01) and low-density lipoprotein cholesterol levels (p = 0.04). Sevelamer showed benefits in terms of retarding CKD progression. Changes in vascular stiffness were not found in this study.