This study aimed to assess associations between serum cartilage oligomeric matrix protein (sCOMP) and phenotypic characteristics in late-stage hip and knee Osteoarthritis (OA) as well as its correlation with further serum markers of possible comorbidities in the Ulm Osteoarthritis Study. Moreover, the prognostic relevance of preoperative sCOMP concentrations for short-term functionality and pain outcomes after hip or knee joint replacement was explored. Preoperative serum samples and detailed information about the health status (i.e., WOMAC scores, Hannover Functionality Status (FFbH)) of 754 OA patients undergoing total joint replacement were included. Spearman rank-correlation coefficients and multiple linear regression models were used to evaluate the relationships between sCOMP, other serum markers, and health outcomes. There was a significant positive association between sCOMP and markers of renal (cystatin C, creatinine, and eGFR) and cardiac (e.g., NT-proBNP) impairment. Since renal failure might cause accumulation of sCOMP, additional adjustment with eGFR was performed. Preoperative sCOMP levels in knee OA but not hip OA patients were positively associated with FFbH, WOMAC function sub-scale and total WOMAC scale as well as the post-operative WOMAC stiffness sub-scale six months after surgery. Our data clearly demonstrate an association between sCOMP and renal function as well as other confounding factors, which should be considered in future biomarker studies.