While cancer research has been focused on tumor cells for many years, evidence is growing that the tumor stroma and in particular cancer-associated fibroblasts (CAFs) in particular play essential roles in the progression of human malignant disease. In human lung cancer, CAFs expressing the transmembrane protein podoplanin were shown to have significant influence on the patients’ prognosis. In this study, we investigated the presence and prognostic role of podoplanin-expressing CAFs in a large series of patients with invasive breast cancer. Podoplanin expression was evaluated immunohistochemically in 367 breast cancers. Tumors with ≥10% distinct podoplanin staining were considered as positive (CAF+). Cytoplasmic podoplanin expression of tumor cells was considered as positive when ≥5% of tumor cells showed a distinct podoplanin expression. In normal breast tissue, no podoplanin-expressing fibrocytes were found. Thirty-three patients (9%) with breast cancer showed podoplanin expression in CAFs. In 28 patients (8%), a podoplanin expression in tumor cells was observed. A strong negative correlation of CAF+ with estrogen receptor status (p < 0.001), and a significant association with higher histological grading (p < 0.001) was seen. In multivariable analysis, CAFs+ was an independent prognostic factor for disease free (1.78 Hazard ratio; p = 0.026) and overall survival (2.304 Hazard ratio; p = 0.002) in patients with breast cancer. Podoplanin-expressing CAFs contribute to the prognosis of invasive breast cancer, indicating a highly aggressive subgroup. CAFs may present a highly selective target for anti-cancer therapies in patients with invasive breast cancer.