Cancers represent complex ecosystems comprising tumor cells and a multitude of non-cancerous cells, embedded in an altered extracellular matrix. The tumor microenvironment (TME) includes diverse immune cell types, cancer-associated fibroblasts, endothelial cells, pericytes, and various additional tissue-resident cell types. These host cells were once considered bystanders of tumorigenesis but are now known to play critical roles in the pathogenesis of cancer. The cellular composition and functional state of the TME can differ extensively depending on the organ in which the tumor arises, the intrinsic features of cancer cells, the tumor stage, and patient characteristics. Here, we review the importance of the TME in each stage of cancer progression, from tumor initiation, progression, invasion, and intravasation to metastatic dissemination and outgrowth. Understanding the complex interplay between tumor cell-intrinsic, cell-extrinsic, and systemic mediators of disease progression is critical for the rational development of effective anti-cancer treatments.