Solid tumor growth and metastasis require the interaction of tumor cells with the surrounding tissue, leading to a view of tumors as tissue-level phenomena rather than exclusively cell-intrinsic anomalies. Due to the ubiquitous nature of adipose tissue, many types of solid tumors grow in proximate or direct contact with adipocytes and adipose-associated stromal and vascular components, such as fibroblasts and other connective tissue cells, stem and progenitor cells, endothelial cells, innate and adaptive immune cells, and extracellular signaling and matrix components. Excess adiposity in obesity both increases risk of cancer development and negatively influences prognosis in several cancer types, in part due to interaction with adipose tissue cell populations. Herein, we review the cellular and noncellular constituents of the adipose “organ,” and discuss the mechanisms by which these varied microenvironmental components contribute to tumor development, with special emphasis on obesity. Due to the prevalence of breast and prostate cancers in the United States, their close anatomical proximity to adipose tissue depots, and their complex epidemiologic associations with obesity, we particularly highlight research addressing the contribution of adipose tissue to the initiation and progression of these cancer types. Obesity dramatically modifies the adipose tissue microenvironment in numerous ways, including induction of fibrosis and angiogenesis, increased stem cell abundance, and expansion of proinflammatory immune cells. As many of these changes also resemble shifts observed within the tumor microenvironment, proximity to adipose tissue may present a hospitable environment to developing tumors, providing a critical link between adiposity and tumorigenesis.