Patients with inflammatory bowel disease (IBD) have increased incidence of colorectal cancer (CRC). IBD-associated cancer follows a well-characterized sequence of intestinal epithelial changes, in which genetic mutations and molecular aberrations play a key role. IBD-associated cancer develops against a background of chronic inflammation and pro-inflammatory immune cells, and their products contribute to cancer development and progression. In recent years, the effect of the immunosuppressive microenvironment in cancer development and progression has gained more attention, mainly because of the unprecedented anti-tumor effects of immune checkpoint inhibitors in selected groups of patients. Even though IBD-associated cancer develops in the background of chronic inflammation which is associated with activation of endogenous anti-inflammatory or suppressive mechanisms, the potential role of an immunosuppressive microenvironment in these cancers is largely unknown. In this review, we outline the role of the immune system in promoting cancer development in chronic inflammatory diseases such as IBD, with a specific focus on the anti-inflammatory mechanisms and suppressive immune cells that may play a role in IBD-associated tumorigenesis.