Nanoparticle based delivery formulations have become a leading delivery strategy for cancer imaging and therapy. The success of nanoparticle-based therapy relies heavily on their ability to utilize the enhanced permeability and retention (EPR) effect and active targeting moieties to their advantage. However, these methods often fail to enable a uniform NP distribution across the tumor, and lead to insufficient local concentrations of drug. Oftentimes, this heterogeneous drug distribution is one of the primary reasons for suboptimal treatment efficacy in NP delivery platforms. Herein, we seek to examine the biophysical causes of heterogeneous NP distribution in stroma-rich desmoplastic tumors; namely the abnormal tumor vasculature, deregulated extracellular matrix and high interstitial hypertension associated with these tumors. It is suggested that these factors help explain the discrepancy between promising outlooks for many NP formulations in preclinical studies, but suboptimal clinical outcomes for most FDA approved nanoformulations. Furthermore, examination into the role of the physicochemical properties of NPs on successful drug delivery was conducted in this review. In light of the many formidable barriers against successful NP drug delivery, we provided possible approaches to mitigate delivery issues from the perspective of stromal remodeling and NP design. In all, this review seeks to provide guidelines for optimizing nanoparticle-based cancer drug delivery through both modified nanoparticle design and alleviation of biological barriers to successful therapy.