How genome divergence eventually leads to speciation is a topic of prime evolutionary interest. Genomic islands of elevated divergence are frequently reported between diverging lineages, and their size is expected to increase with time and gene flow under the speciation-with-gene-flow model. However, such islands can also result from divergent sorting of ancient polymorphisms, recent ecological selection regardless of gene flow, and/or recurrent background selection and selective sweeps in low-recombination regions. It is challenging to disentangle these nonexclusive alternatives, but here we attempt to do this in an analysis of what drove genomic divergence between four lineages comprising a species complex of desert poplar trees. Within this complex we found that two morphologically delimited species, Populus euphratica and Populus pruinosa, were paraphyletic while the four lineages exhibited contrasting levels of gene flow and divergence times, providing a good system for testing hypotheses on the origin of divergence islands. We show that the size and number of genomic islands that distinguish lineages are not associated with either rate of recent gene flow or time of divergence. Instead, they are most likely derived from divergent sorting of ancient polymorphisms and divergence hitchhiking. We found that highly diverged genes under lineage-specific selection and putatively involved in ecological and morphological divergence occur both within and outside these islands. Our results highlight the need to incorporate demography, absolute divergence measurement, and gene flow rate to explain the formation of genomic islands and to identify potential genomic regions involved in speciation.