Monogamy within social groups where there exists a high potential for polygyny poses a challenge to our understanding of mating system evolution. Specifically, the traditional explanation that monogamy evolves due to wide female dispersion, affording males little opportunity to defend multiple females, cannot apply. Instead, monogamy in groups potentially arises because females compete for breeding resources such as breeding sites, food, and paternal care. We conducted manipulative experiments to determine whether females compete over limiting resources within groups of the obligate coral-dwelling goby, Paragobiodon xanthosomus (Gobiidae). Breeding females behaved aggressively toward individuals of their own sex and evicted subordinate females that were large and mature from the group. Experimental removal of nest sites caused breeding partners to breed in alternative nest sites, demonstrating that nest site limitation was not the cause of female competition. Supplemental feeding resulted in an increase in the fecundity of breeding females but no maturation of subordinate females, demonstrating that food-limited female fecundity was a likely cause of female competition. Finally, supplemental feeding of breeding pairs demonstrated that the difference in eggs hatched by fed versus unfed males was less than the difference in eggs laid by fed versus unfed females, suggesting that paternal care limitation might also drive female competition. These results suggest that competition over food and possibly paternal care selects for dominant, breeding females to suppress the maturation of subordinate females to minimize competition. Monogamy in association with group living is therefore likely to have evolved because female competition prevents males from utilizing the potential for polygyny.