Sexually antagonistic (SA) selection, a form of selection that can occur when both sexes have different fitness optima for a trait, is a major force shaping the evolution of organisms. A seminal model developed by Rice (Rice WR. 1984. Sex chromosomes and the evolution of sexual dimorphism. Evolution 38:735–742.) predicts that the X chromosome should be a hotspot for the accumulation of loci under SA selection as compared with the autosomes. Here, we propose a methodological framework designed to detect a specific signature of SA selection on viability, differences in allelic frequencies between the sexes. Applying this method on genome-wide single nucleotide polymorphism (SNP) data in human populations where no sex-specific population stratification could be detected, we show that there are overall significantly more SNPs exhibiting differences in allelic frequencies between the sexes on the X chromosome as compared with autosomes, supporting the predictions of Rice’s model. This pattern is consistent across populations and is robust to correction for potential biases such as differences in linkage disequilibrium, sample size, and genotyping errors between chromosomes. Although SA selection is not the only factor resulting in allelic frequency differences between the sexes, we further show that at least part of the identified X-linked loci is caused by such a sex-specific processes.