Binge-drinking is one of the alcohol drinking patterns with the worst health consequences. Nonetheless, binge-drinking is highly prevalent. The perceived benefits that motivate it are ultimately related to subjective well-being. In this context, we analyzed the relationship between binge-drinking and quality of life. We evaluated 8992 participants of the SUN cohort. We classified as binge-drinkers those who reported consuming six or more drinks on at least one occasion the year before recruitment (n = 3075). We fitted multivariable logistic regression models to calculate the odds ratios (ORs) of a worse physical and mental quality of life, measured with the validated SF-36 questionnaire at 8 years of follow-up (cut-off point = P₇₅ or highest score). Binge-drinking was associated with greater odds of having a worse mental quality of life, even adjusting for quality of life at 4 years of follow-up, used as an approximation to a baseline measure (OR = 1.22 (1.07–1.38)). This value was mainly due to the effects on vitality (OR = 1.17 (1.01–1.34)) and mental health (OR = 1.22 (1.07–1.39)). Binge-drinking may lead to poorer mental quality of life; therefore, binge-drinking for enhancement purposes does not seem to be justified by this effect.