The objective of the study was to estimate the contribution of ultra-processed foods to total caloric intake and investigate whether it differs according to socioeconomic position. We analyzed baseline data from the Brazilian Longitudinal Study of Adult Health (ELSA-Brasil 2008-2010; N = 14.378) and data on dietary intake using a food frequency questionnaire, assigning it into three categories: unprocessed or minimally processed foods and processed culinary ingredients, processed foods, and ultra-processed foods. We measured the associations between socioeconomic position (education, per capita household income, and occupational social class) and the percentage of caloric contribution of ultra-processed foods, using generalized linear regression models adjusted for age and sex. Unprocessed or minimally processed foods and processed culinary ingredients contributed to 65.7% of the total caloric intake, followed by ultra-processed foods (22.7%). After adjustments, the percentage of caloric contribution of ultra-processed foods was 20% lower among participants with incomplete elementary school when compared to postgraduates. Compared to individuals from upper income classes, the caloric contribution of ultra-processed foods was 10%, 15% and 20% lower among the ones from the three lowest income, respectively. The caloric contribution of ultra-processed foods was also 7%, 12%, 12%, and 17% lower among participants in the lowest occupational social class compared to those from high social classes. Results suggest that the caloric contribution of ultra-processed foods is higher among individuals from high socioeconomic positions with a dose-response relationship for the associations.