Research investigating pragmatic abilities in healthy aging suggests that both production and comprehension might be compromised; however, it is not clear how pragmatic abilities evolve in late adulthood, as well as when difficulties are more likely to arise. The aim of this study is to investigate the decline of pragmatic skills in aging, and to explore what cognitive and demographic factors support pragmatic competence. We assessed pragmatic production skills, including discourse abilities such as speech, informativeness, information flow, paralinguistic aspects, as well as the ability to produce informative descriptions of pictures, and pragmatic comprehension skills, which encompassed the ability to understand discourse and the main aspects of a narrative text, to infer non-literal meanings and to comprehend verbal humor in a group of elderly individuals and in a sample of younger participants. Moreover, specific cognitive functions (short-term memory, verbal and visuospatial working memory, inhibition Theory of Mind, and Cognitive Reserve) were assessed in both groups. Pragmatic difficulties seem to occur in late adulthood, likely around 70 years, and emerge more prominently when participants are asked to understand verbal humor. Age was the only predictor of general pragmatic performance in a sample of cognitively unimpaired older adults; conversely, when elderly individuals with less intact inhibitory control are considered, a general role of inhibition emerged, in addition to working memory and ToM in specific tasks.