The auditory modality is fundamentally a temporal sense that requires analysis of changes in sound signals on timescales ranging from microseconds to minutes. To generate a faithful representation of changes in sound intensity and frequency over time, sound offsets (disappearances) as well as sound onsets (appearances) must be encoded by the auditory system. We review here the computational significance, perceptual roles, anatomical locations, and cellular and network origins of sound-offset responses in the mammalian auditory brain. We show that sound-offset responses arise from mechanisms and pathways distinct from those producing sound-onset responses, and are likely to be essential for auditory processing of temporally discontinuous sounds such as speech.