The human auditory system has an amazing ability to separate and understand sounds. We believe that temporal information plays a key role in this ability, more important than the spectral information that is traditionally emphasized in hearing science. In many hearing tasks, such as describing or classifying single sound sources, the underlying mathematical equivalence makes the temporal versus spectral argument moot. We show how the nonlinearity of the auditory system breaks this equivalence, and is especially important in analyzing complex sounds from multiple sources of different characteristics. The auditory system is inherently nonlinear. In a linear system, the component frequencies of a signal are unchanged, and it is easy to characterize the amplitude and phase changes caused by the system. The cochlea and the neural processing that follow are more interesting. The bandwidth of a cochlear “filter” changes at different sound levels, and neurons change their sensitivity as they adapt to sounds. Inner Hair Cells (IHC) produce nonlinear rectified versions of the sound, generating new frequencies such as envelope components. All of these changes make it difficult to describe auditory perception in terms of the spectrum or Fourier transform of a sound.

One characteristic of an auditory signal that is undisturbed by most nonlinear transformations is the periodicity information in the signal. Even if the bandwidth, amplitude, and phase characteristics of a signal are changing, the repetitive characteristics do not. In addition, it is very unlikely that a periodic signal could come from more than one source. Thus the auditory system can safely assume that sound fragments with a consistent periodicity can be combined and assigned to a single source. Consider, for example, a sound formed by opening and closing the glottis four times and filtering the resulting puffs of air with the vocal resonances. After nonlinear processing the lower auditory nervous system will still detect four similar events which will be heard and integrated as coming from a voice. The duplex theory of pitch perception, proposed by Licklider in 1951 [11] as a unifying model of pitch perception, is even more useful as a model for the extraction and representation of temporal structure for both periodic and non-periodic signals. This theory produces a movie-like image of sound which is called a correlogram. We believe that the correlogram, like other representations that summarize the temporal information in a signal, is an important tool for understanding the auditory system.