In this paper we present a linear dynamical system to model the ongoing neural response, measured by EEG, while a listener is selectively attending a speech stream that is presented in a mixture. In our state-space formulation, the latent state variables represent the activity of the underlying neural substrates and the ongoing neural dynamics are captured by a multivariate autoregressive model with exogenous inputs (MVARX model). The observation model projects these neural sources onto the EEG montage. System identification is performed using a novel regularized expectation maximization (EM) algorithm for linear dynamical systems. The model was able to correctly identify which of two simultaneously presented speech streams was attended in roughly 85% of one minute long test trials, averaged over all subjects.