Estimating emotion from speech is an active and ongoing area of research, however most of the literature addresses acted speech and not natural day to day conversational speech. Identifying emotion from the latter is difficult because the emotion expressed by non-actors is not necessarily prominent. In this paper we validate the hypothesis, which is based on the observations that human annotators show large inter and intra person variations in annotating emotions expressed in realistic speech as compared to the acted speech. We propose a method to recognize emotions using the knowledge of events in an interactive voice response setup. The main contribution of the paper is the use of event based knowledge to enhance the identification of emotions in real natural speech.