Emotion Recognition in Conversations (ERC) can capture the speakers’ emotional changes in multiple rounds of conversation, so it has a wide range of applications. In recent years, Graph Neural Networks have been naturally used in ERC tasks due to their ability to capture complex non-Euclidian spatial features. However, how to model conversations easily and effectively to improve the effect of ERC in the complex interaction mode still needs to be explored. To this end, we propose a new approach to construct a speaker-utterance interactive heterogeneous network that effectively models context while taking into account the global characteristics of speakers. On this basis, we propose a graph neural network based on the speaker and the corresponding utterances interactions, which dynamically updates the representations of utterances and speakers according to the order in which the speakers talk. We formulate different update methods for utterance and speaker nodes to ensure that the particularity of the heterogeneous network is fully explored. We conduct extensive experiments on four ERC benchmark datasets, and our approach achieves an average 0.7% performance improvement over the most advanced methods, which validates the effectiveness of properly modeling speakers in ERC tasks.