Scene graph generation is widely studied in recent years, which tries to understand the interactions of different objects as a whole. The earlier researches only recognize a few relationships or model contexts among different relationships, neglecting the associations of predicates for each object pair. In this paper, we propose a two-stage framework named predicate association network (PANet) to properly extract contexts and model predicate association. In the first stage, instance-level and scene-level context are extracted for object classification and further used for predicate classification in the next stage. With a recurrent neural network, alignment technique and attention mechanism are combined to collect the associations of predicates in the second stage. The experiments on the Visual Genome dataset show that our method is effective and outperforms the state-of-the-art methods.