Cross-media retrieval has drawn much attention recently, by which users can retrieve results across different media types like image and text. The existing methods mainly focus on the condition where the training data covers all the categories in the testing data. However, the number of categories is infinite in real world and it is impossible to include all categories in the training data. Due to the limitation of scalability, the performance of existing methods will be not effective when retrieving with unseen categories. For addressing the issues of both “heterogeneity gap” and the gap of seen and unseen categories, this paper proposes a new approach to model both multimedia and external knowledge information. The common semantic representations are generated jointly by media features and category weight vectors which are learned by utilizing online encyclopedias. Experiment on two widely-used datasets shows the effectiveness of our approach for zero-shot cross-media retrieval.