In recent times, there has been continuous interest in the area of content based information retrieval (CBIR) for images and video sequences. Exponential increase of multimedia data has triggered a cause for managing, storing and retrieving multimedia contents in convenient and efficient ways. Visual features from static images and dynamic videos are extracted to perform retrieval task. Once visual features are extracted, there is a need to search and retrieve relevant videos in efficient amount of time. This paper makes use of seven visual features; human detection, emotion, age, gender, activity, scene and object detection followed by sentence generation. Furthermore, generated sentence is used in multi-class recurrent neural network (RNN) to find genre of a video for retrieval task. Accuracy, precision and recall are used for evaluation of this framework on self generated dataset. Experiments show that our system is able to achieve high accuracy of 88.13%.