Cognitive vision systems are able to learn from real world visual scenes and generate semantic descriptions about the exposed scene. At present many researches have been conducted to generate context specific semantics of the visual scene mainly through quantitative methods. Even though those approaches were good enough to tackle a single situation the methods used cannot be generalized to handle multiple scenarios. The limitations are mainly due to use of quantitative approaches and due to use of lack of commonsense knowledge. Therefore, we have design and developed a cognitive vision system that learns protocol rules, which describe the conduct of the visual scene through a qualitative approach and also use domain independent common sense knowledge to reason on the protocol rules, more effectively and realistically. Qualitatively generated protocol rules in our cognitive computer vision system are designed to be able to work as a knowledge base of a standard expert system, where the domain independent common sense knowledge module operates as a window to abstract external knowledge from a user when the existing knowledge of the system is inadequate to answer a given query.