Much of the existing work on action recognition combines simple features with complex
classifiers or models to represent an action. Parameters of such models usually do not have
any physical meaning nor do they provide any qualitative insight relating the action to the
actual motion of the body or its parts. In this paper, we propose a new representation of
human actions called sequence of the most informative joints (SMIJ), which is extremely
easy to interpret. At each time instant, we automatically select a few skeletal joints that are …