Users with severe motor impairment may find it difficult to operate a wheelchair when they are in tight space (e.g., passing doorway) or when avoiding obstacles since they cannot command the wheelchair by means of a conventional joystick. Here we propose a framework that can assist users to overcome such circumstances using a hierarchical semi-autonomous control strategy. Initially multimodal user inputs based on momentary switch and yaw head angle are analyzed to decide a maneuvering mode and assign the direction of travel. Then, environmental information is perceived using the combination of a laser range finder and the Kinect sensor for determining safety map around wheelchair's vicinity. Eventually, the user's inputs are provided to the navigation planner along with the safety map to moderate motion that is collision free and the best for user preference. Experimental results demonstrate the feasibility of the proposed approach.