Detection and tracking of different moving objects in the surrounding environment is an essential capability for any robotic perception system. This enables forecasting of object trajectories and safe navigation of a mobile robot, such as a self-driving car, toward its goal. In this paper, we demonstrate a multi-object tracker from image data in a tracking-by-detection paradigm. The tracking algorithm imposes no prior requirements on the tracked objects. This paper demonstrates how with a simple and intuitive cost function, competitive and real-time performance is achieved. Furthermore, the multi-object tracker is combined with a stereo visual odometry system to obtain a more complete knowledge of the environment. Beside 2D object detections and tracking in the image domain, this provides robot motion estimates and 3D object trajectories in the world frame. Evaluation and discussion of the system is presented along with a study on the effect of removing features lying on tracked objects on visual odometry performance.