In instruction conditioned navigation, agents interpret natural language and their
surroundings to navigate through an environment. Datasets for studying this task typically
contain pairs of these instructions and reference trajectories. Yet, most evaluation metrics
used thus far fail to properly account for the latter, relying instead on insufficient similarity
comparisons. We address fundamental flaws in previously used metrics and show how
Dynamic Time Warping (DTW), a long known method of measuring similarity between two …