Structural displacement is considered an important indicator to assess structural conditions. To directly measure the displacement in the time domain, a visually servoed paired structured light system was proposed. The system is composed of two sides facing each other, each with one or two lasers that are controlled by a visually servoed two-DOF manipulator, a camera, and a screen. The relative six-DOF displacement between the two sides can be estimated by calculating the positions of the projected laser beams on the screens and the rotation angles of manipulators. To verify the performance of the proposed system, two kinds of field tests were carried out. A prototype of the system was built and installed on a steel frame building structure and a railway bridge, respectively. The estimated displacements were compared with the reconstructed displacement from an accelerometer. The test results verify the performance of the system and its applicability to real structures.