Phased antenna arrays have been used extensively for massive MIMO-based multiuser beamforming to improve both the throughput and power efficiency of wireless communication systems. Beamforming can be realized by the Butler matrix, which has been demonstrated in the electric domain. With wireless systems migration toward simplified remote radio units through radio-over-fiber and sharing of signal processing functions located in the central office, it is desirable to implement this beamforming technique in the optical domain. In this paper, we propose an optical implementation of the Butler matrix using purely passive optical components such as directional couplers and delay lines. With the additional use of optical routing elements, the number of signal streams connected to the baseband unit equals the number of the actual users rather than the number of antenna elements, thereby achieving hardware efficiency for multiuser beamforming. Moreover, the proposed optical Butler matrix acts on the optical intensity rather than the optical field, and therefore, low-cost intensity-modulation direct-detection (IMDD) optical transceivers can be used. A proof-of-concept experiment is conducted to demonstrate the optical 2 × 2 Butler matrix. Scaling the optical Butler matrix to high port counts is also discussed.