Anomaly targets in a hyperspectral image (HSI) are often multi-pixel, rather than single-pixel, objects. Therefore, algorithms using a test point vector may ignore the spatial characteristics of the test point. In addition, hyperspectral anomaly detection (AD) algorithms usually use original spectral signatures. In a fractional Fourier transform (FrFT), the signals in the fractional Fourier domain (FrFD) possess complementary characteristics of both the original reflectance spectrum and its Fourier transform. In this paper, a tensor RX (TRX) algorithm based on FrFT (FrFT-TRX) is proposed for hyperspectral AD. First, the fractional order of FrFT is selected by fractional Fourier entropy (FrFE) maximization. Then, the HSI is transformed into the FrFD by FrFT. Next, TRX is employed in the FrFD. Finally, according to the optimal spatial dimensions of the target and background tensors, the optimal AD result is achieved by adjusting the fractional order. TRX employs a test point tensor, making better use of the spatial characteristics of the test point. TRX in the FrFD exploits the complementary advantages of the intermediate domain to increase discrimination between the target and background. Six existing algorithms are used for comparison in order to verify the AD performance of the proposed FrFT-TRX over five real HSIs. The experimental results demonstrate the superiority of the proposed algorithm.