A single channel optical asymmetric cryptosystem for color image in fractional Fourier transform (FrFT) domain is presented. Instead of commonly used RGB channels encryption system, the color image is encoded into grayscale format and encrypted in a single optical channel system. For the design of asymmetric approach, the effective trapdoor one-way function is calculated by employing equal modulus decomposition (EMD). To enhance the security of the proposed cryptosystem, the Ushiki chaotic system is performed to generate the random phase mask in FrFT and a random sequence for scrambling the private key. The sensitive initial values and chaotic data can be regarded as the additional keys and public key. Various numerical experiments are given to verify the validity and capability of the proposed color cryptosystem.