The duration of moving stimuli is overestimated compared to that of static stimuli (motion-induced duration dilation). In contrast, after participants visually adapt to a moving stimulus, they underestimate the duration of a following moving stimulus (adaptation-induced duration compression). These two motion-related time distortions have not been examined using the same stimuli within a study, and it remains unknown whether these phenomena have similar characteristics. Here, we used luminance-modulated and isoluminant chromaticity-modulated moving stimuli and tested whether these types of motion induce perceptual distortions of duration. As isoluminant-color motion is perceived slower than luminance motion at the same physical speed, the speeds of the two types of motion were either physically same (Experiment 1) or perceptually matched (Experiment 2). We found that when motion speeds were physically identical, luminance motion induced larger duration distortions than did isoluminant-color motion. When motion speeds were perceptually identical, luminance motion still induced a larger motion-induced duration dilation than isoluminant-color motion did, while luminance motion and isoluminant-color motion induced approximately the same amount of adaptation-induced duration compression. We also found that, for both effects, the amount of duration distortion induced by luminance motion positively correlated with that induced by isoluminant-color motion. These results indicate robust and consistent individual differences in motion-related duration distortions that are common to luminance motion and isoluminant-color motion.