The control of line-edge roughness (LER) of features printed in photoresist poses significant challenges to next-generation lithography techniques such as extreme-ultraviolet (EUV) lithography. Achieving adequately low LER levels will require accurate resist characterization as well as the ability to separate resist effects from other potential contributors to LER. One significant potential contributor is LER on the mask. Here we explicitly study the mask to resist LER coupling using both analytical and computer-simulation methods. We present what is to our knowledge a new imaging transfer function referred to as the LER transfer function (LTF), which fundamentally differs from both the conventional modulation transfer function and the optical transfer function. Moreover, we present experimental results demonstrating the impact of current EUV masks on projection-lithography-based LER experiments.