The use of nonreturn-to-zero (NRZ), return-to-zero (RZ), and carrier-suppressed return-to-zero (CSRZ) modulation formats in an ultradense wavelength-division multiplexing (UDWDM) scenario at 40 Gb/s is investigated. The results of a simulative analysis on back-to-back setups are presented. Narrow optical filtering at the receiver and at the transmitter, as well as the orthogonal polarization launch of adjacent channels, is studied in order to improve spectral efficiency. We demonstrate that standard setups do not allow acceptable performances with 50-GHz channel spacing for all the considered modulation formats, while the use of a transmission optical filter may dramatically improve the performance of RZ and CSRZ modulation formats that become suitable for the use in UDWDM systems. We show that, due to the narrow transmission filtering, the RZ pulse becomes NRZ-like, and the CSRZ modulation is duobinary coded. Furthermore, we demonstrate that NRZ modulation does not benefit from the introduction of a transmission optical filter, while it takes advantage of the orthogonal polarization launch of adjacent channels, but its performance is still worse than the RZ and CSRZ performance in a UDWDM scenario.