Optical communication systems (OCSs) mainly represent the backbone of modern long-haul communication networks because of low loss transmission over long distances and ultra-high capacity. However high data-rate transmission through optical fiber suffers from deterioration due to nonlinear impairments, such as four-wave mixing (FWM) in particular. At high launch power levels, which are required for the long-haul transmission over hundreds of km, these nonlinear effects become more severe which imposes a challenge to achieve satisfactory transmission performance. In this paper, a theoretical model for the FWM effects and its mitigation is presented and validated through simulation results. Moreover, two other nonlinear effects, polarization mode dispersion and nonlinear dispersion variations are also investigated for various values of launch power level. The transmission performance of the proposed OCS model is evaluated on the basis of bit error rate, optical signal-to-noise ratio and quality factor using different transmission channel parameters such as effective area, nonlinear refractive index, nonlinear dispersion, and linear dispersion.