Abstract: Four wave mixing is a prominent and major performance deteriorating factor in WDM systems. To suppress the FWM effects and increase data rate supportability of WDM systems, optimal modulation formats are required. In this research, a WDM system supporting 8 channels and each at 10 Gbps is proposed by incorporating 2 and 3D orthogonal modulation formats. Polarization shift keying is an essential modulation and studied at different distances in terms of Q-factor. Moreover, FWM power is analyzed and comparison has been established for 1-3D modulations at different frequency spacings such as 25, 50 and 100 GHz. The 2-D orthogonal modulation exhibits minimum FWM and thus, recommended to use in WDM systems at ultra dense spacings.