This paper presents a possible utilization of advances signal processing techniques in the optical transmission system at the signal transmission. A contribution briefly introduces basic characteristics of the optical fiber, Forward Error Correction techniques and digital modulation formats considered for analysis. The paper mainly focuses on RS and BCH codes used in conjunction with OOK and BPSK techniques and their practical exploitation in the optical transmission system. Then, a created simulation program for the optical transmission system is presented with detailed description of blocks related to mutual cooperation RS and BCH codes with binary modulation techniques (OOK, DBPSK and BPSK). In the third part, more detailed analyses of RS and BCH performances are presented.