Balanced incomplete block design (BIBD) is an area of combinatorial mathematics that was first introduced in 1936 and developed very fast. Because of their unique fixed cross-correlation property, BIBDs have attracted the attention of researchers in communication systems. BIBDs have been used in various areas, such as error control coding, cryptography, code division multiple access networks, key distribution in sensor networks and watermarking. Among different BIBDs, symmetric designs are often used as codes; in order to simplify the signal processing, cyclic structures are of greatest interest.In this dissertation, we introduce new applications of symmetric BIBDs in communication systems, with special emphasis on optical communications, and use them primarily in constructing the symbols of novel M -ary modulation schemes. The first modulation technique introduced is a spectral amplitude encoded scheme that uses BIBDs as the encoding pattern, and is shown to achieve a high bit-rate in non-line-of-sight ultraviolet (NLOS-UV) systems. The other three modulation schemes are based on pulse position modulation technique, and are proposed for application is systems with certain restrictions, such as peak-power limited systems. Because of the structure of the symbols in these modulation schemes, the symbols are susceptible to interference between adjacent time-chips, and the error probability increases in dispersive channels. We introduce a technique to decrease the interference effect on these three modulation schemes by interleaving the transmitted pulses at the transmitter and de-interleaving them at the receiver.We then study the application of the new modulation schemes to visible light communication (VLC) systems. VLC is an appealing technology that is proposed for wireless indoor networking. Using this technique internet access can be provided by LED-based lighting system. Due to the integration of the illumination system with the communication system, it is crucial that the modulation techniques support lighting features. Dimming and low-fluctuations are the most challenging features to be supported by a communication system. We show that the proposed modulation techniques are able to provide a wide range of dimming levels and have a low fluctuation effect. We also present two techniques that enable VLC systems to provide simultaneous high-speed network access for multiple users.First and foremost, I would like to express my sincere gratitude towards my supervisor, Professor Maïté Brandt-Pearce, for her support, encouragement, and patience during my PhD career.