Currently, secure multimedia applications are becoming a very hot research topic, especifically over the Internet and wireless communication networks due to their rapid progress. Several researchers have implemented various chaotic image and video encryption algorithms to achieve data stability and communication security. This paper presents a novel bit-level video frame encryption algorithm that is dependent on the piecewise linear chaotic maps (PWLCMs). It is implemented for orbital angular momentum (OAM) modulation over different turbulence channels. Firstly, the mathematical model for the bit error rate (BER) of OAM is derived over the gamma-gamma turbulence channel. After that, a comparison between the theoretical results from mathematica and the simulation results from MATLAB for different turbulence strengths, signal-to-noise ratios (SNRs), and propagation distance values is presented to assure that there is a perfect match. The proposed OAM video cryptosystem is checked using entropy analysis, histogram testing, attack analysis, time analysis, correlation testing, differential analysis, and other quality and security evaluation metrics. The simulation results and the performance analysis confirm that the proposed cryptosystem is reliable and secure for video frame encryption, and communication with different turbulence conditions in free space.INDEX TERMS PWLCM, Multimedia security, Free-space optics, Gamma-gamma turbulence channel, Orbital angular momentum.