Security is always an important issue in wireless communications. Physical layer encryption (PLE) is an effective way to enhance wireless communication security and prevent eavesdropping. Rather than replacing cryptography at higher layers, PLE's benefit is to enable using lightweight cryptosystems or provide enhanced security at the signal level. The upper cryptography is faced with a noise-free channel, and the processing object is bit data. In PLE, the effects of channel and noise can be exploited to enhance security and prevent deciphering. In addition, since the processing object is complex vector signals, there are more operational functions to select and design for PLE. The mathematical models, design frameworks, and cryptographic primitives of PLE are established. Two design frameworks are proposed: stream PLE and block PLE. For stream PLE, a new 3D security constellation mapping is derived. For block PLE, two types of sub-transforms are defined: isometry transformations and stochastic transformations. Furthermore, a practical system operation mode PLE-block chaining (PBC) is proposed to enhance the practical system security. The proposed PLE framework can resist known plaintext attacks and chosen-plaintext attacks. The simulation shows that the proposed isometry transformation method has good performances in terms of bit error ratio (BER) penalty and confusion degree.