We propose a novel physical layer security scheme that exploits an optimization method as a one-way function. The proposed scheme builds on nonsquare differential multipleinput multiple-output (MIMO), which is capable of noncoherent detection even in massive MIMO scenarios and thus resilient against risky pilot insertion and pilot contamination attacks. In contrast to conventional nonsquare differential MIMO schemes, which require space-time projection matrices designed via highly complex, discrete, and combinatorial optimization, the proposed scheme utilizes projection matrices constructed via low-complexity continuous optimization designed to maximize the coding gain of the system. Furthermore, using a secret key generated from the true randomness nature of the wireless channel as an initial value, the proposed continuous optimizationbased projection matrix construction method becomes a one wayfunction (in a cryptographic sense), making the proposed scheme a physical layer secure differential MIMO system. An attack algorithm to challenge the proposed scheme is also devised, which demonstrate that the security level achieved improves as the number of transmit antennas increases, even in an environment where the eavesdropper can perfectly estimate channel coefficients and experiences asymptotically large signal-to-noise ratios.