In this paper, we propose a differential multiple-input multiple-output (MIMO) scheme based on the novel concept of chaos-based time-varying unitary matrices to demonstrate-for the first time in the literature-the ability of differential encoding in achieving practical physical layer security even without the need for using channel estimation. In the proposed scheme, an erroneous secret key, which is extracted from the wireless nature, is used to initialize a chaos sequence that is responsible for generating artificially time-varying unitary matrices capable of obfuscating the transmitted data symbols from illegitimate eavesdroppers. Contrary to conventional studies, the key agreement ratio in this study is assumed to be imperfect, which is often true and very realistic in high-mobility scenarios. Following this, we conceive a new calibration algorithm for reconciling the chaotic sequence generated at the legitimate parties, thus making this calibration algorithm a unique, novel solution to the key sharing problem of conventional chaos-based communication techniques, which has been overlooked over the past few decades. It is found out that differential encoding obviates additional complexity and insecurity in dealing with channel estimation, whereas an eavesdropper must tackle the complicated differentially encoded patterns, which have an exponentially increasing complexity order. In addition, the obtained simulation results demonstrate that the proposed scheme can outperform conventional chaos-based MIMO schemes that assume perfect channel knowledge.