We propose a scheme to create optical nonreciprocal transmission in a double-cavity magnonics system, where one yttrium iron garnet sphere is placed in one of the cavities and a mechanical oscillator is shared by the coupled double microwave cavities. By manipulating the magnon–microwave photon coupling, we reveal the nonreciprocal propagation of electromagnetic fields at microwave frequencies in red and blue-detuned regimes. Furthermore, the nonreciprocal isolation ratio and group delay are analyzed. This scheme can inspire methods for constructing nonreciprocal devices.