In this work, we present a method for characterizing the transmission matrices of complex scattering media using a physics-informed, multi-plane neural network (MPNN) without the requirement of a known optical reference field. I n c ontrast t o p revious t echniques, o ur m ethod i s a ble t o m easure c omplete i nformation about the transmission matrix, which is necessary for coherent control of light through a complex medium. Here, we design a neural network that describes the exact physical apparatus consisting of a trainable layer describing the unknown transmission matrix. We then employ randomized measurements to train the neural network which accurately recovers the transmission matrix of a commercial multi-mode fiber. We demonstrate how our method is significantly more accurate, and noise-robust than the standard method of phase-stepping holography and show how it can be generalized to characterize a cascade of transmission matrices. This work presents an essential tool for accurate light control through complex media, with applications ranging from classical optical networks, biomedical imaging, to quantum information processing.