We establish a topological model of alkali borophosphate and calcium borophosphate glasses, which describes the effect of both the network formers and network modifiers on physical properties. We show that the glass transition temperature (Tg), Vickers hardness (HV), liquid fragility (m), and isobaric heat capacity jump at Tg (ΔCp) of these glasses are related to the network topology, which is determined by structure of the glass. Therefore, we also demonstrate that the temperature dependent constraint theory can quantitatively explain the mixed network former effect in borophosphate glasses. The origin of the effect of the type of network modifying oxide on Tg, HV, m, and ΔCp of calcium borophosphate glasses is revealed in terms of the modifying ion sub-network. The same topological principles quantitatively explain the significant differences in physical properties between the alkali and the calcium borophosphate glasses. This work has implications for quantifying structure-property relations in complex glass forming systems containing several types of network forming and modifying oxides.