Correlated neural activity is a known feature of the brain [1] and evidence increases that it is closely linked to information processing [2]. The temporal shape of covariances has early been related to synaptic interactions and to common input shared by pairs of neurons [3]. Recent theoretical work explains the small magnitude of covariances in inhibition dominated recurrent networks by active decorrelation [4,5,6]. For binary neurons the mean-field approach takes random fluctuations into account to accurately predict the average activity in such networks [7] and expressions for covariances follow from a master equation [8], both briefly reviewed here for completeness. In our recent work we have shown how to map different network models, including binary networks, onto linear dynamics [9]. Binary neurons with a strong non-linear Heaviside gain function are inaccessible to the classical treatment [8]. Here we show how random fluctuations generated by the network effectively linearize the system and implement a self-regulating mechanism, that renders population-averaged covariances independent of the interaction strength and keeps the system away from instability.