We use deep Chandra X-ray imaging to measure the distribution of specific black hole accretion rates (L X relative to the stellar mass of the galaxy) and thus trace AGN activity within star-forming and quiescent galaxies, as a function of stellar mass (from 10 8.5 -10 11.5 M ⊙ ) and redshift (to z ∼ 4). We adopt near-infrared selected samples of galaxies from the CANDELS and UltraVISTA surveys, extract X-ray data for every galaxy, and use a flexible Bayesian method to combine these data and to measure the probability distribution function of specific black hole accretion rates, λ sBHAR . We identify a broad distribution of λ sBHAR in both starforming and quiescent galaxies-likely reflecting the stochastic nature of AGN fuelling-with a roughly power-law shape that rises toward lower λ sBHAR , a steep cutoff at λ sBHAR 0.1−1 (in Eddington equivalent units), and a turnover or flattening at λ sBHAR 10 −3 − 10 −2 . We find that the probability of a star-forming galaxy hosting a moderate λ sBHAR AGN depends on stellar mass and evolves with redshift, shifting toward higher λ sBHAR at higher redshifts. This evolution is truncated at a point corresponding to the Eddington limit, indicating black holes may self-regulate their growth at high redshifts when copious gas is available. The probability of a quiescent galaxy hosting an AGN is generally lower than that of a star-forming galaxy, shows signs of suppression at the highest stellar masses, and evolves strongly with redshift. The AGN duty cycle in high-redshift (z 2) quiescent galaxies thus reaches ∼20 per cent, comparable to the duty cycle in star-forming galaxies of equivalent stellar mass and redshift.