We quantify the importance of mass accretion during AGN phases in the growth of supermassive black holes (BH) by comparing the mass function of black holes in the local universe with that expected from AGN relics, which are black holes grown entirely with mass accretion during AGN phases. The local BH mass function (BHMF) is estimated by applying the well-known correlations between BH mass, bulge luminosity and stellar velocity dispersion to galaxy luminosity and velocity functions. We find that different correlations provide the same BHMF only if they have the same intrinsic dispersion. The density of supermassive black holes in the local universe which we estimate is ρ BH = 4.6 +1.9 −1.4 h 2 0.7 × 10 5 M ⊙ Mpc −3 . The relic BHMF is derived from the continuity equation with the only assumption that AGN activity is due to accretion onto massive BH's and that merging is not important. We find that the relic BHMF at z = 0 is generated mainly at z < 3 where the major part of BH's growth takes place. Moreover, the BH growth is anti-hierarchical in the sense that smaller BH's (M BH < 10 7 M ⊙ ) grow at lower redshifts (z < 1) with respect to more massive one's (z ∼ 1 − 3). Unlike previous work, we find that the BHMF of AGN relics is perfectly consistent with the local BHMF indicating the local black holes were mainly grown during AGN activity. This agreement is obtained while satisfying, at the same time, the constraints imposed from the X-ray background. The comparison between the local and relic BHMF's also suggests that the merging process is not important in shaping the relic BHMF, at least at low redshifts (z < 3), and allows us to estimate the average radiative efficiency (ε), the ratio between emitted and Eddington luminosity (λ) and the average lifetime of active BH's. Our analysis thus suggests the following scenario: local black holes grew during AGN phases in which accreting matter was converted into radiation with efficiencies ε = 0.04 − 0.16 and emitted at a fraction λ = 0.1 − 1.7 of the Eddington luminosity. The average total lifetime of these active phases ranges from ≃ 4.5 × 10 8 yr for M BH < 10 8 M ⊙ to ≃ 1.5 × 10 8 yr for M BH > 10 9 M ⊙ but can become as large as ∼ 10 9 yr for the lowest acceptable ǫ and λ values.