Intermediate-mass black holes (IMBHs: masses between 100 − 10 6 M ⊙ ) historically comprise of an elusive population compared to stellar-mass and supermassive BHs. Recently IMBHs have started to be observed at the centers of low-mass galaxies. We perform cosmological hydrodynamical simulations of (2h −1 Mpc) 3 comoving boxes and investigate the growth and feedback of central IMBHs in dwarf galaxies (DGs). The earliest BHs appear at z ∼ 18 − 25, and grow thereafter by accreting gas and by merger with other BHs. We find that, starting from 10 2 M ⊙ , it is possible to build up IMBHs of a few×10 5 − 10 6 M ⊙ by z = 5, when the BHs are seeded in halos less massive than 4 × 10 7 M ⊙ . The BH accretion rates increase with time, and reaches ṀBH = (0.2 − 0.8) ṀEdd for the massive IMBHs by z = 4. The star formation rate density (SFRD) evolution of the DGs (stellar mass 10 5 − 10 8 M ⊙ ) has a peak plateau between z = 4 − 6. Star formation is quenched between z = 9 − 4. The SFRD is reduced by factors up to 3, when the BHs have grown to a few times 10 5 M ⊙ . Even in the presence of stronger SN-driven mass ejection, the BHs continue to grow up to z ∼ 6, sustained by gas inflows driven by galaxy mergers and interactions in a cosmological environment. Our conclusions, based on numerical simulation results, support the scenario that early feedback from IMBHs in gas-rich DGs at z = 5 − 8 can potentially solve several anomalies in the DG mass range within the concordance ΛCDM cosmological scenario (Silk 2017). Our results suggest that IMBHs at DG centers grow faster than their host galaxies in the early Universe, and the resulting BH feedback turns the DGs and the BHs dormant.