We discuss the possibilities for the growth of primordial black holes (PBHs) via the accretion of dark matter. In agreement with previous works, we find that accretion during the radiation-dominated era does not lead to a significant mass increase. However, during matter domination, PBHs may grow by up to 2 orders of magnitude in mass through the acquisition of large dark matter halos. We discuss the possibility of PBHs being an important component in dark matter halos of galaxies, as well as their potential to explain the ultraluminous X-ray sources (ULXs) observed in nearby galactic disks. We point out that although PBHs are ruled out as the dominant component of dark matter, there is still a great deal of parameter space that is open to their playing a role in the modern-day universe. For example, a primordial halo population of PBHs each at 10 2.5 M , making up 0.1% of the dark matter, grows to 10 4.5 M via the accumulation of dark matter halos and accounts for $10% of the dark matter mass by a redshift of z % 30. These intermediate-mass black holes may then ''light up'' when passing through molecular clouds, becoming visible as ULXs at the present day, or they may form the seeds for supermassive black holes at the centers of galaxies.