Graphene has emerged as an ultrafast photonic material for on-chip all-optical modulation. However, its atomic thickness limits its interaction with guided optical modes, which results in a high switching energy per bit or low modulation efficiencies. Nonetheless, it is possible to enhance the interaction of guided light with graphene by nanophotonic means. Herein, we present a practical design of an all-optical modulator that is based on graphene and hexagonal boron nitride (hBN) heterostructures that are hybrid integrated into silicon slot waveguides. Using this device, a high extinction ratio (ER) of 7.3 dB, an ultralow insertion loss (IL) of <0.6 dB, and energy-efficient switching (<0.33 pJ/bit) are attainable for a 20 µm long modulator with double layer graphene. In addition, the device performs ultrafast switching with a recovery time of <600 fs, and could potentially be employed as a high-performance all-optical modulator with an ultra-high bandwidth in the hundreds of GHz. Moreover, the modulation efficiency of the device is further enhanced by stacking additional layers of graphene-hBN heterostructures, while theoretically maintaining an ultrafast response. The proposed device exhibits highly promising performance metrics, which are expected to serve the needs of next-generation photonic computing systems.