Graphene is used as flexible electrodes in various optoelectronic devices. In these applications, ultrafast charge transfer from semiconducting light absorbers to graphene can impact the overall device performance. Here, we propose a mechanism in which the charge-transfer rate can be controlled by varying the number of graphene layers and their stacking. Using an organic semiconducting molecule as a light absorber, the charge-transfer rate to graphene is measured by using time-resolved photoemission spectroscopy. Compared to graphite, the charge transfer to monolayer graphene is about 2 times slower. Surprisingly, the charge transfer to A-B-stacked bilayer graphene is slower than that to both monolayer graphene and graphite. This anomalous behavior disappears when the two graphene layers are randomly stacked. The observation is explained by a charge-transfer model that accounts for the band-structure difference in mono-and bilayer graphene, which predicts that the charge-transfer rate depends nonintuitively on both the layer number and stacking of graphene.