Insulin secretion is key for glucose homeostasis. Insulin secretory granules (SGs) exist in different functional pools, with young SGs being more mobile and preferentially secreted. However, the principles governing the mobility of age-distinct SGs remain undefined. Using the time-reporter insulin-SNAP to track age-distinct SGs we now show that their dynamics can be classified into three components: highly dynamic, restricted, and nearly immobile. Young SGs display all three components, whereas old SGs are either restricted or nearly immobile. Both glucose stimulation and F-actin depolymerization recruit a fraction of nearly immobile young, but not old, SGs for highly dynamic, microtubule-dependent transport. Moreover, F-actin marks multigranular bodies/lysosomes containing aged SGs. These data demonstrate that SGs lose their responsiveness to glucose stimulation and competence for microtubule-mediated transport over time while changing their relationship with F-actin.