To thrive in dynamic environments, animals can generate flexible behavior and rapidly adapt responses to a changing context and internal state. Examples of behavioral flexibility include faster stimulus responses when attentive and slower responses when distracted. Contextual modulations may occur early in the cortical hierarchy and may be implemented via afferent projections from top-down pathways or neuromodulation onto sensory cortex. However, the computational mechanisms mediating the effects of such projections are not known. Here, we investigate the effects of afferent projections on the information processing speed of cortical circuits. Using a biologically plausible model based on recurrent networks of excitatory and inhibitory neurons arranged in cluster, we classify the effects of cell-type specific perturbations on the circuit's stimulus-processing capability. We found that perturbations differentially controlled processing speed, leading to counterintuitive effects such as improved performance with increased input variance. Our theory explains the effects of all perturbations in terms of gain modulation, which controls the timescale of the circuit dynamics. We tested our model using large-scale electrophysiological recordings from the visual hierarchy in freely running mice, where a decrease in single-cell gain during locomotion explained the observed acceleration of visual processing speed. Our results establish a novel theory of cell-type specific perturbations linking connectivity, dynamics, and information processing via gain modulations.