When an individual engages in a task, the associated evoked activities build upon already ongoing activity, itself shaped by an underlying functional connectivity baseline (Fox et al., 2009; Smith et al., 2009; Tavor et al., 2016). To facilitate understanding the building blocks of cognition we incorporate the idea that task-induced functional connectivity modulation with respect to its underlying resting state functional connectivity is task-specific. Here, we introduce a framework incorporating task potency, providing direct access to task-specificity through enabling direct comparison between task paradigms. In particular, to study functional connectivity modulations related to cognitive involvement in a task we define task potency as the amplitude of connectivity modulations away from the brain’s baseline functional connectivity architecture as observed during a resting state acquisition. We demonstrate the use of our framework by comparing three tasks (visuo-spatial working memory, reward processing, and stop signal task) available within a large cohort. Using task potency, we demonstrate that cognitive operations are supported by a common baseline of within-network interactions, supplemented by connections between large-scale networks in order to solve a specific task.Highlights-Task potency framework defines modulation of functional connectivity away from baseline resting state-More within-than between-network modulations are induced by task performance-Between-network modulations are task-specific-Edges modulated by multiple tasks are mostly within-network-The task potency can be used to define the most potent task