Cognition has evolved to allow organisms to process, use and store information in their natural environment. Yet, cognitive abilities are traditionally measured in controlled laboratory conditions to obtain consistent and accurate measurements. Consequently, little is known about the actual effect of natural environmental variation on cognitive performances. Being able to modify cognitive performance according to environmental conditions (e.g., plasticity of attentional performances according to current predator densities) could provide evolutionary advantages. In this mini-review, we compile evidence for what we call "cognitive performance plasticity" (i.e., flexible adjustment of cognitive performance in response to the current environment). We then discuss methodological approaches associated with measurement of cognitive performance plasticity and cognition in general. Finally, we discuss the implications of acknowledging plasticity in cognitive performance, including a better understanding of the reproducibility crisis observed in cognitive sciences (Open Science Collaboration, 2015) and new lines of inquiry into the evolution of cognition and the adaptive value of cognitive performance plasticity.