Primary visual cortex (V1) neurons integrate motor and multisensory information with visual inputs during sensory processing. However, whether V1 neurons also integrate and encode higher-order cognitive variables is less understood. We trained mice to perform a context-dependent cross-modal decision task where the interpretation of identical audio-visual stimuli depends on task context. We performed silicon probe population recordings of neuronal activity in V1 during task performance and showed that task context (whether the animal should base its decision on visual or auditory stimuli) can be decoded during both intertrial intervals and stimulus presentations. Context and visual stimuli were represented in overlapping populations but were orthogonal in the population activity space. Context representation was not static but displayed distinctive dynamics upon stimulus onset and offset. Thus, activity patterns in V1 independently represent visual stimuli and cognitive variables relevant to task execution.