To further develop three-dimensional (3D) applications, it is important to elucidate the negative effects of 3D applications on the human body and mind. Thus, this study investigated differences in the effects of visual fatigue on cognition and brain activity using visual and auditory tasks induced by watching a 1-h movie in two dimensions (2D) and 3D. Eighteen young men participated in this study. Two conditions were randomly performed for each participant on different days, namely, watching the 1-h movie on television in 2D (control condition) and 3D (3D condition). Before and after watching the 1-h movie on television, critical flicker fusion frequency (CFF: an index of visual fatigue), and response accuracy and reaction time for the cognitive tasks were determined. Brain activity during the cognitive tasks was evaluated using a multi-channel near-infrared spectroscopy system. In contrast to the control condition, the decreased CFF, and the lengthened reaction time and the decreased activity around the right primary somatosensory cortex during Go/NoGo blocks in the visual task at post-viewing in the 3D condition were significant, with significant repeated measures correlations among them. Meanwhile, in the auditory task, the changes in cognitive performance and brain activity during the Go/NoGo blocks were not significant in the 3D condition. These results suggest that the failure or delay in the transmission of visual information to the primary somatosensory cortex due to visual fatigue induced by watching a 3D movie reduced the brain activity around the primary somatosensory cortex, resulting in poor cognitive performance for the visual task. This suggests that performing tasks that require visual information, such as running in the dark or driving a car, immediately after using a 3D application, may create unexpected risks in our lives. Thus, the findings of this study will help outlining precautions for the use of 3D applications.