Histone acetylation is a major mechanism of chromatin remodeling, contributing to epigenetic regulation of gene transcription. Histone deacetylases (HDACs) are involved in both physiological and pathological conditions by regulating the status of histone acetylation. Although histone deacetylase 4 (HDAC4), a member of the HDAC family, may lack HDAC activity, it is actively involved in regulating the transcription of genes involved in synaptic plasticity, neuronal survival, and neurodevelopment by interacting with transcription factors, signal transduction molecules and HDAC3, another member of the HDAC family. HDAC4 is highly expressed in brain and its homeostasis is crucial for the maintenance of cognitive function. Accumulated evidence shows that HDAC4 expression is dysregulated in several brain disorders, including neurodegenerative diseases and mental disorders. Moreover, cognitive impairment is a characteristic feature of these diseases. It indicates that aberrant HDAC4 expression plays a pivotal role in cognitive impairment of these disorders. This review aims to describe the current understanding of HDAC4’s role in the maintenance of cognitive function and its dysregulation in neurodegenerative diseases and mental disorders, discuss underlying molecular mechanisms, and provide an outlook into targeting HDAC4 as a potential therapeutic approach to rescue cognitive impairment in these diseases.