The association between milk intake and cognitive disorders has been investigated in several epidemiological studies, but the findings are still conflicting. No quantitative assessment has been performed to evaluate the potential relationship of milk intake and cognitive disorders. From the inception to October 2016, the PubMed and the Embase databases were searched for observational studies reporting the association of milk consumption and cognitive disorders (Alzheimer’s disease, dementia, and cognitive decline/impairment). A generic inverse-variance random-effects method was used to pool the Odds Ratios (ORs) and corresponding 95% confidence intervals (CIs) for the highest compared with the lowest level of milk intake. Subgroup and meta-regression analyses were used to assess the heterogeneity between subgroups. We identified seven articles involving a total of 10,941 participants. The highest level of milk consumption was significantly associated with a decreased risk of cognitive disorders, and the pooled OR (95% CI) was 0.72 (0.56, 0.93), with evidence of significant heterogeneity (I2 = 64%, p = 0.001). Subgroup analysis indicated that the association was more pronounced in ischemic stroke patients based on a single study. Furthermore, the inverse association between milk intake and cognitive disorders was limited to Asian subjects, and the African populations showed an intermediate non-significant trend. Although we have obtained a significant association, an established relationship cannot be drawn due to the study limitation. Large prospective studies are needed to quantify the potential dose-response patterns of milk intake and to explore the association in populations with different characteristics.