Changes in the composition and proportions of the gut microbiota may be associated with numerous diseases, including cognitive impairment. Over the recent years, the growing interest in this relation is observed, but there are still many unknowns, especially in the elderly. To the best of our knowledge, this is the first work that synthesizes and critically evaluates existing evidence on the possible association between human gut microbiota and cognitive function in the elderly. For this purpose, comprehensive literature searches were conducted using the electronic databases PubMed, Google Scholar, and ScienceDirect. The gut microbiota of cognitively healthy and impaired elderly people may differ in the diversity and abundance of individual taxes, but specific taxes cannot be identified. However, some tendencies to changing the Firmicutes/Bacteroidetes ratio can be identified. Currently, clinical trials involving probiotics, prebiotics, and synbiotics supplementation have shown that there are premises for the claim that these factors can improve cognitive functions, however there is no single intervention beneficial to the elderly population. More reliable evidence from large-scale, long-period RCT is needed. Despite proposing several potential mechanisms of the gut microbiota’s influence on the cognitive function impairment, prospective research on this topic is extremely difficult to conduct due to numerous confounding factors that may affect the gut microbiota. Heterogeneity of research outcomes impairs insight into these relations.