The potential of milk-derived glycomacropeptide (GMP) and lactose for modulating the human gut microbiota of older people, in whom loss of diversity correlates with inferior health, was investigated. We used an in vitro batch fermentation (artificial colon model) to simulate colonic fermentation processes of two GMP products, i.e., a commercially available GMP concentrate and a semipurified GMP concentrate, and lactose. Faecal samples were collected from healthy and frail older people. Samples were analyzed by Illumina Miseq sequencing of rRNA gene amplicons. The commercial GMP preparation had a positive effect on the growth of Coprococcus and Clostridium cluster XIVb and sustained a higher faecal microbiota diversity compared to control substrates or lactose. Lactose fermentation promoted the growth of Proteobacteria including Escherichia/Shigella. This work provides an in-depth insight on the potential of GMP and lactose for modulating the gut microbiota and contributes more evidence confirming the prebiotic activity of GMP.