Listeners regularly cope with highly variable input to successfully recognize speech. One way this occurs is by adapting to the systematicities of individual talkers. Research on talker-specific adaptation have found that listeners generalize talker-specific phoneme categories to new talkers (Eisner & McQueen, 2005; Kraljic & Samuel, 2006, 2007) or individuate such categories, creating talker-specific categories (Luthra et al., 2021; Tamminga et al., 2020). Across four experiments, we investigate under what conditions listeners generalize or individuate talker-specific phoneme categories in multi-talker adaptation scenarios using a distributional learning paradigm. In Experiment 1, participants (N = 413) acquired a novel voicing boundary for a new talker and generalized this boundary to another novel talker. In a later session (1-3 days later), they were trained on a voicing boundary for the second novel talker but showed no evidence of learning the second talker and no evidence of retention for the first talker. Experiment 2 (N=355) demonstrated that the lack of retention from the second session of Experiment 1 did not derive from interference from learning in the second session. Finally, we asked if listeners individuate talkers when exposed to multiple talkers simultaneously in a distributional learning paradigm (Experiment 3, N = 113) and in a learning paradigm with feedback (Experiment 4, N = 125), and showed no evidence for talker-specific learning. Taken together, participants are likely adapting categories to the current listening environment imperfectly, but good enough. We discuss implications of our findings within the broader speech processing and learning literature.