This paper proposes a novel approach to audio game design by introducing the concept of speech puzzles (spuzzles) to describe the utilisation of recorded voice for the creation of audio puzzles in ways that challenge players’ different listening modes. In the fields of audio games and audio-interactive applications, speech serves instructive, descriptive, narrative, and in some cases—in the form of hints or quizzes—gameplay purposes by addressing users through language. The suggested approach of spuzzles extends this potential by including, besides encoded meaning, the acoustic properties of sound, thus engaging the user’s causal and reduced listening modes in parallel with the semantic listening mode. An audio game consisting of four inherently different spuzzles was designed as proof of concept and tested by seven third-year students of Audiovisual Arts, who elaborated on their experience through a focus group semi-structured discussion. Despite their difficulty, the spuzzles were well accepted by most of the participants (5/7), whereas all participants agreed on their acoustic richness, need for concentration, and independence from pre-existing musical knowledge. Therefore, the authors suggest that the proposed design approach could serve as a paradigm for future research in the design of complex audio-based game mechanics.