Text generation is defined as a component of natural language processing that makes use of computational linguistics techniques to produce text that cannot be distinguished from human written text. This study aims to develop and analyse a Generative Pre-Trained Transformer 2 (GPT-2) language model to generate Sepedi phrases. The under-resourced Sepedi language is regarded as a disjunctive language. The Sepedi language orthographic representation presents challenges and has limited resources. The GPT-2 transformer requires large datasets, as well as state-of-the-art computational resources. The unstructured National Centre for Human Language Technology (NCHLT) Sepedi text dataset was used. The text generation model developed with the small dataset managed to get the lowest loss value of 2.36. The output text generated using this model produces a text that is syntactically correct with instances of grammatical errors. The model performed better than previously developed Sepedi text generation models by using transformer-based technique.