In the modern digital landscape, government agencies globally are shifting services online to enhance transparency and public engagement. However, the vast digital content can be daunting for citizens seeking information. Addressing this, our research evaluates the efficacy of Large Language Models (LLMs), like ChatGPT, in the public sector, highlighting their potential in extracting relevant insights and optimizing information navigation. Our approach integrates non-parametric data from various sources focusing on information posted on three irish websites; the government publications, health services, and the Citizens Information websites, using retrieval-augmented models. Empirical evaluations show that the llama2 model, with $13$ billion parameters, achieves up to 90% for government publication releases and up to 96.12% for health information enhancement when complemented with retrieval augmentation, with other models also showing substantial improvements. These results emphasize the transformative potential of retrieval-augmented frameworks in keeping LLMs updated with the evolving public information domain.