Explainability of Large Language Models (LLMs) in Providing Cybersecurity Advice

Abstract

Artificial intelligence has transformed various domains, including cybersecurity, by introducing models capable of understanding and generating human language. The novel approach of leveraging these models to provide cybersecurity advice offers significant potential yet raises concerns about their explainability and reliability. This research systematically investigates the ability of advanced language models to distinguish between defensive and offensive cybersecurity advice, examines the impact of excessive caution and political correctness on the quality of their recommendations, and provides a comprehensive framework for evaluating their performance. The findings highlight the strengths and limitations of current models, emphasizing the need for improved interpretability and practical utility in AI-driven cybersecurity solutions. By proposing specific recommendations and enhancements, the study aims to advance the development of more transparent, reliable, and effective cybersecurity tools.