This research aims to assess the usage of unsupervised learning in detecting financial fraud across various financial industries by identifying cognitive constructs, benefits, economic optimization, and challenges associated with fraud detection necessitating innovative approaches for effective detection. This study conducts Systematic Literature Review following PRISMA protocol for article selection of 27 journal articles published between 2010 and 2023, sourced from Scopus database. The analysis discloses that unsupervised learning has been implemented across diverse financial sectors, including online payments, insurance, and prominently in banking, especially for identifying anomalies in credit card transactions. K-Means is the most popular method used in unsupervised learning. Nevertheless, there are ongoing challenges that require solutions to ensure the efficacy of machine learning implementation, encompassing issues like class imbalance and the complexity of fraudulent activities. In theoretical terms, this research provides an understanding of cognitive concepts, benefits and applications, challenges, and practical recommendations in the use of unsupervised learning for financial fraud detection. This is useful for practical implementation, benefiting industry practitioners in selecting appropriate models with datasets that have the potential to enhance detection system accuracy and reduce financial losses due to fraud.