Monitoring complex engineering systems is an important countermeasure in managing the risk of faulty events. Observing the response of each process flow will avoid further damages in the production cycle. Both fault-tolerant approach that bears with faulty events and scheduled maintenance that helps to reduce tool wearing are deeply involved in condition-based monitoring methods implemented in factories. Thus, identification of faulty equipment is need to avoid major breakdown in the production system. A classification framework shows good performance in classifying faulty events, but a labelled dataset is usually financially consuming. Machine learning (ML) techniques have become a prospective tool in the unsupervised fault detection (UFD) approach to prevent total failures in complex engineering system. However, the efficiency of UFD applications, on the other hand, is determined by the selected ML method. This paper presents a systematic literature review of ML methods applied for UFD, highlighting the methods explored in this field and the success of today's state-of-the-art machine learning techniques. This review focuses on the Scopus scientific database and provides a useful information on ML techniques, challenges and opportunities, and new research works in the UFD field.