Deep learning (DL) is an exciting field of interest for many researchers and business. Due to a massive leap in DL based research, many domains like Business, science and government sectors make use of DL for various applications. This work puts forward the importance of DL and its application in a few critical electrical segments. Initially, an introduction to Artificial Intelligence (AI) and Machine Learning (ML) is presented. Then the need for DL and the popular architectures, algorithms and frameworks used are presented. A summary of different techniques used in DL is outlined, and finally, a review on the application of deep learning techniques in some popular electrical applications is presented. Five critical electrical applications, namely identification of bearing faults, hot spots on the surface of PV panels, insulator faults, an inspection of power lines and Electric vehicles have been considered for review in this work. The primary aim of this work is to present chronologically, a survey of different areas in which it applies DL along with their architectures, frameworks and techniques to provide a deeper understanding of DL for widespread use in real-time applications.