This paper aims to comprehensively explore challenges and opportunities to design highly efficient Neural Network (NN) systems through Approximate Computing (AxC) techniques while ensuring fault tolerance properties. By highlighting the intrinsic conflicting goals of AxC and fault tolerance principles, the study aims to stimulate and contribute to a deeper understanding of how important it is to consider fault tolerance requirements while designing approximate-computingbased systems. This is key to developing highly efficient faulttolerant architectures for Neural Networks.