In this paper, we present a new face detection scheme using deep learning and achieving state-of-the-art recognition performance using real-world datasets.  We designed and implemented a face recognition system using Principal Component Analysis (PCA) and Faster R Convolutional Neural Network (Faster R CNN). In particular, we improve the state-of-the-art Faster RCNN framework by using Principal Component Analysis (PCA) technique and Faster R CNN to detect and recognise faces in a face database.  The Principal Component Analysis (PCA) was used to extract features and dimensionality reduction from the face database, while the Faster R Convolutional Neural Network algorithm was used to identify patterns in the dataset via training the neural network. The three real-world datasets used in our experiment are ORL, Yale, and California face dataset. When implemented on the ORL face dataset, the algorithm achieved average recognition accuracy of 99%, with a recognition time of 147.72 seconds for 10 runs, and the recognition time/image was 0.3 sec/image on 400 images. The Yale face dataset achieved average recognition accuracy of 99.24% with a recognition time of 63.45 seconds for 10 runs, and the recognition time/image was 0.53 sec/image on 120 images. Finally, on California Face Database (CFD), it achieved average recognition accuracy of 99.52% with a recognition time of 226.05 seconds for 10 runs, and the recognition time/image was 0.27 sec/image on 827 images. On the CFD dataset, however, the proposed approach has excellent classification performance when the recall ratio is high. The proposed method achieves a higher recall and accuracy ratio than the Faster RCNN without PCA method. For the F-score, the proposed method achieved 0.98, which is significantly higher than the 0.95 achieved by the Faster-RCNN. This demonstrates the superiority of our model performance-wise as against state-of-the-art, both in terms of accuracy and fast recognition. Therefore our model is more efficient when compared to the latest researches done in the area of facial recognition.