Convolutional Neural Network is known to achieve high accuracy in solving classification, recognition, and detection problems. In a real-time environment, time is an important factor of consideration. Even though most CNN-based architectures achieved considerably high accuracy, they are still slow even with high-end hardware. Therefore, this paper compares the time-accuracy tradeoff between two recent CNN-based learning architectures in detecting a driver's fatigue status. In our work, we define fatigue based on the rate of eye blinking. We developed a proof of concept systems, and evaluate the systems based on accuracy and detection speed. The accuracy and speed of both learning architectures were trained and tested using the Closed Eyes in the Wild (CEW) containing 1,193 closed eyes images and 1,232 opened eyes images. As MobileNet-SSD and YOLO v3 were pre-trained using a general COCO dataset, they were further configured and fine-tuned to optimize the results based on the CEW datasets, The results showed that YOLO v3 has slightly higher meanAveragePrecision(mAP) than MobileNet-SSD but slower detection speed(ms), while MobileNet-SSD proved that it has much faster speed but still maintaining high accuracy. The results of the research also showed that there is a trade-off between speed and accuracy which there was a loss of accuracy to obtain faster speed. This research also proved that lightweight MobileNet-SSD can minimize the accuracy loss to gain speed. The accuracy of the MobileNet-SSD learning model was still considered high and the detection speed was far higher than YOLO v3 learning model. Therefore, MobileNetSSD learning model was selected to have the best speed and accuracy trade-off in this research.