Drowsiness Detection System Through Eye and Mouth Analysis

Bey-Ee Belle Lim - Multimedia University, 63100, Cyberjaya, Selangor, Malaysia
Kok Why Ng - Multimedia University, 63100, Cyberjaya, Selangor, Malaysia
Sew Lai Ng - Multimedia University, 63100, Cyberjaya, Selangor, Malaysia


Citation Format:



DOI: http://dx.doi.org/10.62527/joiv.7.4.2288

Abstract


Traffic jams are one of the serious issues in many developed countries. After the pandemic, many employees were allowed to travel interstate to work. This contributes to more severe jams, especially in the capital and nearby states. Long-distance driving and congestion can easily make the drivers sleepy and thus lead to traffic accidents. This paper aims to study and analyze facial cues to detect early symptoms of drowsy driving. The proposed method employs a deep learning approach, utilizing ensemble CNNs and Dlib's 68 landmark face detectors to analyze the facial cues. The analyzed symptoms include the frequency of eyes opened or closed and yawning or no yawning. Three individual CNN models and an ensemble CNN structure are built for the classification of the eyes and mouth yawn. The model training and validation accuracy graph and training loss and validation loss graph are plotted to verify that the models have not been overfitted. The ensemble CNN models achieved an approximate accuracy of 97.4% from the eyes and 96.5% from the mouth. It outperforms the other pre-trained models. The proposed system can immediately alert the driver and send text drowsy messages and emails to the third party, ensuring timely intervention to prevent accidents. The proposed method can be integrated into vehicles and transportation systems to ensure driver's safety. It can also be applied to monitor the driving behavior of those who drive long distances

Keywords


Drowsiness Detection System; Driver Monitoring; Facial Expression Recognition; Ensemble CNN; Image Processing.

Full Text:

PDF

References


R. Jabbar, M. Shinoy, M. Kharbeche, K. Al-Khalifa, M. Krichen, and K. Barkaoui, "Driver Drowsiness Detection Model Using Convolutional Neural Networks Techniques for Android Application," Jan. 2020, [Online]. Available: http://arxiv.org/abs/2002.03728

A. U. I. Rafid, A. I. Chowdhury, A. R. Niloy, and N. Sharmin, "A Deep Learning Based Approach for Real-time Driver Drowsiness Detection," in 2021 5th International Conference on Electrical Engineering and Information and Communication Technology, ICEEICT 2021, Institute of Electrical and Electronics Engineers Inc., 2021. doi: 10.1109/ICEEICT53905.2021.9667944.

M. Elham Walizad, M. Hurroo, and D. Sethia, "Driver Drowsiness Detection System using Convolutional Neural Network," in 2022 6th International Conference on Trends in Electronics and Informatics, ICOEI 2022 - Proceedings, Institute of Electrical and Electronics Engineers Inc., 2022, pp. 1073–1080. doi: 10.1109/ICOEI53556.2022.9777182.

Y. Suresh, R. Khandelwal, M. Nikitha, M. Fayaz, and V. Soudhri, "Driver Drowsiness Detection using Deep Learning," in Proceedings - 2nd International Conference on Smart Electronics and Communication, ICOSEC 2021, Institute of Electrical and Electronics Engineers Inc., 2021, pp. 1526–1531. doi: 10.1109/ICOSEC51865.2021.9591957.

V. R. Reddy Chirra, S. R. Uyyala, and V. K. Kishore Kolli, "Deep CNN: A machine learning approach for driver drowsiness detection based on eye state," Revue d'Intelligence Artificielle, vol. 33, no. 6, pp. 461–466, 2019, doi: 10.18280/ria.330609.

T. Faisal, I. Negassi, G. Goitom, M. Yassin, A. Bashir, and M. Awawdeh, "Systematic development of real-time driver drowsiness detection system using deep learning," IAES International Journal of Artificial Intelligence, vol. 11, no. 1, pp. 148–160, Mar. 2022, doi: 10.11591/ijai.v11.i1.pp148-160.

M. Dua, Shakshi, R. Singla, S. Raj, and A. Jangra, "Deep CNN models-based ensemble approach to driver drowsiness detection," Neural Comput Appl, vol. 33, no. 8, pp. 3155–3168, Apr. 2021, doi: 10.1007/s00521-020-05209-7.

M. Ahmed, S. Masood, M. Ahmad, and A. A. Abd El-Latif, "Intelligent Driver Drowsiness Detection for Traffic Safety Based on Multi CNN Deep Model and Facial Subsampling," IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 10, pp. 19743–19752, Oct. 2022, doi: 10.1109/TITS.2021.3134222.

S. Park, F. Pan, S. Kang, and C. D. Yoo, "Driver drowsiness detection system based on feature representation learning using various deep networks."

V. Kumar, S. Sharma, and Ranjeet, "Driver drowsiness detection using modified deep learning architecture," Evol Intell, 2022, doi: 10.1007/s12065-022-00743-w.

J. M. Guo and H. Markoni, "Driver drowsiness detection using hybrid convolutional neural network and long short-term memory," Multimed Tools Appl, vol. 78, no. 20, pp. 29059–29087, Oct. 2019, doi: 10.1007/s11042-018-6378-6.

I. Journal and R. V K, "IRJET-Drowsy Driving Detection via Hybrid Deep Learning Cite this paper Drowsy Driving Detection via Hybrid Deep Learning," International Research Journal of Engineering and Technology, 2020, [Online]. Available: www.irjet.net

M. ÖZTÜRK, A. KÜÇÜKMANİSA, and O. URHAN, "Drowsiness Detection System Based on Machine Learning Using Eye State," Balkan Journal of Electrical and Computer Engineering, vol. 10, no. 3, pp. 258–263, Jul. 2022, doi: 10.17694/bajece.1028110.

S. Mehta, P. Mishra, A. J. Bhatt, and P. Agarwal, "AD3S: Advanced Driver Drowsiness Detection System using Machine Learning," in 2019 Fifth International Conference on Image Information Processing (ICIIP), IEEE, Nov. 2019, pp. 108–113. doi: 10.1109/ICIIP47207.2019.8985844.

C. Y. Lin, P. Chang, A. Wang, and C. P. Fan, "Machine Learning and Gradient Statistics Based Real-Time Driver Drowsiness Detection," in 2018 IEEE International Conference on Consumer Electronics-Taiwan, ICCE-TW 2018, Institute of Electrical and Electronics Engineers Inc., Aug. 2018. doi: 10.1109/ICCE-China.2018.8448747.

S. Dey, S. A. Chowdhury, S. Sultana, Md. A. Hossain, M. Dey, and S. K. Das, "Real Time Driver Fatigue Detection Based on Facial Behaviour along with Machine Learning Approaches," in 2019 IEEE International Conference on Signal Processing, Information, Communication & Systems (SPICSCON), IEEE, Nov. 2019, pp. 135–140. doi: 10.1109/SPICSCON48833.2019.9065120.

Q. Cheng, W. Wang, X. Jiang, S. Hou, and Y. Qin, "Assessment of Driver Mental Fatigue Using Facial Landmarks," IEEE Access, vol. 7, pp. 150423–150434, 2019, doi: 10.1109/ACCESS.2019.2947692.

C. B. S. Maior, M. J. das C. Moura, J. M. M. Santana, and I. D. Lins, "Real-time classification for autonomous drowsiness detection using eye aspect ratio," Expert Syst Appl, vol. 158, Nov. 2020, doi: 10.1016/j.eswa.2020.113505.

E. Magán, M. P. Sesmero, J. M. Alonso-Weber, and A. Sanchis, "Driver Drowsiness Detection by Applying Deep Learning Techniques to Sequences of Images," Applied Sciences (Switzerland), vol. 12, no. 3, Feb. 2022, doi: 10.3390/app12031145.

M. Hashemi, A. Mirrashid, and A. Beheshti Shirazi, "Driver Safety Development: Real-Time Driver Drowsiness Detection System Based on Convolutional Neural Network," SN Comput Sci, vol. 1, no. 5, Sep. 2020, doi: 10.1007/s42979-020-00306-9.

X.-H. Lam, K.-W. Ng, Y.-J. Yoong, and S.-B. Ng, "WBC-based segmentation and classification on microscopic images: a minor improvement," F1000Res, vol. 10, p. 1168, Nov. 2021, doi: 10.12688/f1000research.73315.1.

J.-S. Ang, "STOCK MARKET PREDICTION USING DEEP LEARNING APPROACH," 2022.

Y. Lim, K.-W. Ng, P. Naveen, and S.-C. Haw, "Emotion Recognition by Facial Expression and Voice: Review and Analysis," Journal of Informatics and Web Engineering, vol. 1, no. 2, pp. 45–54, Sep. 2022, doi: 10.33093/jiwe.2022.1.2.4.

Adrian Rosebrock, "Facial landmarks with dlib, OpenCV, and Python." Accessed: Oct. 26, 2023. [Online]. Available: https://pyimagesearch.com/2017/04/03/facial-landmarks-dlib-opencv-python/

Adrian Rosebrock, "Convolutional Neural Networks (CNNs) and Layer Types." Accessed: Oct. 26, 2023. [Online]. Available: https://pyimagesearch.com/2021/05/14/convolutional-neural-networks-cnns-and-layer-types/

R. M. Salman, M. Rashid, R. Roy, M. M. Ahsan, and Z. Siddique, "Driver Drowsiness Detection Using Ensemble Convolutional Neural Networks on YawDD," Dec. 2021, [Online]. Available: http://arxiv.org/abs/2112.10298

Z. Zhao, N. Zhou, L. Zhang, H. Yan, Y. Xu, and Z. Zhang, "Driver Fatigue Detection Based on Convolutional Neural Networks Using EM-CNN," Comput Intell Neurosci, vol. 2020, 2020, doi: 10.1155/2020/7251280.

M. F. Shakeel, N. A. Bajwa, A. M. Anwaar, A. Sohail, A. Khan, and Haroon-ur-Rashid, "Detecting Driver Drowsiness in Real Time Through Deep Learning Based Object Detection," in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Springer Verlag, 2019, pp. 283–296. doi: 10.1007/978-3-030-20521-8_24.

Radovan and Fusek, “MRL Eye Dataset.” Accessed: Oct. 26, 2023. [Online]. Available: http://mrl.cs.vsb.cz/eyedataset

DAVID VAZQUEZ, “Yawn Dataset,” 2021, Accessed: Oct. 26, 2023. [Online]. Available: https://www.kaggle.com/datasets/davidvazquezcic/yawn-dataset