Journal of Applied Science and Engineering

Published by Tamkang University Press

ESCI jase impact factor scopus logo open access rate of Scopus journal

Driver fatigue recognition using local ensemble convolutional neural network and encoding vector

 2020

 Volume 23, Issue 3

Lei, Zhao1 and Guoxin Zhang2

1School of Mechanical and Electronic Engineering, Shandong Jianzhu University, No.1000 of Fengming Road, Jinan, China

2Department of Mechanical Engineering, Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong

Received: August 07, 2019
Accepted: March 26, 2020
Publication Date: May 10, 2026

上傳圖片

Overall structure of the proposed framework.

 Copyright The Author(s). This is an open access article distributed under the terms of the Creative Commons Attribution License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are cited.

Download Citation:  BibTeX | http://dx.doi.org/10.6180/jase.202009_23(3).0001  

Download PDF

A novel approach for the detecting dynamic facial fatigue behavior of drivers is proposed in this work. First, the eye and mouth regions are extracted from a frame in a video to classify the static facial fatigue state by using a local ensemble convolutional neural network (LECNN) model. A transfer learning strategy is used to pretrain the LECNN model for improved recognition accuracy. Second, the classification results for the frames in the video are encoded and connected to construct an encoding vector that represents the facial state in the video. Finally, k-nearest neighbors classifier is adopted to classify the constructed vector and obtain the recognition result of facial fatigue. The proposed system is tested on a human fatigue dataset of images that are captured in different environments. Results demonstrate the stable and robust performance of the proposed method with approximately 96.76% accuracy.

Keywords: Driver fatigue recognition; Local ensemble convolutional neural networks; Transfer learning

  1. [1] Jaeik Jo, Sung Joo Lee, Kang Ryoung Park, Ig Jae Kim, and Jaihie Kim. Detecting driver drowsiness using feature-level fusion and user-specific classification. Expert Systems with Applications, 41(4 PART 1):1139–1152, jan 2014.
  2. [2] Bogusław Cyganek and Sławomir Gruszczyński. Hybrid computer vision system for drivers’ eye recognition and fatigue monitoring. Neurocomputing, 126:78–94, feb 2014.
  3. [3] Anirban Dasgupta, Anjith George, S. L. Happy, and Aurobinda Routray. A vision-based system for monitoring the loss of attention in automotive drivers. IEEE Transactions on Intelligent Transportation Systems, 14(4):1825–1838, dec 2013.
  4. [4] Chihang Zhao, Jie Lian, Qian Dang, and Can Tong. Classification of driver fatigue expressions by combined curvelet features and gabor features, and random subspace ensembles of support vector machines. Journal of Intelligent and Fuzzy Systems, 26(1):91–100, 2014.
  5. [5] Chihang Zhao, Yunsheng Zhang, Xiaozheng Zhang, and Jie He. Recognition of driver’s fatigue expression using Local Multiresolution Derivative Pattern. Journal of Intelligent and Fuzzy Systems, 30(1):547–560, 2016.
  6. [6] Changyuan Wang, Ting Yan, and Hongbo Jia. SpatialTemporal Feature Representation Learning for Facial Fatigue Detection. International Journal of Pattern Recognition and Artificial Intelligence, 32(12):1856018, dec 2018.
  7. [7] Rateb Jabbar, Khalifa Al-Khalifa, Mohamed Kharbeche, Wael Alhajyaseen, Mohsen Jafari, and Shan Jiang. Realtime Driver Drowsiness Detection for Android Application Using Deep Neural Networks Techniques. In Procedia Computer Science, volume 130, pages 400–407, 2018.
  8. [8] Paul Viola and Michael J. Jones. Robust Real-Time Face Detection. International Journal of Computer Vision, 57(2):137–154, may 2004.
  9. [9] Akshay Asthana, Stefanos Zafeiriou, Shiyang Cheng, and Maja Pantic. Incremental face alignment in the wild. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 1859–1866, 2014.
  10. [10] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2323, 1998.
  11. [11] Patrick Lucey, Jeffrey F. Cohn, Takeo Kanade, Jason Saragih, Zara Ambadar, and Iain Matthews. The extended Cohn-Kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition – Workshops, CVPRW 2010, pages 94–101, 2010.
  12. [12] Maja Pantic, Michel Valstar, Ron Rademaker, and Ludo Maat. Web-based database for facial expression analysis. In 2005 IEEE International Conference on Multimedia and Expo, pages 317–321, Amsterdam, Netherlands, 2005.
  13. [13] Tanja Banziger and Klaus R. Scherer. Introducing the Geneva Multimodal Emotion Portrayal (GEMEP) Corpus. A Blueprint for Affective Computing: A sourcebook and manual, pages 271–294, 2010.
  14. [14] Ian J. Goodfellow, Dumitru Erhan, Pierre Luc Carrier, Aaron Courville, Mehdi Mirza, Ben Hamner, Will Cukierski, Yichuan Tang, David Thaler, Dong Hyun Lee, Yingbo Zhou, Chetan Ramaiah, Fangxiang Feng, Ruifan Li, Xiaojie Wang, Dimitris Athanasakis, John Shawe-Taylor, Maxim Milakov, John Park, Radu Ionescu, Marius Popescu, Cristian Grozea, James Bergstra, Jingjing Xie, Lukasz Romaszko, Bing Xu, Zhang Chuang, and Yoshua Bengio. Challenges in representation learning: A report on three machine learning contests. In International Conference on neural Information Processing, pages 117–124, Daegu, korea, 2013.
  15. [15] Michael J. Lyons. Automatic classification of single facial images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(12):1357–1362, dec 1999.
  16. [16] Alex Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Toronto, ON, Canada, 2009.
  17. [17] Lei Zhao, Zengcai Wang, and Guoxin Zhang. Facial Expression Recognition from Video Sequences Based on Spatial-Temporal Motion Local Binary Pattern and Gabor Multiorientation Fusion Histogram. Mathematical Problems in Engineering, 2017, 2017.
  18. [18] Lei Zhao, Zengcai Wang, Xiaojin Wang, Yazhou Qi, Qing Liu, and Guoxin Zhang. Human fatigue expression recognition through image-based dynamic multiinformation and bimodal deep learning. Journal of Electronic Imaging, 25(5):053024, oct 2016.