Regular Articles

Human fatigue expression recognition through image-based dynamic multi-information and bimodal deep learning

[+] Author Affiliations
Lei Zhao, Xiaojin Wang, Yazhou Qi, Qing Liu, Guoxin Zhang

Shandong University, School of Mechanical Engineering, Vehicle Engineering Research Institute, No. 17923 of Jingshi Road, Jinan 250061, China

Zengcai Wang

Shandong University, School of Mechanical Engineering, Vehicle Engineering Research Institute, No. 17923 of Jingshi Road, Jinan 250061, China

Shandong University, School of Mechanical Engineering, Ministry of Education, Key Laboratory of High Efficiency and Clean Mechanical Manufacture, No. 17923 of Jingshi Road, Jinan 250061, China

J. Electron. Imaging. 25(5), 053024 (Oct 06, 2016). doi:10.1117/1.JEI.25.5.053024
History: Received May 14, 2016; Accepted September 15, 2016
Text Size: A A A

Abstract.  Human fatigue is an important cause of traffic accidents. To improve the safety of transportation, we propose, in this paper, a framework for fatigue expression recognition using image-based facial dynamic multi-information and a bimodal deep neural network. First, the landmark of face region and the texture of eye region, which complement each other in fatigue expression recognition, are extracted from facial image sequences captured by a single camera. Then, two stacked autoencoder neural networks are trained for landmark and texture, respectively. Finally, the two trained neural networks are combined by learning a joint layer on top of them to construct a bimodal deep neural network. The model can be used to extract a unified representation that fuses landmark and texture modalities together and classify fatigue expressions accurately. The proposed system is tested on a human fatigue dataset obtained from an actual driving environment. The experimental results demonstrate that the proposed method performs stably and robustly, and that the average accuracy achieves 96.2%.

Figures in this Article
© 2016 SPIE and IS&T

Citation

Lei Zhao ; Zengcai Wang ; Xiaojin Wang ; Yazhou Qi ; Qing Liu, et al.
"Human fatigue expression recognition through image-based dynamic multi-information and bimodal deep learning", J. Electron. Imaging. 25(5), 053024 (Oct 06, 2016). ; http://dx.doi.org/10.1117/1.JEI.25.5.053024


Access This Article
Sign in or Create a personal account to Buy this article ($20 for members, $25 for non-members).

Some tools below are only available to our subscribers or users with an online account.

Related Content

Customize your page view by dragging & repositioning the boxes below.

Related Book Chapters

Topic Collections

PubMed Articles
Advertisement
  • Don't have an account?
  • Subscribe to the SPIE Digital Library
  • Create a FREE account to sign up for Digital Library content alerts and gain access to institutional subscriptions remotely.
Access This Article
Sign in or Create a personal account to Buy this article ($20 for members, $25 for non-members).
Access This Proceeding
Sign in or Create a personal account to Buy this article ($15 for members, $18 for non-members).
Access This Chapter

Access to SPIE eBooks is limited to subscribing institutions and is not available as part of a personal subscription. Print or electronic versions of individual SPIE books may be purchased via SPIE.org.