One of the base functions of the affective computing is facial emotion recognition (FER) which is an ability to understand human psychological state from visual information. In this paper, we propose a hybrid deep learning system that combines a CNN to extract spatial features from a detected face and an LSTM to model temporal emotions, in a real-time face detection system based on YOLOv7. The proposed CNN+LSTM system is trained and evaluated on FER-2013 (35,887 images, 7 classes) and on the test dataset it achieves an accuracy of 89.6%, a precision of 88.2%, a recall of 87.5% and an F1-score of 87.8%, which is a 5.3 percentage-point improvement over the single-frame CNN baseline. YOLOv7 face detector delivers an accuracy of 95.1% on 48 FPS with 93.8% mAP@0.5. Ablation studies validate the distinct contribution of the LSTM temporal module (+5.3%) and of the detection backbone of YOLOv7 (+2.5% with respect to YOLOv5). Facial expression recognition is an integral part of Human-Computer Interaction (HCI) and Affective Computing (AC) which is essential to understand human emotions and respond to them accordingly. In this project, the development and implementation of an automated emotion recognition system using traditional computer vision technique and latest deep learning approach is presented. Firstly, several classical approaches for emotion classification are investigated, including Support Vector Machine (SVM) and hand-crafted feature extractors, such as Local Binary Pattern (LBP) and Histogram of Oriented Gradient (HOG). Despite their effectiveness in some cases, these methods do not scale well to out-of-the-box situations because they cannot apply the input to a variety of facial expressions and environmental conditions. To overcome these drawbacks, the project incorporates cutting-edge deep learning methods, including Convolutional Neural Networks (CNNs) known for their proficiency in extracting hierarchical features from raw image data. Face detection is done in real-time with the help of two object detection models that are widely used in the field of deep learning, namely YOLOv5 and YOLOv7, both of which are considered to be very fast and accurate. Facial regions detected are then fed into CNN models for accurate emotion classification. The system is tested on public datasets and proven to withstand lighting changes, occlusion and cross-cultural differences in expression. Furthermore, privacy and ethical issues are taken into account, highlighting the significance of responsible AI usage. The aim of this research is to contribute to the development of emotion-aware systems that are finding applications in several domains such as healthcare monitoring, online education, smart surveillance, customer service, and interactive entertainment, where understanding the emotions of humans has a positive impact on the user experience and the intelligence of the system.
Keywords : Facial Emotion Recognition, CNN, LSTM, YOLOv7, FER-2013, Affective Computing, Temporal Modelling.
Authors : M Anjan Kumar , P Viswanatha Reddy , T Gopikrishna , MD Akbar
Title : Real-Time Facial Emotion Recognition Using Deep CNN and YOLO-Based Face Detection: A Hybrid CNN–LSTM Framework
Volume/Issue : 2025;2( 2 ( April - June ))
Page No : 14 - 23