Multimodal AI-enabled Feedback Mechanism for Oral Foreign Language Proficiency: Integrating Speech Recognition and Sentiment Analysis

Yingying  He

doi:10.6180/jase.202608_31.068

Multimodal AI-enabled Feedback Mechanism for Oral Foreign Language Proficiency: Integrating Speech Recognition and Sentiment Analysis

Computer Science and Information Engineering

Yingying HeThis email address is being protected from spambots. You need JavaScript enabled to view it.

School of Foreign Languages and International Trade, Hubei University of Automotive Technology, 442002 China

Received: January 1, 2026
Accepted: March 9, 2026
Publication Date: March 27, 2026

Copyright The Author(s). This is an open access article distributed under the terms of the Creative Commons Attribution License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are cited.

Download Citation: ||https://doi.org/10.6180/jase.202608_31.068

Oral proficiency is a core competency in foreign language learning, yet the traditional foreign language classroom often faces challenges such as inadequate oral feedback, delayed evaluation, and subjective assessment bias. To address these issues, this study proposes a multimodal AI-enabled feedback mechanism that integrates automatic speech recognition (ASR) and sentiment analysis (SA) technologies. First, the ASR module extracts phonetic features (e.g., pronunciation accuracy, fluency) from learners’ oral outputs, while the SA module captures emotional cues (e.g., confidence, anxiety) through vocal prosody and textual semantics. Then, a weighted fusion algorithm is designed to integrate the two modalities of information, generating personalized and actionable feedback. To verify the effectiveness of the mechanism, an experiment was conducted with 86 foreign language learners divided into experimental and control groups. Objective evaluation indicators include ASR recognition accuracy, pronunciation error detection rate, and feedback response time; subjective indicators include learner satisfaction, perceived usefulness, and teacher evaluation consistency. Experimental results show that the proposed mechanism achieves an ASR recognition accuracy of 92.3%, a pronunciation error detection rate of 88.7%, and a feedback response time of less than two seconds. Compared with the traditional feedback method, the experimental group’s oral proficiency score increased by 15.6% on average, and the learner satisfaction rate reached 89.5%. This study provides a new technical solution for improving the efficiency and personalization of oral feedback in foreign language classrooms, and enriches the application research of multimodal AI in language education.

Keywords: Multimodal AI; Oral Foreign Language Proficiency; Speech Recognition; Sentiment Analysis; Feedback Mechanism

[1] Y.-B. Zheng, Y.-X. Zhou, X.-D. Chen, and X.-D. Ye, (2025) “The influence of large language models as collaborative dialogue partners on EFL English oral proficiency and foreign language anxiety" Computer Assisted Language Learning: 1–27. DOI: 10.1080/09588221.2025. 2453191.
[2] J.-m. Dewaele, D. Guedat-Bittighoffer, and M.-A. Dat, (2025) “Foreign language enjoyment overcomes anxiety and boredom to boost oral proficiency in the first year of English foreign language learning" International Journal of Applied Linguistics 35(1): 152–167. DOI: 10.1111/ijal.12607.
[3] R.KhanyandA.Kamalvand, (2025) “A closer look at the impact of social semiotics and smartphone photography on Iranian English learners’ oral proficiency" Computer Assisted Language Learning: 1–31. DOI: 10.1080/ 09588221.2025.2490657.
[4] H. Ahlawat, N. Aggarwal, and D. Gupta, (2025) “Automatic Speech Recognition: A survey of deep learning techniques and approaches" International Journal of Cognitive Computing in Engineering 6: 201–237. DOI: 10.1016/j.ijcce.2024.12.007.
[5] R. Nath, N. Singh, V. Katiyar, A. Srivastava, et al., (2026) “Speech Emotion Recognition Using MFCC Features and LSTM-Based Deep Learning" Indian Journal of Signal Processing (IJSP) 6(1): 1–6. DOI: 10.54105/ijsp.A1017.06010226.
[6] Z. Zhang, (2025) “Construction of an English oral pronunciation evaluation model based on deep learning algorithms" International Journal of High Speed Electronics and Systems: 2540307. DOI: 10.1142/ S0129156425403079.
[7] H. Wang and J. Zhang, (2025) “Automatic English Speaking Fluency Scoring System Based on Automatic Speech Assessment" Circuits, Systems, and Signal Processing: 1–27. DOI: 10.1007/s00034-025-03429-7.
[8] L. Judijanto, A. Aswamedhika, I. Aksan, I. Mustofa, et al., (2023) “Student sentiment analysis: implementation of artificial intelligence in improving teaching quality" Journal of Social Science Utilizing Technology 1(4): 227–238. DOI: 10.70177/jssut.v1i4.664.
[9] M. D. Pawar and R. D. Kokate, (2021) “Convolution neural network based automatic speech emotion recognition using Mel-frequency Cepstrum coefficients" Multi media Tools and Applications 80(10): 15563–15587. DOI: 10.1007/s11042-020-10329-2.
[10] J. Zhao, (2025) “Advancing english language education: Amixed-methods analysis of AI-driven tools’ impact on engagement, personalization, and learning outcomes" Education and Information Technologies 30(15): 21773–21813. DOI: 10.1007/s10639-025-13560-2.
[11] C.-T. Lu, Y.-Y. Lu, Y.-R. Lu, Y.-C. Pan, and Y.-C. Liu, (2025) “Implementation of an AI English-speaking in teractive training system using multi-model neural net works" IEEE Access 13: 132052–132066. DOI: 10.1109/ACCESS.2025.3592632.
[12] M. H. Batubara, (2025) “Harnessing Artificial Intelligence for English Language Learning: A Systematic Literature Review" Journal of Linguistics, Literature, and Language Teaching (JLLLT) 4(2): 107–122. DOI: 10.37249/jlllt.v4i2.853.
[13] S. A. Alzahrani. “The future of AI in language education: Trends and predictions”. In: Bridging global divides for transnational higher education in the AI era. IGI Global Scientific Publishing, 2025, 309–332. DOI: 10.4018/979-8-3693-7016-2.ch015.
[14] C. Wang and B. Zou, (2025) “AI versus teachers and peers: a comparative study on speaking performance, willingness to communicate, self-perceived communicative competence and foreign language anxiety" International Review of Applied Linguistics in Language Teach ing (0): DOI: 10.1515/iral-2024-0310.
[15] S. Yin, H. Li, A. A. Laghari, L. Teng, T. R. Gadekallu, and A. Almadhor, (2024) “FLSN-MVO: edge computing and privacy protection based on federated learning Siamese network with multi-verse optimization algorithm for industry 5.0" IEEE Open Journal of the Communications Society 6: 3443–3458. DOI: 10.1109/OJCOMS. 2024.3520562.
[16] M. M. H. Emon. “Leveraging AI on Social Media & Digital Platforms to Enhance English Language Teaching and Learning”. In: AI-Powered English Teach ing. IGI Global Scientific Publishing, 2026, 211–238. DOI: 10.4018/979-8-3373-1952-0.ch007.
[17] A. Aravind, M. Durairaj, P. Chitkara, Y. A. B. El Ebiary, E. Muniyandy, L. Ushasree, and M. B. Am mar, (2025) “Adaptive AI-Based Personalized Learning for Accelerated Vocabulary and Syntax Mastery in Young English Learners." International Journal of Advanced Computer Science & Applications 16(4): DOI: 10.14569/ijacsa.2025.0160467.
[18] Z. Wei, (2026) “Enhancing Cross-Cultural Critical Thinking in English Learners Through IT-Assisted Intelligent Learning Strategies Based on Artificial Intelligence" International Journal of High Speed Electronics and Systems 35(02): 2540705. DOI: 10.1142/ S0129156425407053.