Bidirectional Long and Short Time Memory Network based on Graph Neural Network for Community Student Behavior Recognition

Hui  Liu; Yang  Liu; Hui  Yang; Ying  Gao; Bing  Yan; Ziyi  Wang; Yiran  Jin

doi:10.6180/jase.202411_27(11).0001

Bidirectional Long and Short Time Memory Network based on Graph Neural Network for Community Student Behavior Recognition

Computer Science and Information Engineering

Hui Liu¹, Yang Liu²This email address is being protected from spambots. You need JavaScript enabled to view it., Hui Yang³, Ying Gao¹, Bing Yan¹, Ziyi Wang⁴, and Yiran Jin⁴

¹School of Foreign Languages, Dalian University of Technology, Dalian, China

²School of Marxism, Shenyang Jianzhu University, Shenyang, China

³Fushun Vocational Technology Institute, Fushun, China

⁴International School of Information Science and Engineering, Dalian University of Technology, Dalian, China

Received: November 5, 2023
Accepted: December 10, 2023
Publication Date: January 15, 2024

Copyright The Author(s). This is an open access article distributed under the terms of the Creative Commons Attribution License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are cited.

Download Citation: ||https://doi.org/10.6180/jase.202411_27(11).0001

With the cross-integration of artificial intelligence and deep learning in the field of education, behavior recognition technology provides a new method for students’ classroom behavior observation, which is different from the traditional one. Traditional behavior recognition algorithms cannot effectively suppress spatial background information, make full use of context information, and model global temporal correlation. Therefore, we propose a novel feature fusion method for community student behavior recognition, which combines graph neural network and bidirectional long and short time memory network. The image depth features are extracted by the graph neural network, and the feature fusion mechanism is introduced to enhance the information interaction between different convolution layers. Then the obtained depth features are input to the Bi-LSTM network to model the time information of the students’ behavior. Finally, Sigmoid function is used to classify the recognition results. Experiment results on UCF101 and HMDB51 data sets show that the proposed method has great advantages over other methods in identifying student behavior.

Keywords: Community student behavior recognition; feature fusion; graph neural network; bidirectional long and short time memory network; Sigmoid function

[1] F. Martinez, S. Taut, and K. Schaaf, (2016) “Classroom observation for evaluating and improving teaching: An international perspective" Studies in Educational Evaluation 49: 15–29. DOI: doi.org/10.1016/j.stueduc.2016.03.002.
[2] E. D. Grimm, T. Kaufman, and D. Doty, (2014) “Rethinking classroom observation" Educational Leadership 71(8): 24–29.
[3] D. Zhang, (2022) “Affective Cognition of Students’ Autonomous Learning in College English Teaching Based on Deep Learning" Frontiers in psychology 12: 808434. DOI: doi.org/10.3389/fpsyg.2021.808434.
[4] L. Hao, (2018) “The research of theoretical construction and effect of preschool wisdom education system in the background of big data [J]" Cluster Computing 5: 1–7. DOI: doi.org/10.1007/s10586-018-2102-6.
[5] L. J. Thompson and D. West, (2013) “Professional development in the contemporary educational context: Encouraging practice wisdom" Social Work Education 32(1): 118–133. DOI: doi.org/10.1080/02615479.2011.648178.
[6] J. Yu, H. Li, S.-L. Yin, and S. Karim, (2020) “Dynamic gesture recognition based on deep learning in human-tocomputer interfaces" Journal of Applied Science and Engineering 23(1): 31–38. DOI: doi.org/10.6180/jase. 202003_23(1).0004.
[7] G. Arora, A. K. Dubey, Z. A. Jaffery, and A. Rocha, (2020) “Bag of feature and support vector machine based early diagnosis of skin cancer" Neural Computing and Applications: 1–8. DOI: doi.org/10.1007/s00521-020-05212-y.
[8] H. Wang, A. Kläser, C. Schmid, and C.-L. Liu, (2013) “Dense trajectories and motion boundary descriptors for action recognition" International journal of computer vision 103: 60–79. DOI: doi.org/10.1007/s11263-012-0594-8.
[9] K. Ohnishi, M. Hidaka, and T. Harada. “Improved dense trajectory with cross streams”. In: Proceedings of the 24th ACM international conference on Multimedia. 2016, 257–261. DOI: doi.org/10.1145/2964284.2967222.
[10] S. Yin, L. Meng, J. Liu, et al., (2019) “A new apple segmentation and recognition method based on modified fuzzy C-means and hough transform" Journal of Applied Science and Engineering 22(2): 349–354. DOI: doi.org/10.6180/jase.201906_22(2).0016.
[11] W. Zhang and C. Liu. “Research on human abnormal behavior detection based on deep learning”. In: 2020 International Conference on Virtual Reality and Intelligent Systems (ICVRIS). IEEE. 2020, 973–978. DOI: 10.1109/ICVRIS51417.2020.00237.
[12] L. Wang, Y. Xiong, Z. Wang, Y. Qiao, D. Lin, X. Tang, and L. Van Gool. “Temporal segment networks: Towards good practices for deep action recognition”. In: European conference on computer vision. Springer. 2016, 20–36. DOI: doi.org/10.1007/978-3-319-46484-8_2.
[13] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. “Learning spatiotemporal features with 3d convolutional networks”. In: Proceedings of the IEEE international conference on computer vision. 2015, 4489–4497.
[14] K. Liu, W. Liu, C. Gan, M. Tan, and H. Ma. “T-C3D: Temporal convolutional 3D network for real-time action recognition”. In: Proceedings of the AAAI conference on artificial intelligence. 32. 1. 2018. DOI: doi.org/10.1609/aaai.v32i1.12333.
[15] Z. Hao, Z. Wang, D. Bai, B. Tao, X. Tong, and B. Chen, (2022) “Intelligent detection of steel defects based on improved split attention networks" Frontiers in Bioengineering and Biotechnology 9: 810876. DOI: doi.org/10.3389/fbioe.2021.810876.
[16] F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini, (2008) “The graph neural network model" IEEE transactions on neural networks 20(1): 61–80. DOI: 10.1109/TNN.2008.2005605.
[17] H. Yang. “Aligraph: A comprehensive graph neural network platform”. In: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining. 2019, 3165–3166. DOI: doi.org/10.1145/3292500.3340404.
[18] J.-Y. Xu, S. Zhang, C.-C. Wu, W.-C. Lin, and Q.-L. Yuan, (2022) “Application of an Adaptive Adjacency Matrix-Based Graph Convolutional Neural Network in Taxi Demand Forecasting" Mathematics 10(19): 3694. DOI: doi.org/10.3390/math10193694.
[19] M. Liu, H. Gao, and S. Ji. “Towards deeper graph neural networks”. In: Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. 2020, 338–348. DOI: doi.org/10.1145/3394486.3403076.
[20] Z. Hu, Y. Dong, K. Wang, K.-W. Chang, and Y. Sun. “Gpt-gnn: Generative pre-training of graph neural networks”. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2020, 1857–1867. DOI: /doi.org/10.1145/3394486.3403237.
[21] X. Song, M. Mao, and X. Qian, (2021) “Auto-metric graph neural network based on a meta-learning strategy for the diagnosis of Alzheimer’s disease" IEEE Journal of Biomedical and Health Informatics 25(8): 3141–3152. DOI: 10.1109/JBHI.2021.3053568.
[22] N. Kyurkchiev, (2022) “A note on a hypothetical piecewise smooth sigmoidal growth function: reaction network analysis, applications" International Journal of Differential Equations and Applications 21(1): 1–17.
[23] D. Tran, J. Ray, Z. Shou, S.-F. Chang, and M. Paluri, (2017) “Convnet architecture search for spatiotemporal feature learning" arXiv preprint arXiv:1708.05038: DOI: doi.org/10.48550/arXiv.1708.05038.
[24] K. Simonyan and A. Zisserman, (2014) “Two-stream convolutional networks for action recognition in videos" Advances in neural information processing systems 27:
[25] L. Wang, Y. Qiao, and X. Tang. “Action recognition with trajectory-pooled deep-convolutional descriptors”. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2015, 4305–4314.
[26] S. Ji, W. Xu, M. Yang, and K. Yu, (2012) “3D convolutional neural networks for human action recognition" IEEE transactions on pattern analysis and machine intelligence 35(1): 221–231. DOI: 10.1109/TPAMI.2012.59.
[27] D. Tran, H. Wang, L. Torresani, J. Ray, Y. LeCun, and M. Paluri. “A closer look at spatiotemporal convolutions for action recognition”. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2018, 6450–6459.
[28] M. Zolfaghari, K. Singh, and T. Brox. “Eco: Efficient convolutional network for online video understanding”. In: Proceedings of the European conference on computer vision (ECCV). 2018, 695–712.
[29] L. Sevilla-Lara, S. Zha, Z. Yan, V. Goswami, M. Feiszli, and L. Torresani. “Only time can tell: Discovering temporal data for temporal modeling”. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2021, 535–544.
[30] Y. Jiang and S. Yin, (2023) “Heterogenous-view occluded expression data recognition based on cycle-consistent adversarial network and K-SVD dictionary learning under intelligent cooperative robot environment" Computer Science and Information Systems (00): 34–34. DOI: doi.org/10.2298/CSIS221228034J.
[31] S. Yin, (2023) “Object Detection Based on Deep Learning: A Brief Review" IJLAI Transactions on Science and Engineering 1(02): 1–6.