Journal of Applied Science and Engineering

Published by Tamkang University Press

ESCI jase impact factor scopus logo open access rate of Scopus journal

Research on Optimal Control Strategies Based on Deep Reinforcement Learning under Differential Algebraic Equation Constraints

 2026

 Volume 33

Guoxing Si

Basic Department of Zhengzhou Vocational College of Industrial Safety, Zhengzhou, Henan, 451192, China

Received: March 12, 2026
Accepted: June 19, 2026
Publication Date: July 25, 2026

上傳圖片

Illustration of ADSCO for topology control

 Copyright The Author(s). This is an open access article distributed under the terms of the Creative Commons Attribution License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are cited.

Download Citation:  BibTeX | http://dx.doi.org/10.6180/jase.202610_33.058  

Download PDF

Optimal control of differential-algebraic equations represents a major challenge due to the intricate relationship between the state dynamics and the algebraic constraints, especially for large-scale power transmission systems with discrete topology actions. The limitations of conventional controllers and unconstrained deep reinforcement learning algorithms lie in the failure to ensure the feasibility, safety, and long-term stability of the system while
maintaining the DAE simulation consistency. To overcome the limitations of the existing solutions, the present work proposes a structured DRL approach, which integrates the Attentive Differentiable Structure for Control Optimization and the Policy Embedded Gradient Field Navigation algorithms. The ADSCO method represents the grid observations and the topology actions using a shared latent space, while the attentive energy modeling ensures the state-action compatibility using feasibility-aware signals from the DAE constraints. The PEGFN method refines the policy outputs using the constraint-informed gradient field defined on the relaxed action simplex. The effectiveness of the proposed method was tested using the L2RPN and RL2Grid test cases and proved to give better results compared to both previous methods and the current best reinforcement learning algorithms. Specifically speaking about the paper under discussion, ’simulation consistency with respect to DAEs’ is the property that ensures that all control measures that are implemented within the proposed framework are consistent with the dynamics of the power system represented by DAEs.

Keywords: Optimal control (OC), DRL, differential-algebraic systems, interpretable AI, intelligent systems

  1. [1] H. Zhou, S. Zhang, J. Peng, S. Zhang, J. Li, H. Xiong, and W. Zhang. “Informer: Beyond efficient transformer for long sequence time-series forecasting”. In: Proceedings of the AAAI conference on artificial intelligence. 35. 12. 2021, 11106–11115. DOI: https://doi.org/10.1609/aaai.v35i12.17325.
  2. [2] A. Angelopoulos, E. Candes, and R. J. Tibshirani, (2023) “Conformal pid control for time series prediction” Advances in neural information processing systems 36: 23047–23074. DOI: https://ojs.aaai.org/index.php/AAAI/article/view/17325.
  3. [3] L. Shen and J. Kwok. “Non-autoregressive conditional diffusion models for time series prediction”. In: International Conference on Machine Learning. PMLR. 2023, 31016–31029. DOI: https://proceedings.mlr.press/v202/shen23d.html.
  4. [4] M. S. Sial, Q. Fu, and T. Vianna Brugni, (2022) “Allocation of Interline Power Flow Controller-Based Congestion Management in Deregulated Power System” Advances in Engineering and Intelligence Systems 1(01): 65–74. DOI: https://doi.org/10.22034/aeis.2022.148478.
  5. [5] X. Wen and W. Li, (2023) “Time series prediction based on LSTM-attention-LSTM model” IEEE access 11: 48322–48331. DOI: https://doi.org/10.1109/ACCESS.2023.3276628.
  6. [6] L. Ren, Z. Jia, Y. Laili, and D. Huang, (2023) “Deep learning for time-series prediction in IIoT: progress, challenges, and prospects” IEEE transactions on neural networks and learning systems 35(11): 15072–15091. DOI: https://doi.org/10.1109/TNNLS.2023.3291371.
  7. [7] Y. Li, K. Wu, and J. Liu, (2023) “Self-paced ARIMA for robust time series prediction” Knowledge-Based Systems 269: 110489. DOI: https://www.sciencedirect.com/science/article/pii/S0950705123002393.
  8. [8] Y. Wang, (2024) “Enhancing transient stability of DFIG-based wind turbine systems using deep learning-controlled resistance-based fault current limiters” Advances in Engineering and Intelligence Systems 3(01): 10–22. DOI: https://doi.org/10.22034/aeis.2024.423904.1139.
  9. [9] D. M. Durairaj and B. K. Mohan, (2022) “A convolutional neural network based approach to financial time series prediction” Neural Computing and Applications 34(16): 13319–13337. DOI: https://doi.org/10.1007/s00521-022-07143-2.
  10. [10] L. Yin, L. Wang, T. Li, S. Lu, J. Tian, Z. Yin, X. Li, and W. Zheng, (2023) “U-Net-LSTM: time series-enhanced lake boundary prediction model” Land 12(10): 1859. DOI: https://doi.org/10.3390/land12101859.
  11. [11] C. Yu, F. Wang, Z. Shao, T. Sun, L. Wu, and Y. Xu. “Dsformer: A double sampling transformer for multivariate time series long-term prediction”. In: Proceedings of the 32nd ACM international conference on information and knowledge management. 2023, 3062–3072. DOI: https://doi.org/10.1145/3583780.361485.
  12. [12] W. Zheng and G. Chen, (2021) “An accurate GRU-based power time-series prediction approach with selective state updating and stochastic optimization” IEEE Transactions on Cybernetics 52(12): 13902–13914. DOI: https://doi.org/10.1109/TCYB.2021.3121312.
  13. [13] B. Lindemann, T. Müller, H. Vietz, N. Jazdi, and M. Weyrich, (2021) “A survey on long short-term memory networks for time series prediction” Procedia Cirp 99: 650–655. DOI: https://doi.org/10.1016/j.procir.2021.03.088.
  14. [14] R. Chandra, S. Goyal, and R. Gupta, (2021) “Evaluation of deep learning models for multi-step ahead time series prediction” Ieee Access 9: 83105–83123. DOI: https://doi.org/10.1109/ACCESS.2021.3085085.
  15. [15] J. Fan, K. Zhang, Y. Huang, Y. Zhu, and B. Chen, (2023) “Parallel spatio-temporal attention-based TCN for multivariate time series prediction” Neural Computing and Applications 35(18): 13109–13118. DOI: https://link.springer.com/article/10.1007/s00521-021-05958-z.
  16. [16] M. Hou, C. Xu, Z. Li, Y. Liu, W. Liu, E. Chen, and J. Bian. “Multi-granularity residual learning with confidence estimation for time series prediction”. In: Proceedings of the ACM Web Conference 2022. 2022, 112–121. DOI: https://doi.org/10.1145/3485447.351205.
  17. [17] H. V. Dudukcu, M. Taskiran, Z. G. C. Taskiran, and T. Yildirim, (2023) “Temporal Convolutional Networks with RNN approach for chaotic time series prediction” Applied soft computing 133: 109945. DOI: https://doi.org/10.1016/j.asoc.2022.109945.
  18. [18] I. Amalou, N. Mouhni, and A. Abdali, (2022) “Multivariate time series prediction by RNN architectures for energy consumption forecasting” Energy Reports 8: 1084–1091. DOI: https://doi.org/10.1016/j.egyr.2022.07.139.
  19. [19] Y. Xiao, H. Yin, Y. Zhang, H. Qi, Y. Zhang, and Z. Liu, (2021) “A dual-stage attention-based Conv-LSTM network for spatio-temporal correlation and multivariate time series prediction” International Journal of Intelligent Systems 36(5): 2036–2057. DOI: https://doi.org/10.1002/int.22370.
  20. [20] J. Wang, W. Jiang, Z. Li, and Y. Lu, (2021) “A new multi-scale sliding window LSTM framework (MSSW-LSTM): a case study for GNSS time-series prediction” Remote Sensing 13(16): 3328. DOI: https://doi.org/10.3390/rs13163328.
  21. [21] M. Xu, M. Han, C. P. Chen, and T. Qiu, (2018) “Recurrent broad learning systems for time series prediction” IEEE transactions on cybernetics 50(4): 1405–1417. DOI: https://doi.org/10.1109/TCYB.2018.2863020.
  22. [22] W. Zheng and J. Hu, (2022) “Multivariate time series prediction based on temporal change information learning method” IEEE Transactions on Neural Networks and Learning Systems 34(10): 7034–7048. DOI: https://doi.org/10.1109/TNNLS.2021.3137178.
  23. [23] Y. Sun, S. Dang, W. Fang, W. Huang, and Y. Xu, (2025) “Design and analysis of electricity anomaly detection and diagnosis models based on generative adversarial networks” International Journal of High Speed Electronics and Systems 34(04): 2540330. DOI: https://doi.org/10.1142/S0129156425403304.
  24. [24] W. R. Moskolai, W. Abdou, A. Dipanda, and Kolyang, (2021) “Application of deep learning architectures for satellite image time series prediction: A review” Remote Sensing 13(23): 4822. DOI: https://doi.org/10.3390/rs13234822.
  25. [25] Z. Karevan and J. A. Suykens, (2020) “Transductive LSTM for time-series prediction: An application to weather forecasting” Neural Networks 125: 1–9. DOI: https://doi.org/10.1016/j.neunet.2019.12.030.
  26. [26] S. Wang, H. Wu, X. Shi, T. Hu, H. Luo, L. Ma, J. Zhang, and J. Zhou. “Timemixer: Decomposable multiscale mixing for time series forecasting”. In: International conference on learning representations. 2024. 2024, 38626–38652. DOI: https://iclr.cc/paper_files/paper/2024/hash/a7ac8a21e5a27e7ab31a5f42a0117bdb-Abstract-Conference.html.
  27. [27] A. Wachi, X. Shen, and Y. Sui, (2024) “A survey of constraint formulations in safe reinforcement learning” arXiv preprint arXiv:2402.02025: DOI: https://doi.org/10.48550/arXiv.2402.02025.
  28. [28] W. Zhao, T. He, R. Chen, T. Wei, and C. Liu, (2023) “State-wise safe reinforcement learning: A survey” arXiv preprint arXiv:2302.03122: DOI: https://doi.org/10.48550/arXiv.2302.03122.
  29. [29] H. Wang, S. Bo, S. M. Fazeli, A. Yavari, B. Liu, Q. Zhao, and J. Liu, (2026) “Constrained stabilization control via safe reinforcement learning and control Lyapunov value function” Control Engineering Practice 174: 107065. DOI: https://doi.org/10.1016/j.conengprac.2026.107065.
  30. [30] S. Bandyopadhyay and S. Bhasin, (2025) “Lagrangian-based online safe reinforcement learning for state-constrained systems” Automatica 179: 112458. DOI: https://doi.org/10.1016/j.automatica.2025.112458.
  31. [31] M. Opmeer and O. Staffans, (2024) “On linear quadratic optimal control and algebraic Riccati equations for infinite-dimensional differential-algebraic equations” DAE Panel 2: DOI: https://doi.org/10.52825/dae-p.v2i.1388.
  32. [32] A. Ilchmann, T. Reis, L. Biegler, S. Campbell, C. Führer, R. März, S. Trenn, P. Kunkel, R. Riaza, V. H. Linh, et al. Differential-algebraic equations forum. 2017. Springer, 2013. DOI: https://doi.org/10.1007/978-3-642-27555-5.
  33. [33] H. Gernandt and T. Reis, (2024) “Linear-quadratic optimal control for abstract differential-algebraic equations” IFAC-PapersOnLine 58(17): 310–315. DOI: https://doi.org/10.1016/j.ifacol.2024.10.187.
  34. [34] J. Wang, Z. Peng, X. Wang, C. Li, and J. Wu, (2020) “Deep fuzzy cognitive maps for interpretable multivariate time series prediction” IEEE transactions on fuzzy systems 29(9): 2647–2660. DOI: https://doi.org/10.1109/TFUZZ.2020.3005293.
  35. [35] S. Karasu and A. Altan, (2022) “Crude oil time series prediction model based on LSTM network with chaotic Henry gas solubility optimization” Energy 242: 122964. DOI: https://doi.org/10.1016/j.energy.2021.122964.
  36. [36] J. Wen, J. Yang, B. Jiang, H. Song, and H. Wang, (2020) “Big data driven marine environment information forecasting: a time series prediction network” IEEE transactions on fuzzy systems 29(1): 4–18. DOI: https://doi.org/10.1109/TFUZZ.2020.3012393.
  37. [37] M. A. Morid, O. R. L. Sheng, and J. Dunbar, (2023) “Time series prediction using deep learning methods in healthcare” ACM Transactions on Management Information Systems 14(1): 1–29. DOI: https://doi.org/10.1145/3531326.
  38. [38] H. Zhang, J. Zeng, J. Ma, Y. Fang, C. Ma, Z. Yao, and Z. Chen, (2021) “Time series prediction of microseismic multi-parameter related to rockburst based on deep learning” Rock Mechanics and Rock Engineering 54(12): 6299–6321. DOI: https://doi.org/10.1007/s00603-021-02614-9.
  39. [39] H. Widiputra, A. Mailangkay, and E. Gautama, (2021) “Multivariate CNN-LSTM model for multiple parallel financial time-series prediction” Complexity 2021(1): 9903518. DOI: https://doi.org/10.1155/2021/9903518.
  40. [40] M. Hammad Saeed, S. Iqbal, M. Sohail, and U. Akram, (2023) “An optimal adaptive control of DC-Microgrid with the aim of accurate current sharing and voltage regulation according to the load variation and gray-wolf optimization” Advances in Engineering and Intelligence Systems 2(03): 50–60. DOI: https://doi.org/10.22034/aeis.2023.406515.1110.
  41. [41] S. Nurpatsha, (2025) “A New Analysis of Web Customer Service Text Classification of Alexa Virtual Assistant Commands Using a Deep Learning Model” Journal of Artificial Intelligence and System Modelling 3(02): 76–90. DOI: https://doi.org/10.22034/jaism.2025.515660.1120.