Journal of Electrical and Computer Engineering Innovations (JECEI)
مقاله 6 ، دوره 14، شماره 1 ، مهر 2026، صفحه 55-72 2.6 M )
نوع مقاله: Original Research Paper
شناسه دیجیتال (DOI): 10.22061/jecei.2025.11563.815 نویسندگان
Mahdi Shahbazi Khojasteh ؛ Armin Salimi Badr*
تاریخ دریافت : 29 بهمن 1402 ،
تاریخ بازنگری : 26 اردیبهشت 1404 ،
تاریخ پذیرش : 16 خرداد 1404
چکیده Background and Objectives: Unmanned Aerial Vehicles (UAVs) face significant challenges in navigating narrow passages within GPS-denied environments due to sensor and computational limitations. While deep reinforcement learning (DRL) has improved navigation, many methods rely on costly sensors like depth cameras or LiDAR. This study addresses these issues using a vision-based DRL framework with a monocular camera for autonomous UAV navigation.Methods: We propose a DRL-based navigation system utilizing Proximal Policy Optimization (PPO). The system processes a stack of grayscale monocular images to capture short-term temporal dependencies, approximating the partially observable environment. A custom reward function encourages trajectory optimization by assigning higher rewards for staying near the passage center while penalizing further distances. The navigation system is evaluated in a 3D simulation environment under a GPS-denied scenario.Results: The proposed method achieves a high success rate, surpassing 97% in challenging narrow passages. The system demonstrates superior learning efficiency and robust generalization to new configurations compared to baseline methods. Notably, using stacked frames mitigates computational overhead while maintaining policy effectiveness.Conclusion: Our vision-based DRL approach enables autonomous UAV navigation in GPS-denied environments with reduced sensor requirements, offering a cost-effective and efficient solution. The findings highlight the potential of monocular cameras paired with DRL for real-world UAV applications such as search and rescue and infrastructure inspection. Future work will extend the framework to obstacle avoidance and general trajectory planning in dynamic environments.کلیدواژهها
Deep Reinforcement Learning ؛ Obstacle Avoidance ؛ Trajectory Planning ؛ Autonomous Navigation ؛ Unmanned Aerial Vehicle
مراجع
[1] N. Elmeseiry, N. Alshaer, T. Ismail, "A detailed survey and future directions of Unmanned Aerial Vehicles (UAVs) with potential applications," Aerospace, 8(12): 363, 2021.
[2] M. A. Tahir, I. Mir, T. U. Islam, "A review of UAV platforms for autonomous applications: Comprehensive analysis and future directions," IEEE Access, 11: 52540-52554, 2023.
[3] S. A. H. Mohsan, M. A. Khan, F. Noor, I. Ullah, M. H. Alsharif, "Towards the Unmanned Aerial Vehicles (UAVs): A comprehensive review," Drones, 6(6): 147, 2022.
[4] M. Javaid, A. Haleem, S. Rab, R. P. Singh, R. Suman, "Sensors for daily life: A review," Sens. Int., 2: 100121, 2021.
[5] D. J. Yeong, G. Velasco-Hernandez, J. Barry, J. Walsh, "Sensor and sensor fusion technology in autonomous vehicles: A review," Sensors, 21(6): 2140, 2021.
[6] K. Telli et al., "A comprehensive review of recent research trends on Unmanned Aerial Vehicles (UAVs)," Systems, 11(8): 400, 2023.
[7] S. Campbell et al., "Sensor technology in autonomous vehicles : A review," in Proc. 2018 29th Irish Signals and Systems Conference (ISSC): 1-4, 2018.
[8] Y. Lu, Z. Xue, G. S. Xia, L. Zhang, "A survey on vision-based UAV navigation," Geo-spatial Inf. Sci., 21(1): 21-32, 2018.
[9] L. He, N. Aouf, B. Song, "Explainable deep reinforcement learning for UAV autonomous path planning," Aerosp. Sci. Technol., 118: 107052, 2021.
[10] M. Kim, J. Kim, M. Jung, H. Oh, "Towards monocular vision-based autonomous flight through deep reinforcement learning," Expert Syst. Appl., 198: 116742, 2022.
[11] J. Li, X. Xiong, Y. Yan, Y. Yang, "A survey of indoor UAV obstacle avoidance research," IEEE Access, 11: 51861-51891, 2023.
[12] S. Rezwan, W. Choi, "Artificial intelligence approaches for UAV navigation: Recent advances and future challenges," IEEE Access, 10: 26320-26339, 2022.
[13] B. Mahdipour, S. H. Zahiri, I. Behravan, "An intelligent two and three dimensional path planning, based on a metaheuristic method," J. Electr. Comput. Eng. Innovations, 13(1): 93-116, 2025.
[14] S. Y. Choi, D. Cha, "Unmanned aerial vehicles using machine learning for autonomous flight; state-of-the-art," Adv. Rob., 33(6): 265-277, 2019.
[15] A. Carrio, C. Sampedro, A. Rodriguez-Ramos, P. Campoy, "A review of deep learning methods and applications for unmanned aerial vehicles," J. Sens., 2017(1): 3296874, 2017.
[16] R. P. Padhy, S. Verma, S. Ahmad, S. K. Choudhury, P. K. Sa, "Deep neural network for autonomous UAV navigation in indoor corridor environments," Procedia Comput. Sci., 133: 643-650, 2018.
[17] J. Gu et al., "Recent advances in convolutional neural networks," Pattern Recognit., 77: 354-377, 2018.
[18] Y. Chang, Y. Cheng, U. Manzoor, J. Murray, "A review of UAV autonomous navigation in GPS-denied environments," Rob. Auton. Syst., 170: 104533, 2023.
[19] R. S. Sutton, "Reinforcement learning: An introduction," A Bradford Book, MIT Press, Cambridge, MA, 2018.
[20] H. Taheri, S. Rasoul Hosseini, M. A. Nekoui, "Deep reinforcement learning with enhanced ppo for safe mobile robot navigation," arXiv e-prints, p. arXiv:2405.16266, 2024.
[21] A. S. Sadr, M. S. Khojasteh, H. Malek, A. Salimi-Badr, "An efficient planning method for autonomous navigation of a wheeled-robot based on deep reinforcement learning," in Proc. 2022 12th International Conference on Computer and Knowledge Engineering (ICCKE): 136-141, 2022.
[22] S. Mashhouri, M. Rahmati, Y. Borhani, E. Najafi, "Reinforcement learning based sequential controller for mobile robots with obstacle avoidance," in Proc. 2022 8th International Conference on Control, Instrumentation and Automation (ICCIA): 1-5, 2022.
[23] S. Sabzekar, M. Samadzad, A. Mehditabrizi, A. N. Tak, "A deep reinforcement learning approach for UAV path planning incorporating vehicle dynamics with acceleration control," Unmanned Syst., 12(03): 477-498, 2024.
[24] H. S. M. Mahalegi, A. Farhadi, G. Molnár, E. Nagy, "Enhancing UAV autonomous navigation in indoor environments using reinforcement learning and convolutional neural networks," in Proc. 2024 IEEE 22nd Jubilee International Symposium on Intelligent Systems and Informatics (SISY): 000091-000100, 2024.
[25] M. Ramezani, M. A. Amiri Atashgah, A. Rezaee, "A fault-tolerant multi-agent reinforcement learning framework for unmanned aerial vehicles–unmanned ground vehicle coverage path planning," Drones, 8(10): 537, 2024.
[26] P. R. Gervi, A. Harati, S. K. Ghiasi-Shirazi, "Vision-based obstacle avoidance in drone navigation using deep reinforcement learning," in Proc. 2021 11th International Conference on Computer Engineering and Knowledge (ICCKE): 363-368, 2021.
[27] M. S. Khojasteh, A. Salimi-Badr, "Autonomous quadrotor path planning through deep reinforcement learning with monocular depth estimation," IEEE Open J. Veh. Technol., 6: 34-51, 2025.
[28] A. P. Kalidas, C. J. Joshua, A. Q. Md, S. Basheer, S. Mohan, S. Sakri, "Deep reinforcement learning for vision-based navigation of UAVs in avoiding stationary and mobile obstacles," Drones, 7(4): 245, 2023.
[29] A. Pachauri, V. More, P. Gaidhani, N. Gupta, "Autonomous Ingress of a UAV through a window using Monocular Vision," arXiv preprint arXiv:1607.07006, 2016.
[30] T. Bera, A. Sinha, A. K. Sadhu, R. Dasgupta, "Vision based autonomous quadcopter navigation through narrow gaps using visual servoing and monocular SLAM," in Proc. 2019 Sixth Indian Control Conference (ICC): 25-30, 2019.
[31] J. Kumar, Himanshu, H. Kandath, P. Agrawal, "Vision based micro-uav navigation through narrow passages," in Proc. 2023 11th International Conference on Control, Mechatronics and Automation (ICCMA): 97-102, 2023 .
[32] F. Valenti, D. Giaquinto, L. Musto, A. Zinelli, M. Bertozzi, A. Broggi, "Enabling computer vision-based autonomous navigation for unmanned aerial vehicles in cluttered gps-denied environments," in Proc. 2018 21st International Conference on Intelligent Transportation Systems (ITSC): 3886-3891, 2018.
[33] S. Veerawal, S. Bhushan, M. R. Mansharamani, B. Sharma, "Vision based autonomous drone navigation through enclosed spaces," in Proc. Computer Vision and Image Processing: 104-115, 2021.
[34] R. Xiao, H. Du, C. Xu, W. Wang, "An efficient real-time indoor autonomous navigation and path planning system for drones based on RGB-D sensor," in Proc. 2019 Chinese Intelligent Automation Conference: 32-44, 2020.
[35] A. G. Caldeira, J. V. R. Vasconcelos, M. Sarcinelli-Filho, A. S. Brandão, "UAV path-following strategy for crossing narrow passages," in Proc. 2020 International Conference on Unmanned Aircraft Systems (ICUAS): 26-31, 2020.
[36] Y. Xue, W. Chen, "A UAV navigation approach based on deep reinforcement learning in large cluttered 3D environments," IEEE Trans. Veh. Technol., 72(3): 3001-3014, 2023.
[37] S. Aggarwal, N. Kumar, "Path planning techniques for unmanned aerial vehicles: A review, solutions, and challenges," Comput. Commun., 149: 270-299, 2020.
[38] D. Debnath, F. Vanegas, J. Sandino, A. F. Hawary, F. Gonzalez, "A review of UAV path-planning algorithms and obstacle avoidance methods for remote sensing applications," Remote Sens., 16(21): 4019, 2024.
[39] B. Y. Li, H. Lin, H. Samani, L. Sadler, T. Gregory, B. Jalaian, "On 3D autonomous delivery systems: Design and development," in Proc. 2017 International Conference on Advanced Robotics and Intelligent Systems (ARIS): 1-6, 2017.
[40] A. Puente-Castro, D. Rivero, A. Pazos, E. Fernandez-Blanco, "A review of artificial intelligence applied to path planning in UAV swarms," Neural Comput. Appl., 34(1): 153-170, 2022.
[41] J. Gao, Y. Zheng, K. Ni, Q. Mei, B. Hao, L. Zheng, "Fast path planning for firefighting UAV based on a-star algorithm," J. Phys. Conf. Ser., 2029(1): 012103, 2021.
[42] L. Liu, X. Wang, X. Yang, H. Liu, J. Li, P. Wang, "Path planning techniques for mobile robots: Review and prospect," Expert Syst. Appl., 227: 120254, 2023.
[43] W. Zu, G. Fan, Y. Gao, Y. Ma, H. Zhang, H. Zeng, "Multi-UAVs cooperative path planning method based on improved rrt algorithm," in Proc. 2018 IEEE International Conference on Mechatronics and Automation (ICMA): 1563-1567, 2018.
[44] Y. Guo, X. Liu, X. Liu, Y. Yang, W. Zhang, "FC-RRT*: An improved path planning algorithm for UAV in 3D complex environment," ISPRS Int. J. of Geo-Inf., 11(2): 112, 2022.
[45] K. N. McGuire, G. C. H. E. de Croon, K. Tuyls, "A comparative study of bug algorithms for robot navigation," Rob. Auton. Syst., 121: 103261, 2019.
[46] M. Vazirpanah, S. H. Attarzadeh-Niaki, A. Salimi-Badr, "ROS-based co-simulation for formal cyber-physical robotic system design," in Proc. 2022 27th International Computer Conference, Computer Society of Iran (CSICC): 1-5, 2022.
[47] N. Mahdian, S. H. Attarzadeh-Niaki, A. Salimi-Badr, "A systematic embedded software design flow for robotic applications," in Proc. 2021 11th International Conference on Computer Engineering and Knowledge (ICCKE): 217-222, 2021.
[48] T. Elmokadem, A. V. Savkin, "Towards fully autonomous UAVs: A survey," Sensors, 21(18): 6223, 2021.
[49] S. Y. Shin, Y. W. Kang, Y. G. Kim, "Automatic drone navigation in realistic 3D landscapes using deep reinforcement learning," in Proc. 2019 6th International Conference on Control, Decision and Information Technologies (CoDIT): 1072-1077, 2019.
[50] Y. Yu, X. Si, C. Hu, J. Zhang, "A review of recurrent neural networks: LSTM cells and network architectures," Neural Comput., 31(7): 1235-1270, 2019.
[51] S. Chehelgami, E. Ashtari, M. A. Basiri, M. Tale Masouleh, A. Kalhor, "Safe deep learning-based global path planning using a fast collision-free path generator," Rob. Auton. Syst., 163: 104384, 2023.
[52] R. J. Alitappeh, N. Mahmoudi, M. R. Jafari, A. Foladi, "Autonomous robot navigation: Deep learning approaches for line following and obstacle avoidance," in Proc. 2024 20th CSI International Symposium on Artificial Intelligence and Signal Processing (AISP): 1-6, 2024.
[53] L. O. Rojas-Perez, J. Martinez-Carranza, "DeepPilot: A CNN for autonomous drone racing," Sensors, 20(16): 4524, 2020.
[54] S. Daftry, S. Zeng, J. A. Bagnell, M. Hebert, "Introspective perception: Learning to predict failures in vision systems," in Proc. 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS): 1743-1750, 2016.
[55] A. V. R. Katkuri, H. Madan, N. Khatri, A. S. H. Abdul-Qawy, K. S. Patnaik, "Autonomous UAV navigation using deep learning-based computer vision frameworks: A systematic literature review," Array, 23: 100361, 2024.
[56] Z. Xue, T. Gonsalves, "Vision based drone obstacle avoidance by deep reinforcement learning," AI, 2(3): 366-380, 2021.
[57] S. S. Mousavi, M. Schukat, E. Howley, "Deep reinforcement learning: An overview," in Proc. SAI Intelligent Systems Conference (IntelliSys): 426- 440, 2016.
[58] F. AlMahamid, K. Grolinger, "Autonomous unmanned aerial vehicle navigation using reinforcement learning: A systematic review," Eng. Appl. Artif. Intell., 115: 105321, 2022.
[59] A. K. Shakya, G. Pillai, S. Chakrabarty, "Reinforcement learning algorithms: A brief survey," Expert Syst. Appl., 231: 120495, 2023.
[60] A. Singla, S. Padakandla, S. Bhatnagar, "Memory-based deep reinforcement learning for obstacle avoidance in UAV with limited environment knowledge," IEEE Trans. Intell. Transport Syst., 22(1): 107-118, 2021.
[61] O. Walker, F. Vanegas, F. Gonzalez, S. Koenig, "A deep reinforcement learning framework for UAV navigation in indoor environments," in Proc. 2019 IEEE Aerospace Conference: 1-14, 2019.
[62] Y. Chen, N. González-Prelcic, R. W. Heath, "Collision-Free UAV Navigation with a Monocular Camera Using Deep Reinforcement Learning," in Proc. 2020 IEEE 30th International Workshop on Machine Learning for Signal Processing (MLSP): 1-6, 2020.
[63] C. Wang, J. Wang, Y. Shen, X. Zhang, "Autonomous navigation of UAVs in large-scale complex environments: A deep reinforcement learning approach," IEEE Trans. Veh. Technol., 68(3): 2124-2136, 2019.
[64] F. Garcia, E. Rachelson, "Markov decision processes," in Markov Decision Processes in Artificial Intelligence: 1-38, 2013.
[65] M. Hausknecht, P. Stone, "Deep recurrent q-learning for partially observable MDPs," in Proc. 2015 AAAI Fall Symposium: 9, 2015.
[66] H. Kurniawati, "Partially observable Markov decision processes and robotics," Annu. Rev. Control Rob. Auton. Syst., 5: 253-277, 2022.
[67] L. Graesser and W. L. Keng, Foundations of deep reinforcement learning: theory and practice in Python. Addison-Wesley Professional, 2019.
[68] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, "Proximal policy optimization algorithms," arXiv preprint arXiv:1707.06347, 2017.
[69] R. Hoseinnezhad, "A comprehensive review of deep learning techniques in mobile robot path planning: Categorization and analysis," Appl. Sci., 15(4): 2179, 2025.
[70] T. Czarnecki, M. Stawowy, A. Kadłubowski, "Cost-effective autonomous drone navigation using reinforcement learning: simulation and real-world validation," Appl. Sci., 15(1): 179, 2025.
[71] W. Skarka, R. Ashfaq, "Hybrid machine learning and reinforcement learning framework for adaptive UAV obstacle avoidance," Aerosp., 11(11): 870, 2024.
[72] K. Arulkumaran, M. P. Deisenroth, M. Brundage, A. A. Bharath, "Deep reinforcement learning: a brief survey," IEEE Signal Process. Mag., 34(6): 26-38, 2017.
[73] J. Schulman, P. Moritz, S. Levine, M. Jordan, P. Abbeel, "High-dimensional continuous control using generalized advantage estimation," arXiv preprint arXiv:1506.02438, 2015.
[74] L. Engstrom et al., "Implementation matters in deep rl: A case study on ppo and trpo," in Proc. ICLR 2020 Conference Blind Submission, 2020.
[75] M. Andrychowicz et al., "What matters for on-policy deep actor-critic methods? a large-scale study," in Proc. ICLR 2021 Conference, 2021.
[76] S. Shah, D. Dey, C. Lovett, A. Kapoor, "AirSim: High-fidelity visual and physical simulation for autonomous vehicles," Cham, 2018: Springer International Publishing, in Field and Service Robotics, pp. 621-635.
[77] B. Kabas, "Autonomous UAV navigation via deep reinforcement learning using PPO," in Proc. 2022 30th Signal Processing and Communications Applications Conference (SIU): 1-4, 2022.
آمار
تعداد مشاهده مقاله: 461
تعداد دریافت فایل اصل مقاله: 98
