Structure Learning for Deep Neural Networks with Competitive Synaptic Pruning

Ahmadi, A.; Mahboobi Esfanjani, R.

doi:10.22061/jecei.2024.11017.758

تعداد نشریات	11
تعداد شماره‌ها	237
تعداد مقالات	2,398
تعداد مشاهده مقاله	3,830,613
تعداد دریافت فایل اصل مقاله	2,775,223

	Structure Learning for Deep Neural Networks with Competitive Synaptic Pruning
Journal of Electrical and Computer Engineering Innovations (JECEI)
مقاله 15، دوره 13، شماره 1، فروردین 2025، صفحه 189-196 اصل مقاله (939.83 K)
نوع مقاله: Original Research Paper
شناسه دیجیتال (DOI): 10.22061/jecei.2024.11017.758
نویسندگان
A. Ahmadi؛ R. Mahboobi Esfanjani^*
Department of Electrical Engineering, Sahand University of Technology, Tabriz, Iran.
تاریخ دریافت: 05 مرداد 1403، تاریخ بازنگری: 25 مهر 1403، تاریخ پذیرش: 10 آبان 1403
چکیده
Background and Objectives: A predefined structure is usually employed for deep neural networks, which results in over- or underfitting, heavy processing load, and storage overhead. Training along with pruning can decrease redundancy in deep neural networks; however, it may lead to a decrease in accuracy. Methods: In this note, we provide a novel approach for structure optimization of deep neural networks based on competition of connections merged with brain-inspired synaptic pruning. The efficiency of each network connection is continuously assessed in the proposed scheme based on the global gradient magnitude criterion, which also considers positive scores for strong and more effective connections and negative scores for weak connections. But a connection with a weak score is not removed quickly; instead, it is eliminated when its net score reaches a predetermined threshold. Moreover, the pruning rate is obtained distinctly for each layer of the network. Results: Applying the suggested algorithm to a neural network model of a distillation column in a noisy environment demonstrates its effectiveness and applicability. Conclusion: The proposed method, which is inspired by connection competition and synaptic pruning in the human brain, enhances learning speed, preserves accuracy, and reduces costs due to its smaller network size. It also handles noisy data more efficiently by continuously assessing network connections
کلیدواژه‌ها
Deep Neural Networks؛ Synaptic Pruning؛ Distillation Column؛ PID Tuning

مراجع
[1] Z. Allen-Zhu, Y. Li, Y. Liang, "Learning and overparameterized neural networks, going beyond two layers," Adv. Neural Inf. Process. Syst., 32, 2019. [2] C. Cortes, X. Gonzalvo, V. Kuznetsov, M. Mohri, S. Yang, "AdaNet: Adaptive structural learning of artificial neural networks," in Proc. 34th International Conference on Machine Learning, 70: 874-883, 2017. [3] O. A. Montesinos L., A. Montesinos L., J. C. Montesinos, Multivariate Statistical Machine Learning Methods for Genomic Prediction, Springer, 2022. [4] S. Li, T. Hoefler, "Chimera: efficiently training large-scale neural networks with bidirectional pipelines," in Proc. International Conference for High Performance Computing, Networking, Storage and Analysis, 2021. [5] V. Sze, Y. H. Chen, T. J. Yang, J. Emer, "Efficient processing of deep neural networks: A tutorial and survey," arXiv preprint arXiv:1703.09039, 2017. [6] S. A. Janowsky, "Pruning versus clipping in neural networks," Phys. Rev. A, 39(12): 6600-6603, 1989. [7] G. Chechik, I. Meilijson, E. Ruppin, "Neuronal regulation: a biologically plausible mechanism for efficient synaptic pruning in development," Neurocomputing, 26-27: 633-639, 1999. [8] M. V. Johnston et al., "Plasticity and injury in the developing brain," Dev. Neurosci., 31: 1-10, 2009. [9] G. Hinton, O. Vinyals, J. Dean, "Distilling knowledge in a neural network," arXiv preprint arXiv:1503.02531, 2015. [10] S. Han, J. Pool, J. Tran, W. J. Dally, "Learning both weights and connections for efficient neural networks," Adv. Neural Inf. Process. Syst., 2015: 1135–1143, 2015. [11] P. Molchanov et al., "Importance estimation for neural network pruning," arXiv preprint arXiv:1906.10771, 2019. [12] P. Molchanov et al., "Pruning convolutional neural networks for resource-efficient inference," in Proc. ICLR 2017, 2017. [13] Y. LeCun et al., "Optimal brain damage," Adv. Neural Inf. Process. Syst., 2, 1990. [14] B. Hassibi, D. G. Stork, "Second-order derivatives for network pruning: Optimal Brain Surgery," Adv. Neural Inf. Process. Syst., 5: 164-171, 1993. [15] Y. He et al., "Filter pruning via geometric median for deep convolutional neural network acceleration," in Proc. CVPR 2019: 4340-4349, 2019. [16] R. Yu et al., "NISP: pruning networks using neuron importance score propagation," in Proc. IEEE CVPR 2018: 9194-9203, 2018. [17] H. Li et al., "Pruning filters for efficient convolutional neural networks," arXiv preprint arXiv:1608.08710, 2016. [18] Y. He et al., "Soft filter pruning for accelerating deep convolutional neural networks," arXiv preprint arXiv:1808.06866, 2018. [19] N. Srivastava et al., "Dropout: A simple way to prevent neural networks from overfitting," J. Mach. Learn. Res., 15: 1929-1958, 2014. [20] Z. Liu et al., "Dropout reduces underfitting," in Proc. 40th International Conference on Machine Learning, 202, 2023. [21] X. Sun et al., "MeProp: Sparsified back propagation for accelerated deep learning with reduced overfitting," arXiv preprint arXiv:1706.06197, 2017. [22] S. A. Mirjalili, Evolutionary Algorithms and Neural Networks, Springer, 2019. [23] M. Kotyrba et al., "The influence of genetic algorithms on learning possibilities of artificial neural networks," Computers, 2022. [24] F. Zhao et al., "Towards a brain-inspired developmental neural network by adaptive synaptic pruning," in Proc. ICONIP 2017, 2017. [25] A. Ahmadi, B. Mashoufi, "A new optimized approach for artificial neural network training using genetic algorithms and parallel processing," Int. Rev. Comput. Softw., 7(5): 1828-6003, 2012. [26] M. Morini et al., "Strategies and tools for studying microglial-mediated synapse elimination and refinement," Front. Immunol., 2021. [27] R. P. Guidorzi et al., "The range error test in the structural identification of linear multivariable systems," IEEE Trans. Autom. Control, AC-27: 1044-1054, 1982. [28] A. Saha and S. Patra, "Modeling and control of distillation column using ANFIS," J. Autom.Control, 51: 51-59, 2021. [29] H. Zhao, J. Zhang, M. Wang, "Modeling and optimization of distillation column processes using neural networks and genetic algorithms," IEEE Access, 7: 76345-76354, 2019. [30] A. Singh, P. Gupta, R. Verma, “Hybrid ANFIS and neural network-based approach for fault detection in industrial distillation processes,” Appl. Soft Comput., 96, 106703.
آمار تعداد مشاهده مقاله: 329 تعداد دریافت فایل اصل مقاله: 278

سامانه مدیریت نشریات علمی. طراحی و پیاده سازی از سیناوب

پیوندهای مفید

آمار

Structure Learning for Deep Neural Networks with Competitive Synaptic Pruning