Beiranvand, A., Sarhadi, M., Salimi Sartakhti, J.. (1404). Hybrid Fine-Tuning of Large Language Models Using LoRA: Enhancing Multi-Task Text Classification Through Knowledge Sharing. فناوری آموزش, 13(2), 417-430. doi: 10.22061/jecei.2025.11314.794
A. Beiranvand; M. Sarhadi; J. Salimi Sartakhti. "Hybrid Fine-Tuning of Large Language Models Using LoRA: Enhancing Multi-Task Text Classification Through Knowledge Sharing". فناوری آموزش, 13, 2, 1404, 417-430. doi: 10.22061/jecei.2025.11314.794
Beiranvand, A., Sarhadi, M., Salimi Sartakhti, J.. (1404). 'Hybrid Fine-Tuning of Large Language Models Using LoRA: Enhancing Multi-Task Text Classification Through Knowledge Sharing', فناوری آموزش, 13(2), pp. 417-430. doi: 10.22061/jecei.2025.11314.794
Beiranvand, A., Sarhadi, M., Salimi Sartakhti, J.. Hybrid Fine-Tuning of Large Language Models Using LoRA: Enhancing Multi-Task Text Classification Through Knowledge Sharing. فناوری آموزش, 1404; 13(2): 417-430. doi: 10.22061/jecei.2025.11314.794
Department of Computer Engineering, University of Kashan, Kashan, Iran.
تاریخ دریافت: 12 آبان 1403،
تاریخ بازنگری: 24 بهمن 1403،
تاریخ پذیرش: 12 اسفند 1403
چکیده
Background and Objectives: Large Language Models have demonstrated exceptional performance across various NLP tasks, especially when fine-tuned for specific applications. Full fine-tuning of large language models requires extensive computational resources, which are often unavailable in real-world settings. While Low-Rank Adaptation (LoRA) has emerged as a promising solution to mitigate these challenges, its potential remains largely untapped in multi-task scenarios. This study addresses this gap by introducing a novel hybrid approach that combines LoRA with an attention-based mechanism, enabling fine-tuning across tasks while facilitating knowledge sharing to improve generalization and efficiency. This study aims to address this gap by introducing a novel hybrid fine-tuning approach using LoRA for multi-task text classification, with a focus on inter-task knowledge sharing to enhance overall model performance. Methods: We proposed a hybrid fine-tuning method that utilizes LoRA to fine-tune LLMs across multiple tasks simultaneously. By employing an attention mechanism, this approach integrates outputs from various task-specific models, facilitating cross-task knowledge sharing. The attention layer dynamically prioritizes relevant information from different tasks, enabling the model to benefit from complementary insights. Results: The hybrid fine-tuning approach demonstrated significant improvements in accuracy across multiple text classification tasks. On different NLP tasks, the model showed superior generalization and precision compared to conventional single-task LoRA fine-tuning. Additionally, the model exhibited better scalability and computational efficiency, as it required fewer resources to achieve comparable or better performance. Cross-task knowledge sharing through the attention mechanism was found to be a critical factor in achieving these performance gains. Conclusion: The proposed hybrid fine-tuning method enhances the accuracy and efficiency of LLMs in multi-task settings by enabling effective knowledge sharing between tasks. This approach offers a scalable and resource-efficient solution for real-world applications requiring multi-task learning, paving the way for more robust and generalized NLP models.
