آمار

تعداد نشریات11
تعداد شماره‌ها234
تعداد مقالات2,383
تعداد مشاهده مقاله3,802,783
تعداد دریافت فایل اصل مقاله2,763,682
Gheitasi, A., Farsi, H., Mohamadzadeh, S.. (1403). Hybrid Convolutional Neural Network with Domain adaptation for Sketch based Image Retrieval. فناوری آموزش, 12(2), 497-510. doi: 10.22061/jecei.2024.10778.735
A. Gheitasi; H. Farsi; S. Mohamadzadeh. "Hybrid Convolutional Neural Network with Domain adaptation for Sketch based Image Retrieval". فناوری آموزش, 12, 2, 1403, 497-510. doi: 10.22061/jecei.2024.10778.735
Gheitasi, A., Farsi, H., Mohamadzadeh, S.. (1403). 'Hybrid Convolutional Neural Network with Domain adaptation for Sketch based Image Retrieval', فناوری آموزش, 12(2), pp. 497-510. doi: 10.22061/jecei.2024.10778.735
Gheitasi, A., Farsi, H., Mohamadzadeh, S.. Hybrid Convolutional Neural Network with Domain adaptation for Sketch based Image Retrieval. فناوری آموزش, 1403; 12(2): 497-510. doi: 10.22061/jecei.2024.10778.735

Hybrid Convolutional Neural Network with Domain adaptation for Sketch based Image Retrieval

Journal of Electrical and Computer Engineering Innovations (JECEI)
دوره 12، شماره 2، مهر 2024، صفحه 497-510    اصل مقاله (1.32 M)
نوع مقاله: Original Research Paper
شناسه دیجیتال (DOI): 10.22061/jecei.2024.10778.735
نویسندگان
A. Gheitasi؛ H. Farsi؛ S. Mohamadzadeh*
Department of Electrical and Computer Engineering, University of Birjand, Birjand, Iran.
تاریخ دریافت: 03 فروردین 1403،  تاریخ بازنگری: 28 خرداد 1403،  تاریخ پذیرش: 09 تیر 1403 
چکیده
Background and Objectives: Freehand sketching is an easy-to-use but effective instrument for computer-human connection. Sketches are highly abstract to the domain gap, that exists between the intended sketch and real image. In addition to appearance information, it is believed that shape information is also very efficient in sketch recognition and retrieval.
Methods: In the realm of machine vision, comprehending Freehand Sketches has grown more crucial due to the widespread use of touchscreen devices. In addition to appearance information, it is believed that shape information is also very efficient in sketch recognition and retrieval. The majority of sketch recognition and retrieval methods utilize appearance information-based tactics. A hybrid network architecture comprising two networks—S-Net (Sketch Network) and A-Net (Appearance Network)—is shown in this article under the heading of hybrid convolution. These subnetworks, in turn, describe appearance and shape information. Conversely, a module known as the Conventional Correlation Analysis (CCA) technique module is utilized to match the range and enhance the sketch retrieval performance to decrease the range gap distance. Finally, sketch retrieval using the hybrid Convolutional Neural Network (CNN) and CCA domain adaptation module is tested using many datasets, including Sketchy, Tu-Berlin, and Flickr-15k. The final experimental results demonstrated that compared to more sophisticated methods, the hybrid CNN and CCA module produced high accuracy and results.
Results: The proposed method has been evaluated in the two fields of image classification and Sketch Based Image Retrieval (SBIR). The proposed hybrid convolution works better than other basic networks. It achieves a classification score of 84.44% for the TU-Berlin dataset and 82.76% for the sketchy dataset. Additionally, in SBIR, the proposed method stands out among methods based on deep learning, outperforming non-deep methods by a significant margin.
Conclusion: This research presented the hybrid convolutional framework, which is based on deep learning for pattern recognition. Compared to the best available methods, hybrid network convolution has increased recognition and retrieval accuracy by around 5%. It is an efficient and thorough method which demonstrated valid results in Sketch-based image classification and retrieval on TU-Berlin, Flickr 15k, and sketchy datasets.
کلیدواژه‌ها
Sketch Based Image Retrieval (SBIR)؛ Hybrid CNN؛ Domain Adaptation؛ Deep Learning
مراجع
[1]          D. Birari, D. Hiran, V. Narawade, “Survey on sketch based image and data retrieval,” in Proc. 2nd International Conference on Communications and Cyber Physical Engineering (ICCCE 2019): 285-290, 2020.

[2]          A. Chaudhuri, A. K. Bhunia, Y. Z. Song, A. Dutta, “Data-free sketch-based image retrieval,” arXiv preprint arXiv:2303.07775, 2023.

[3]          A. Sain, A. K. Bhunia, Y. Yang, T. Xiang, Y. Z. Song, “Stylemeup: Towards style-agnostic sketch-based image retrieval,” in Proc. the IEEE/CVF Conference on Computer Vision and Pattern Recognition,  2021.

[4]          P. Xu, T. M. Hospedales, Q. Yin, Y. Zh. Song, T. Xiang, L. Wang, “Deep learning for free-hand sketch: A survey,” IEEE Trans. Pattern Anal. Mach. Intell., 45(1): 285-312, 2023.

[5]          Z. Hossein-Nejad, H. Agahi, A. Mahmoodzadeh, “Remote sensing image registration based on a geometrical model matching,” J. Inf. Syst. Telecommun. (JIST), 5(36): 41, 2021.

[6]          R. Hu, S. James, T. Wang, J. Collomosse, “Markov random fields for sketch based video retrieval,” in Proc. the 3rd ACM conference on International Conference on Multimedia Retrieval (ICMR): 279-286, 2013.

[7]          Y. Li, W. Li, “A survey of sketch-based image retrieval,” Mach. Vision Appl., 29(7): 1083-1100, 2018.

[8]          S. Mohamadzadeh, S. Pasban, J. Zeraatkar-Moghadam, A. K. Shafiei, “Parkinson’s disease detection by using feature selection and sparse representation,” J. Med. Biol. Eng., 41(4): 412-421, 2021.

[9]          X. Zhang, Y. Huang, Q. Zou, Y. Pei, R. Zhang, S. Wang, “A hybrid convolutional neural network for sketch recognition,” Pattern Recognit. Lett., 130: 73-82, 2020.

[10]        P. Xu, T. M. Hospedales, Q. Yin, Y. Zh. Song, T. Xiang, L. Wang, “Deep learning for free-hand sketch: A survey,” IEEE Trans. Pattern Anal. Mach. Intell., 45(1): 285-312, 2020.

[11]        P. Xu, Y. Huang, T. Yuan, K. Pang, Y. Zh. Song, T. Xiang, T. M. Hospedales, Zh. Ma, J. Guo, “Sketchmate: Deep hashing for million-scale human sketch retrieval,” in Proc. the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2018): 8090-8098, 2018.

[12]        A. Dutta, Z. Akata, “Semantically tied paired cycle consistency for zero-shot sketch-based image retrieval,” in Proc. the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2019): 5089-5098, 2019.

[13]        H. Yu, M. Huang, J. J. Zhang, “Domain adaptation problem in sketch based image retrieval,” ACM Trans. Multimidia Comput. Commun. Appl., 19(3): 1-17, 2022.

[14]        A. K. Bhunia, A. Sain, P. H. Shah, A. Gupta, P. N. Chowdhury, T. Xiang, Y. Zh. Song, “Adaptive fine-grained sketch-based image retrieval,” in Proc. 17th European Conference Computer Vision (ECCV 2022), Part XXXVII: 163-181, 2022.

[15]        E. Shechtman, M. Irani, “Matching local self-similarities across images and videos,” in Proc. 2007 IEEE Conference on Computer Vision and Pattern Recognition: 1-8, 2007.

[16]        J. M. Saavedra, “Rst-shelo: Sketch-based image retrieval using sketch tokens and square root normalization,” Multimedia Tools Appl., 76(1): 931-951, 2017.

[17]        R. G. Schneider, T. Tuytelaars, “Sketch classification and classification-driven analysis using fisher vectors,” ACM Trans. Graphics (TOG), 33(6): 1-9, 2014.

[18]        V. Sabeti, “An improved approach to blind image steganalysis using an overlapping blocks idea,” J. Electr. Comput. Eng. Innovations, 11(2): 263-276, 2023.

[19]        M. Imani, “Target detection using multispectral images, A case study: Wheat detection in Chenaran County in Iran,” J. Electr. Comput. Eng. Innovations, 9(1): 11-24, 2020.

[20]        M. Rezaei, M. Rezaei, “Foreground-back ground segmentation using k-means clustering algorithm and support vector machine,” J. Inf. Syst. Telecommun. (JIST), 1(41): 65, 2023.

[21]        S. Fooladi, H. Farsi, S. Mohamadzadeh, “Segmenting the lesion area of brain tumor using convolutional neural networks and fuzzy K-means clustering,” Int. J. Eng., 36(8): 1556-1568, 2023.

[22]        A. Sain, A. K. Bhunia, Y. Yang, T. Xiang, Y. Zh. Song, “Cross-modal hierarchical modelling for fine-grained sketch based image retrieval,” arXiv preprint arXiv:2007.15103, 2020.

[23]        A. K. Bhunia, P. N. Chowdhury, A. Sain, Y. Yang, T. Xiang, Y. Zh. Song,  “More photos are all you need: Semi-supervised learning for fine-grained sketch based image retrieval,” in Proc. the IEEE/CVF Conference on Computer Vision and Pattern Recognition: 4247-4256, 2021.

[24]        A. K. Bhunia, S. Koley, A. F. U. R. Khilji, A. Sain, P. N. Chowdhury, T. Xiang, Y. Zh. Song, “Sketching without worrying: Noise-tolerant sketch-based image retrieval,” in Proc. the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2019): 999-1008, 2022.

[25]        A. Gheitasi, H. Farsi, S. Mohamadzadeh, “Estimation of hand skeletal postures by using deep convolutional neural networks,” Int. J. Eng., 33(4): 552-559, 2020.

[26]        Q. Yu, F. Liu, Y. Zh. Song, T. Xiang, T. M. Hospedales, Ch. Ch. Loy, “Sketch me that shoe,” in Proc. the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2016): 799-807, 2016.

[27]        C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, A. Rabinovich, “Going deeper with convolutions,” in Proc. the IEEE Conference on Computer Vision and Pattern Recognition: 1-9, 2015.

[28]        N. Kumar, R. Ahmed, V. B. Honnakasturi, S. Sowmya Kamath, V. Mayya, “Sketch-based image retrieval using convolutional neural networks based on feature adaptation and relevance feedback,” in Proc. International Conference on Emerging Applications of Information Technology: 103-113, 2022.

[29]        H. Zhang, S. Liu, Ch. Zhang, W. Ren, R. Wang, X. Cao, “Sketchnet: Sketch classification with web images,” in Proc. the IEEE Conference on Computer Vision and Pattern Recognition: 1105-1113, 2016.

[30]        S. D. Bhattacharjee, J. Yuan, W. Hong, X, Ruan, “Query adaptive instance search using object sketches,” in Proc. the 24th ACM International Conference on Multimedia: 1306-1315, 2016.

[31]        F. Lin, M. Li, D. Li, T. Hospedales, Y. Zh. Song, Y. Qi, “Zero-shot everything sketch-based image retrieval,” in Proc. the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2023): 23349-23358, 2023.

[32]        R. Chavhan, Zero-Shot Sketch Based Image Retrieval, Indian Institute of Technology Bombay, 2021.

[33]        J. Y. He, X. Wu, Y. G. Jiang, B. Zhao, Q. Peng, “Sketch recognition with deep visual-sequential fusion model,” in Proc. the 25th ACM International Conference on Multimedia: 448-456, 2017.

[34]        H. Zhao, M. Liu, M. Li, “Feature fusion and metric learning network for zero-shot sketch-based image retrieval,” Entropy, 25(3): 502, 2023.

[35]        E. Askari, S. Motamed, “Computational model for image processing in the minds of people with visual agnosia using fuzzy cognitive map,” J. Inf. Syst. Telecommun. (JIST), 2(42): 102, 2023.

[36]        M. Lin, Q. Chen, S. Yan, “Network in network,” arXiv preprint arXiv:1312.4400, 2013.

[37]        F. Wang, L. Kang, Y. Li, “Sketch-based 3d shape retrieval using convolutional neural networks,” in Proc. the IEEE Conference on Computer Vision and Pattern Recognition: 1875-1883, 2015.

[38]        K. Chatfield, K. Simonyan, A. Vedaldi, A. Zisserman, “Return of the devil in the details: Delving deep into convolutional nets,” arXiv preprint arXiv:1405.3531, 2014.

[39]        Y. Qi, Y. Zh. Song, H. Zhang, J. Liu, “Sketch-based image retrieval via siamese convolutional neural network,” in Proc. 2016 IEEE International Conference on Image Processing (ICIP): 2460-2464, 2016.

[40]        T. Bui, L. Ribeiro, M. Ponti, J. Collomosse, “Generalisation and sharing in triplet convnets for sketch based visual search,” arXiv preprint arXiv:1611.05301, 2016.

[41]        M. Rohani, H. Farsi, S. Mohamadzadeh, “Deep multi-task convolutional neural networks for efficient classification of face attributes,” Int. J. Eng., 36(11): 2102-2111, 2023.

[42]        P. Sangkloy, N. Burnell, C. Ham, J. Hays, “The sketchy database: learning to retrieve badly drawn bunnies,” ACM Trans. Graphics (TOG), 35(4): 1-12, 2016.

[43]        T. Bui, L. Ribiro, M. Ponti, J. Collomosse, “Sketching out the details: Sketch-based image retrieval using convolutional neural networks with multi-stage regression,” Computers & Graphics, 71: 77-87, 2018.

[44]        O. Tursun, S. Denman, S. Sridharan, E. Goan, C. Fookes, “An efficient framework for zero-shot sketch-based image retrieval,” Pattern Recognit., 126: 108528, 2022.

[45]        T. Bui, L. Ribeiro, M. Ponti, J. Collomosse, “Compact descriptors for sketch-based image retrieval using a triplet loss convolutional neural network,” Comput. Vision Image Understanding, 164: 27-37, 2017.

[46]        C. Bai, J. Chen, Q. Ma, P. Hao, Sh. Chen, “Cross-domain representation learning by domain-migration generative adversarial network for sketch based image retrieval,” J. Visual Commun. Image Represent., 71: 102835, 2020.

[47]        A. P. R. G. G. Rajput, “Sketch based image retrieval in large databases using edge features,” Int. J. Recent Technol. Eng. (IJRTE), 08: 2277-3878, 2020.

[48]        M. Eitz, J. Hays, M. Alexa, “How do humans sketch objects?,” ACM Trans. Graphics (TOG), 31(4): 1-10, 2012.

[49]        N. Dalal, B. Triggs, “Histograms of oriented gradients for human detection,” in Proc. 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), 1: 886-893, 2005.

[50]        J. M. Saavedra, “Sketch based image retrieval using a soft computation of the histogram of edge local orientations (shelo),” in Proc. 2014 IEEE International Conference on Image Processing (ICIP): 2998-3002, 2014.

[51]        D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” Int. J. Comput. Vision, 60: 91-110, 2004.

آمار
تعداد مشاهده مقاله: 506
تعداد دریافت فایل اصل مقاله: 313


سامانه مدیریت نشریات علمی. طراحی و پیاده سازی از سیناوب