REMOVAL OF RAIN COMPONENTS FROM SINGLE IMAGES USING A RECURRENT NEURAL NETWORK
DOI:
https://doi.org/10.15588/1607-3274-2023-2-10Keywords:
image processing, rain effect, rain streaks, deep learning, convolutional neural network, attention mechanismAbstract
Context. Removing the undesirable consequences of rain effects from single images is an actual problem in many computer vision tasks, because rain streaks can significantly degrade the visual quality of images and seriously interfere with the operation of various intelligent systems, which are used for their processing and further analysis.
Objective. The goal of the work is to develop a method for detecting and removing undesirable effects of the rain from single images, which is based on the using of a convolutional neural network with a recurrent structure.
Method. The main component of the proposed method is a convolutional neural network, which has a recurrent multi-stage structure. A feature of this network architecture is the use of repeated blocks (layers), at the output of which you can get an intermediate result of «cleaning» the original image. Moreover at the output of each next layer of the network we get an image with less influence of rain components than on the previous one. Each network layer contains two independent sub-networks (branches) for parallel image processing. The main branch is designed to detect and remove the effect of rain from the image and the attention branch is used to improve and speed up the process of detecting undesirable rain components (for rain attention map formation).
Results. An approach has been developed to automatically detect and remove the rain effect from single images. The process of “cleaning” the original image is based on the use of a convolutional neural network with a recurrent structure, which was trained on the Rain100H and Rain100L datasets. The results of computer experiments, which testifies to the effectiveness and expediency of using the proposed method for solving practical tasks of pre-processing “contaminated” images are presented.
Conclusions. The advantage of the developed method for removing undesirable components of rain from images is that the recurrent multi-stage network architecture, on which it is based allows it to be potentially applied to solving tasks under conditions of limited computing resources. The proposed method can be successfully used in the development of intelligent systems for area monitoring with surveillance cameras, autonomous vehicles control, processing aerial photography results, etc. In the future, it should be considered the possibility of forming a separate sub-network to eliminate blurring in the image and train the network on datasets that contain image samples with different components of rain, which will make the method more «resistant» to different forms of the rain effect and increase the quality of image “cleaning”.
References
Wang H. Wu Y., Li M., Zhao Q., Meng D. Survey on rain removal from videos or a single image, Science China Information Sciences, 2022, Vol. 65(1), 111101. DOI:10.1007/s11432-020-3225-9
Yang W., Tan R. T., Wang S., Fang Y., Liu J. Single image deraining: from Model-based to Data-driven and beyond, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, Vol. 43, № 11, pp. 4059-4077. DOI: 10.1109/TPAMI.2020.2995190
Zheng X., Liao Y., Guo W., Fu X., Ding X. Single-imagebased rain and snow removal using multi-guided filter, 20th International Conference on Neural Information Processing (ICONIP-2013), Doegu, 03−07 November, 2013, proceedings. Berlin, Springer, 2013, pp. 258–265. (Lecture Notes in Computer Science, Vol 8228). DOI: 10.1007/978-3-64242051-1_33
Kang L.-W., Lin C.-W. and Fu Y.-H. Automatic singleimage-based rain streaks removal via image decomposition, IEEE Transactions on Image Processing, 2012, Vol. 21, № 4, pp. 1742−1755. DOI: 10.1109/TIP.2011.2179057
Luo Y., Xu Y., Ji H. Removing rain from a single image via discriminative sparse coding, 2015 IEEE International Conference on Computer Vision (ICCV). Santiago, 07−13 December, 2015, proceedings, pp. 3397−3405. DOI: 10.1109/ICCV.2015.388
Zhu L., Fu C.-W., Lischinski D., Heng P.-A. Joint bi-layer optimization for single-image rain streak removal, 2017 IEEE International Conference on Computer Vision (ICCV), Venice, 22−29 October, 2017, proceedings, pp. 2545−2553. DOI: 10.1109/ICCV.2017.276
Li Y., Tan R. T., Guo X., Lu J., Brown M. S. Rain streak removal using layer priors, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, 27−30 June, 2016, proceedings, pp. 2736−2744. DOI: 10.1109/CVPR.2016.299
Eigen D., Krishnan D., Fergus R. Restoring an image taken through a window covered with dirt or rain, 2013 IEEE International Conference on Computer Vision (ICCV). Sydney, 01−08 December, 2013, proceedings, pp. 633−640. DOI: 10.1109/ICCV.2013.84
Fu X., Huang J., Ding X., Liao Y., Paisley J. Clearing the skies: A deep network architecture for single-image rain removal, IEEE Transactions on Image Processing, 2017. Vol. 26, No. 6, pp. 2944–2956. DOI: 10.1109/TIP.2017.2691802
Fu X. Huang J., Zeng D., Huang Y., Ding X., Paisley J. Removing rain from single images via a deep detail network, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, 21−26 July, 2017, proceedings,pp. 1715−1723. DOI: 10.1109/CVPR.2017.186
He K., Zhang X., Ren S., Sun J. Deep residual learning for image recognition, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, 27−30 June, 2016, proceedings, pp. 770−778. DOI: 10.1109/CVPR.2016.90
Zhang H., Patel V. M. Density-aware single image deraining using a multi-stream dense network, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, 18−23 June, 2018: proceedings, pp. 695–704. DOI: 10.1109/CVPR.2018.00079
Wang T. Yang X., Xu K., Chen S., Zhang Q., Lau R. W. H. Spatial attentive single-image deraining with a high quality real rain dataset, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, 15−20 June, 2019: proceedings, pp. 12262−12271. DOI: 10.1109/CVPR.2019.01255
Yang W. Tan R. T., Feng J., Liu J., Guo Z., Yan S. Deep joint rain detection and removal from a single image, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, 21−26 July, 2017, proceedings, pp. 1685−1694. DOI: 10.1109/CVPR.2017.183
Yu F., Koltun V. Multi-scale context aggregation by dilated convolutions, 4th International Conference on Learning Representations (ICLR-2016). San Juan, 02−04 May, 2016: proceedings, pp. 1−13. DOI: 10.48550/arXiv.1511.07122
Yang W., Tan R. T., Feng J., Guo Z., Yan S. and Liu J. Joint rain detection and removal from a single image with contextualized deep networks, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, Vol. 42, № 6, pp. 1377−1393. DOI: 10.1109/TPAMI.2019.2895793
Li X., Wu J., Lin Z., Liu H., Zha H. Recurrent squeeze-andexcitation context aggregation net for single image deraining, 15th European Conference Computer Vision (ECCV2018), Munich, 08−14 September, 2018, proceedings. Cham: Springer, 2018, pp. 262–277. (Lecture Notes in Computer Science, Vol. 11211). DOI: 10.1007/978-3-03001234-2_16
Ren D., Zuo W., Hu Q., Zhu P., Meng D. Progressive image deraining networks: A better and simpler baseline, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, 15−20 June, 2019, proceedings, pp. 3932−3941. DOI: 10.1109/CVPR.2019.00406
Fu X., Liang B., Huang Y., Ding X., Paisley J. Lightweight pyramid networks for image deraining, IEEE Transactions on Neural Networks and Learning Systems, 2020, Vol. 31, № 6, pp. 1794−1807. DOI: 10.1109/TNNLS.2019.2926481
Hu X., Fu C.-W., Zhu L., Heng P.-A. Depth-attentional features for single-image rain removal, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, 15−20 June, 2019, proceedings, pp. 8014−8023. DOI: 10.1109/CVPR.2019.00821
Zhang H., Sindagi V., Patel V. M. Image de-raining using a conditional generative adversarial network, IEEE Transactions on Circuits and Systems for Video Technology, 2020, Vol. 30, № 11, pp. 3943−3956. DOI: 10.1109/TCSVT.2019.2920407
Qian R., Tan R. T., Yang W., Su J., Liu J. Attentive generative adversarial network for raindrop removal from a single image, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Salt Lake City, 18−23 June, 2018, proceedings, pp. 2482−2491. DOI: 10.1109/CVPR.2018.00263
Jin X., Chen Z., Lin J., Chen Z., Zhou W. Unsupervised single image deraining with self-supervised constraints, 2019 IEEE International Conference on Image Processing (ICIP). Taipei, 22−25 September, 2019, proceedings. pp. 2761−2765. DOI: 10.1109/ICIP.2019.8803238
Hochreiter S., Schmidhuber J. Long short-term memory, Neural Computation, 1997, Vol. 9, № 8, pp. 1735–1780. DOI: 10.1162/neco.1997.9.8.1735
Wang Z., Bovik A. C., Sheikh H. R., Simoncelli E. P. Image quality assessment: from error visibility to structural similarity, IEEE Transactions on Image Processing, 2014, Vol. 13, № 4, pp. 600−612. DOI: 10.1109/TIP.2003.819861
Kingma D. P., Ba J. Adam: A method for stochastic optimization, 3rd International Conference on Learning Representation (ICLR-2015), San Diego, 07−09 May, 2015, proceedings, pp. 1−15. DOI: 10.48550/arXiv.1412.6980
Huynh-Thu Q., Ghanbari M. Scope of validity of PSNR in image/video quality assessment, Electronics letters, 2008, Vol. 44, № 13, pp. 800–801. DOI: 10.1049/el:20080522
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 K. E. Petrov, V. V. Kyrychenko
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Creative Commons Licensing Notifications in the Copyright Notices
The journal allows the authors to hold the copyright without restrictions and to retain publishing rights without restrictions.
The journal allows readers to read, download, copy, distribute, print, search, or link to the full texts of its articles.
The journal allows to reuse and remixing of its content, in accordance with a Creative Commons license СС BY -SA.
Authors who publish with this journal agree to the following terms:
-
Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License CC BY-SA that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
-
Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
-
Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work.
