A1 Journal article (refereed)
ITrans : generative image inpainting with transformers (2024)
Miao, W., Wang, L., Lu, H., Huang, K., Shi, X., & Liu, B. (2024). ITrans : generative image inpainting with transformers. Multimedia Systems, 30(1), Article 21. https://doi.org/10.1007/s00530-023-01211-w
JYU authors or editors
Publication details
All authors or editors: Miao, Wei; Wang, Lijun; Lu, Huchuan; Huang, Kaining; Shi, Xinchu; Liu, Bocong
Journal or series: Multimedia Systems
ISSN: 0942-4962
eISSN: 1432-1882
Publication year: 2024
Publication date: 17/01/2024
Volume: 30
Issue number: 1
Article number: 21
Publisher: Springer
Publication country: Germany
Publication language: English
DOI: https://doi.org/10.1007/s00530-023-01211-w
Publication open access: Openly available
Publication channel open access: Partially open access channel
Publication is parallel published (JYX): https://jyx.jyu.fi/handle/123456789/92893
Abstract
Despite significant improvements, convolutional neural network (CNN) based methods are struggling with handling long-range global image dependencies due to their limited receptive fields, leading to an unsatisfactory inpainting performance under complicated scenarios. To address this issue, we propose the Inpainting Transformer (ITrans) network, which combines the power of both self-attention and convolution operations. The ITrans network augments convolutional encoder–decoder structure with two novel designs, i.e., the global and local transformers. The global transformer aggregates high-level image context from the encoder in a global perspective, and propagates the encoded global representation to the decoder in a multi-scale manner. Meanwhile, the local transformer is intended to extract low-level image details inside the local neighborhood at a reduced computational overhead. By incorporating the above two transformers, ITrans is capable of both global relationship modeling and local details encoding, which is essential for hallucinating perceptually realistic images. Extensive experiments demonstrate that the proposed ITrans network outperforms favorably against state-of-the-art inpainting methods both quantitatively and qualitatively.
Keywords: image processing; photographs; neural networks (information technology)
Free keywords: convolutional neural network; image inpainting; global transformer; local transformer
Contributing organizations
Ministry reporting: Yes
VIRTA submission year: 2024
Preliminary JUFO rating: 1