A1 Journal article (refereed)
ITrans : generative image inpainting with transformers (2024)


Miao, W., Wang, L., Lu, H., Huang, K., Shi, X., & Liu, B. (2024). ITrans : generative image inpainting with transformers. Multimedia Systems, 30(1), Article 21. https://doi.org/10.1007/s00530-023-01211-w


JYU authors or editors


Publication details

All authors or editorsMiao, Wei; Wang, Lijun; Lu, Huchuan; Huang, Kaining; Shi, Xinchu; Liu, Bocong

Journal or seriesMultimedia Systems

ISSN0942-4962

eISSN1432-1882

Publication year2024

Publication date17/01/2024

Volume30

Issue number1

Article number21

PublisherSpringer

Publication countryGermany

Publication languageEnglish

DOIhttps://doi.org/10.1007/s00530-023-01211-w

Publication open accessOpenly available

Publication channel open accessPartially open access channel

Publication is parallel published (JYX)https://jyx.jyu.fi/handle/123456789/92893


Abstract

Despite significant improvements, convolutional neural network (CNN) based methods are struggling with handling long-range global image dependencies due to their limited receptive fields, leading to an unsatisfactory inpainting performance under complicated scenarios. To address this issue, we propose the Inpainting Transformer (ITrans) network, which combines the power of both self-attention and convolution operations. The ITrans network augments convolutional encoder–decoder structure with two novel designs, i.e., the global and local transformers. The global transformer aggregates high-level image context from the encoder in a global perspective, and propagates the encoded global representation to the decoder in a multi-scale manner. Meanwhile, the local transformer is intended to extract low-level image details inside the local neighborhood at a reduced computational overhead. By incorporating the above two transformers, ITrans is capable of both global relationship modeling and local details encoding, which is essential for hallucinating perceptually realistic images. Extensive experiments demonstrate that the proposed ITrans network outperforms favorably against state-of-the-art inpainting methods both quantitatively and qualitatively.


Keywordsimage processingphotographsneural networks (information technology)

Free keywordsconvolutional neural network; image inpainting; global transformer; local transformer


Contributing organizations


Ministry reportingYes

VIRTA submission year2024

Preliminary JUFO rating1


Last updated on 2024-03-07 at 00:46