A1 Alkuperäisartikkeli tieteellisessä aikakauslehdessä
ITrans : generative image inpainting with transformers (2024)

Miao, W., Wang, L., Lu, H., Huang, K., Shi, X., & Liu, B. (2024). ITrans : generative image inpainting with transformers. Multimedia Systems, 30(1), Article 21. https://doi.org/10.1007/s00530-023-01211-w

JYU-tekijät tai -toimittajat

Julkaisun tiedot

Julkaisun kaikki tekijät tai toimittajat: Miao, Wei; Wang, Lijun; Lu, Huchuan; Huang, Kaining; Shi, Xinchu; Liu, Bocong

Lehti tai sarja: Multimedia Systems

ISSN: 0942-4962

eISSN: 1432-1882

Julkaisuvuosi: 2024

Ilmestymispäivä: 17.01.2024

Volyymi: 30

Lehden numero: 1

Artikkelinumero: 21

Kustantaja: Springer

Julkaisumaa: Saksa

Julkaisun kieli: englanti

DOI: https://doi.org/10.1007/s00530-023-01211-w

Julkaisun avoin saatavuus: Avoimesti saatavilla

Julkaisukanavan avoin saatavuus: Osittain avoin julkaisukanava

Julkaisu on rinnakkaistallennettu (JYX): https://jyx.jyu.fi/handle/123456789/92893

Tiivistelmä

Despite significant improvements, convolutional neural network (CNN) based methods are struggling with handling long-range global image dependencies due to their limited receptive fields, leading to an unsatisfactory inpainting performance under complicated scenarios. To address this issue, we propose the Inpainting Transformer (ITrans) network, which combines the power of both self-attention and convolution operations. The ITrans network augments convolutional encoder–decoder structure with two novel designs, i.e., the global and local transformers. The global transformer aggregates high-level image context from the encoder in a global perspective, and propagates the encoded global representation to the decoder in a multi-scale manner. Meanwhile, the local transformer is intended to extract low-level image details inside the local neighborhood at a reduced computational overhead. By incorporating the above two transformers, ITrans is capable of both global relationship modeling and local details encoding, which is essential for hallucinating perceptually realistic images. Extensive experiments demonstrate that the proposed ITrans network outperforms favorably against state-of-the-art inpainting methods both quantitatively and qualitatively.

YSO-asiasanat: kuvankäsittely; valokuvat; neuroverkot

Vapaat asiasanat: convolutional neural network; image inpainting; global transformer; local transformer

Liittyvät organisaatiot

OKM-raportointi: Kyllä

VIRTA-lähetysvuosi: 2024

Alustava JUFO-taso: 1