Masked Self-Supervised Pre-Training for Text Recognition Transformers on Large-Scale Datasets

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216305%3A26230%2F26%3A0197661" target="_blank" >RIV/00216305:26230/26:0197661 - isvavai.cz</a>

Result on the web

<a href="https://link.springer.com/chapter/10.1007/978-3-032-09368-4_4" target="_blank" >https://link.springer.com/chapter/10.1007/978-3-032-09368-4_4</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1007/978-3-032-09368-4_4" target="_blank" >10.1007/978-3-032-09368-4_4</a>

Result language

angličtina

Original language name

Masked Self-Supervised Pre-Training for Text Recognition Transformers on Large-Scale Datasets

Original language description

Self-supervised learning has emerged as a powerful approach for leveraging large-scale unlabeled data to improve model performance in various domains. In this paper, we explore masked self-supervised pre-training for text recognition transformers. Specifically, we propose two modifications to the pre-training phase: progressively increasing the masking probability, and modifying the loss function to incorporate both masked and non-masked patches. We conduct extensive experiments using a dataset of 50M unlabeled text lines for pre-training and four differently sized annotated datasets for fine-tuning. Furthermore, we compare our pre-trained models against those trained with transfer learning, demonstrating the effectiveness of the self-supervised pre-training. In particular, pre-training consistently improves the character error rate of models, in some cases up to 30 % relatively. It is also on par with transfer learning but without relying on extra annotated text lines.

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

<a href="/en/project/DH23P03OVV060" target="_blank" >DH23P03OVV060: semANT - Semantic Document Exploration</a><br>

Continuities

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Publication year

2025

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

Document Analysis and Recognition – ICDAR 2025 Workshops

ISBN

978-3-032-09367-7

ISSN

—

e-ISSN

—

Number of pages

18

Pages from-to

53-70

Publisher name

Springer Nature Switzerland

Place of publication

Cham

Event location

Wuhan, Čína

Event date

Sep 16, 2025

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

—