Heimatkunde: Dataset for Multi-Modal Historical Document Analysis

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F49777513%3A23520%2F24%3A43971565" target="_blank" >RIV/49777513:23520/24:43971565 - isvavai.cz</a>

Výsledek na webu

<a href="https://www.scitepress.org/Link.aspx?doi=10.5220/0012428500003636" target="_blank" >https://www.scitepress.org/Link.aspx?doi=10.5220/0012428500003636</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.5220/0012428500003636" target="_blank" >10.5220/0012428500003636</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Heimatkunde: Dataset for Multi-Modal Historical Document Analysis

Popis výsledku v původním jazyce

This paper introduces a novel Heimatkunde dataset comprising printed documents in German, specifically designedfor evaluating layout analysis methods with a focus on multi-modality. The dataset is openly accessiblefor research purposes. The study further presents baseline results for instance segmentation and multi-modalelement classification. Three advanced models, Mask R-CNN, YOLOv8, and LayoutLMv3, are employed forinstance segmentation, while a fusion-based model integrating BERT and various vision Transformers are proposedfor multi-modal classification. Experimental findings reveal that optimal bounding box segmentation isachieved with YOLOv8 using an input image size of 1280 pixels, and the best segmentation mask is producedby LayoutLMv3 with PubLayNet weights. Moreover, the research demonstrates superior multi-modal classificationresults using BERT for textual and Vision Transformer for image modalities. The study concludesby suggesting the integration of the proposed models into the historical Porta fontium portal to enhance theinformation retrieval from historical data.

Název v anglickém jazyce

Heimatkunde: Dataset for Multi-Modal Historical Document Analysis

Popis výsledku anglicky

This paper introduces a novel Heimatkunde dataset comprising printed documents in German, specifically designedfor evaluating layout analysis methods with a focus on multi-modality. The dataset is openly accessiblefor research purposes. The study further presents baseline results for instance segmentation and multi-modalelement classification. Three advanced models, Mask R-CNN, YOLOv8, and LayoutLMv3, are employed forinstance segmentation, while a fusion-based model integrating BERT and various vision Transformers are proposedfor multi-modal classification. Experimental findings reveal that optimal bounding box segmentation isachieved with YOLOv8 using an input image size of 1280 pixels, and the best segmentation mask is producedby LayoutLMv3 with PubLayNet weights. Moreover, the research demonstrates superior multi-modal classificationresults using BERT for textual and Vision Transformer for image modalities. The study concludesby suggesting the integration of the proposed models into the historical Porta fontium portal to enhance theinformation retrieval from historical data.

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

S - Specificky vyzkum na vysokych skolach

Rok uplatnění

2024

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

Proceedings of the 16th International Conference on Agents and Artificial Intelligence

ISBN

978-989-758-680-4

ISSN

2184-3589

e-ISSN

2184-433X

Počet stran výsledku

7

Strana od-do

995-1001

Název nakladatele

ScitePress

Místo vydání

Setúbal

Místo konání akce

Řím, Itálie

Datum konání akce

24. 2. 2024

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

—