Semantic Fusion of Text and Images: A Novel Multimodal-RAG Framework for Document Analysis

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216305%3A26220%2F26%3A0190080" target="_blank" >RIV/00216305:26220/26:0190080 - isvavai.cz</a>

Výsledek na webu

—

DOI - Digital Object Identifier

—

Jazyk výsledku

angličtina

Název v původním jazyce

Semantic Fusion of Text and Images: A Novel Multimodal-RAG Framework for Document Analysis

Popis výsledku v původním jazyce

This work presents the development of an advanced multimodal Retrieval-Augmented Generation (MM-RAG) framework, specifically designed to integrate and process both textual and visual data for comprehensive document analysis. Unlike traditional systems that handle only text, this framework employs cutting-edge techniques to extract and embed unstructured information from PDFs containing both text and images, ensuring a more holistic understanding of complex documents. Textual data is segmented into manageable chunks and embedded using transformer-based models, such as Gemini, which operates within a 768-dimensional embedding space to capture nuanced textual information. Simultaneously, visual data is processed through sophisticated vision-language models, which generate high-level semantic summaries that encapsulate the visual content's meaning. The MM-RAG framework then seamlessly unifies these text and image embeddings into a cohesive multimodal representation, significantly enhancing the system's ability to perform complex document retrieval and question-answering tasks. This integration enables more accurate and contextually relevant responses, making it a powerful tool for detailed document analysis.

Název v anglickém jazyce

Semantic Fusion of Text and Images: A Novel Multimodal-RAG Framework for Document Analysis

Popis výsledku anglicky

This work presents the development of an advanced multimodal Retrieval-Augmented Generation (MM-RAG) framework, specifically designed to integrate and process both textual and visual data for comprehensive document analysis. Unlike traditional systems that handle only text, this framework employs cutting-edge techniques to extract and embed unstructured information from PDFs containing both text and images, ensuring a more holistic understanding of complex documents. Textual data is segmented into manageable chunks and embedded using transformer-based models, such as Gemini, which operates within a 768-dimensional embedding space to capture nuanced textual information. Simultaneously, visual data is processed through sophisticated vision-language models, which generate high-level semantic summaries that encapsulate the visual content's meaning. The MM-RAG framework then seamlessly unifies these text and image embeddings into a cohesive multimodal representation, significantly enhancing the system's ability to perform complex document retrieval and question-answering tasks. This integration enables more accurate and contextually relevant responses, making it a powerful tool for detailed document analysis.

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

20203 - Telecommunications

Projekt

<a href="/cs/project/VK01010153" target="_blank" >VK01010153: Vývoj umělé inteligence pro systém multimodální nedestruktivní forenzní analýzy materiálů</a><br>

Návaznosti

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)<br>S - Specificky vyzkum na vysokych skolach

Rok uplatnění

2024

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

ICUMT 2024; 16th International Congress on Ultra Modern Telecommunications and Control Systems and Workshops

ISBN

978-3-8007-6544-7

ISSN

—

e-ISSN

2157-023X

Počet stran výsledku

6

Strana od-do

106-110

Název nakladatele

—

Místo vydání

Meloneras

Místo konání akce

Meloneras, Gran Canaria, Spain

Datum konání akce

26. 11. 2024

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

—