Semantic Fusion of Text and Images: A Novel Multimodal-RAG Framework for Document Analysis

Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216305%3A26220%2F26%3A0190080" target="_blank" >RIV/00216305:26220/26:0190080 - isvavai.cz</a>
Result on the web
—
DOI - Digital Object Identifier
—

Result language
angličtina
Original language name
Semantic Fusion of Text and Images: A Novel Multimodal-RAG Framework for Document Analysis
Original language description
This work presents the development of an advanced multimodal Retrieval-Augmented Generation (MM-RAG) framework, specifically designed to integrate and process both textual and visual data for comprehensive document analysis. Unlike traditional systems that handle only text, this framework employs cutting-edge techniques to extract and embed unstructured information from PDFs containing both text and images, ensuring a more holistic understanding of complex documents. Textual data is segmented into manageable chunks and embedded using transformer-based models, such as Gemini, which operates within a 768-dimensional embedding space to capture nuanced textual information. Simultaneously, visual data is processed through sophisticated vision-language models, which generate high-level semantic summaries that encapsulate the visual content's meaning. The MM-RAG framework then seamlessly unifies these text and image embeddings into a cohesive multimodal representation, significantly enhancing the system's ability to perform complex document retrieval and question-answering tasks. This integration enables more accurate and contextually relevant responses, making it a powerful tool for detailed document analysis.
Czech name
—
Czech description
—

Project
<a href="/en/project/VK01010153" target="_blank" >VK01010153: Development of artificial intelligence for multimodal non-destructive forensic material analysis system</a><br>
Continuities
P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)<br>S - Specificky vyzkum na vysokych skolach

Publication year
2024
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection
ICUMT 2024; 16th International Congress on Ultra Modern Telecommunications and Control Systems and Workshops
ISBN
978-3-8007-6544-7
ISSN
—
e-ISSN
2157-023X
Number of pages
6
Pages from-to
106-110
Publisher name
—
Place of publication
Meloneras
Event location
Meloneras, Gran Canaria, Spain
Event date
Nov 26, 2024
Type of event by nationality
WRD - Celosvětová akce
UT code for WoS article
—

Similar results(10)