Multimodal Shannon Game with Images
The result's identifiers
Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F25%3A10511672" target="_blank" >RIV/00216208:11320/25:10511672 - isvavai.cz</a>
Result on the web
<a href="https://link.springer.com/chapter/10.1007/978-3-031-82487-6_3#citeas" target="_blank" >https://link.springer.com/chapter/10.1007/978-3-031-82487-6_3#citeas</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.1007/978-3-031-82487-6_3" target="_blank" >10.1007/978-3-031-82487-6_3</a>
Alternative languages
Result language
angličtina
Original language name
Multimodal Shannon Game with Images
Original language description
The Shannon game has long been used as a thought experiment in linguistics and NLP, asking participants to guess the next letter in a sentence based on its preceding context. We extend the game by introducing an optional extra modality in the form of image information. To investigate the impact of multimodal information in this game, we use human participants and a language model (LM, GPT-2). We show that the addition of image information improves both self-reported confidence and accuracy for both humans and LM. Certain word classes, such as nouns and determiners, benefit more from the additional modality information. The priming effect in both humans and the LM becomes more apparent as the context size (extra modality information + sentence context) increases. These findings highlight the potential of multimodal information in improving language understanding and modeling.
Czech name
—
Czech description
—
Classification
Type
D - Article in proceedings
CEP classification
—
OECD FORD branch
10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
Result continuities
Project
Result was created during the realization of more than one project. More information in the Projects tab.
Continuities
P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)
Others
Publication year
2025
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Data specific for result type
Article name in the collection
ACAIN
ISBN
978-3-031-82486-9
ISSN
—
e-ISSN
—
Number of pages
14
Pages from-to
32-45
Publisher name
Springer
Place of publication
Cham, Switzerland
Event location
Castiglione della Pescaia, Italy
Event date
Sep 22, 2024
Type of event by nationality
WRD - Celosvětová akce
UT code for WoS article
—