AIC CTU system at AVeriTeC: Re-framing automated fact-checking as a simple RAG task

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21230%2F24%3A00381246" target="_blank" >RIV/68407700:21230/24:00381246 - isvavai.cz</a>

Result on the web

<a href="https://doi.org/10.18653/v1/2024.fever-1.16" target="_blank" >https://doi.org/10.18653/v1/2024.fever-1.16</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.18653/v1/2024.fever-1.16" target="_blank" >10.18653/v1/2024.fever-1.16</a>

Result language

angličtina

Original language name

AIC CTU system at AVeriTeC: Re-framing automated fact-checking as a simple RAG task

Original language description

This paper describes our 3rd place submission in the AVeriTeC shared task in which we attempted to address the challenge of fact-checking with evidence retrieved in the wild using a simple scheme of Retrieval-Augmented Generation (RAG) designed for the task, leveraging the predictive power of Large Language Models. We release our codebase and explain its two modules – the Retriever and the Evidence & Label generator – in detail, justifying their features such as MMR-reranking and Likert-scale confidence estimation. We evaluate our solution on AVeriTeC dev and test set and interpret the results, picking the GPT-4o as the most appropriate model for our pipeline at the time of our publication, with Llama 3.1 70B being a promising open-source alternative. We perform an empirical error analysis to see that faults in our predictions often coincide with noise in the data or ambiguous fact-checks, provoking further research and data augmentation.

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

<a href="/en/project/FW10010200" target="_blank" >FW10010200: Domain service of large AI language models using GPT upskilling.</a><br>

Continuities

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Publication year

2024

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

Seventh Workshop on Fact Extraction and VERification (FEVER 2024)

ISBN

979-8-3313-0845-2

ISSN

—

e-ISSN

—

Number of pages

14

Pages from-to

137-150

Publisher name

Association for Computational Linguistics (ACL)

Place of publication

Stroudsburg

Event location

Miami

Event date

Nov 16, 2024

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

—