Generative pre-trained transformer 4o (GPT-4o) in solving text-based multiple response questions for European Diploma in Radiology (EDiR): a comparative study with radiologists

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00064165%3A_____%2F25%3A10496386" target="_blank" >RIV/00064165:_____/25:10496386 - isvavai.cz</a>

Alternative codes found

RIV/00216208:11110/25:10496386 RIV/00216208:11130/25:10496386 RIV/00064203:_____/25:10496386

Result on the web

<a href="https://verso.is.cuni.cz/pub/verso.fpl?fname=obd_publikace_handle&handle=-3VHFRtG8d" target="_blank" >https://verso.is.cuni.cz/pub/verso.fpl?fname=obd_publikace_handle&handle=-3VHFRtG8d</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1186/s13244-025-01941-7" target="_blank" >10.1186/s13244-025-01941-7</a>

Result language

angličtina

Original language name

Generative pre-trained transformer 4o (GPT-4o) in solving text-based multiple response questions for European Diploma in Radiology (EDiR): a comparative study with radiologists

Original language description

OBJECTIVES: This study aims to assess the accuracy of generative pre-trained transformer 4o (GPT-4o) in answering multiple response questions from the European Diploma in Radiology (EDiR) examination, comparing its performance to that of human candidates. MATERIALS AND METHODS: Results from 42 EDiR candidates across Europe were compared to those from 26 fourth-year medical students who answered exclusively using the ChatGPT-4o in a prospective study (October 2024). The challenge consisted of 52 recall or understanding-based EDiR multiple-response questions, all without visual inputs. RESULTS: The GPT-4o achieved a mean score of 82.1 +- 3.0%, significantly outperforming the EDiR candidates with 49.4 +- 10.5% (p &lt; 0.0001). In particular, chatGPT-4o demonstrated higher true positive rates while maintaining lower false positive rates compared to EDiR candidates, with a higher accuracy rate in all radiology subspecialties (p &lt; 0.0001) except informatics (p = 0.20). There was near-perfect agreement between GPT-4 responses (κ = 0.872) and moderate agreement among EDiR participants (κ = 0.334). Exit surveys revealed that all participants used the copy-and-paste feature, and 73% submitted additional questions to clarify responses. CONCLUSIONS: GPT-4o significantly outperformed human candidates in low-order, text-based EDiR multiple-response questions, demonstrating higher accuracy and reliability. These results highlight GPT-4o&apos;s potential in answering text-based radiology questions. Further research is necessary to investigate its performance across different question formats and candidate populations to ensure broader applicability and reliability. CRITICAL RELEVANCE STATEMENT: GPT-4o significantly outperforms human candidates in factual radiology text-based questions in the EDiR, excelling especially in identifying correct responses, with a higher accuracy rate compared to radiologists. KEY POINTS: In EDiR text-based questions, ChatGPT-4o scored higher (82%) than EDiR participants (49%). Compared to radiologists, GPT-4o excelled in identifying correct responses. GPT-4o responses demonstrated higher agreement (κ = 0.87) compared to EDiR candidates (κ = 0.33).

Czech name

—

Czech description

—

Type

J_imp - Article in a specialist periodical, which is included in the Web of Science database

CEP classification

—

OECD FORD branch

30224 - Radiology, nuclear medicine and medical imaging

Project

—

Continuities

I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace

Publication year

2025

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Name of the periodical

Insights into Imaging

ISSN

1869-4101

e-ISSN

1869-4101

Volume of the periodical

16

Issue of the periodical within the volume

1

Country of publishing house

DE - GERMANY

Number of pages

8

Pages from-to

66

UT code for WoS article

001468337300004

EID of the result in the Scopus database

2-s2.0-105000709828