When LLMs Can’t Help: Real-World Evaluation of LLMs in Nutrition
The result's identifiers
Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F25%3A10511616" target="_blank" >RIV/00216208:11320/25:10511616 - isvavai.cz</a>
Result on the web
<a href="https://aclanthology.org/2025.inlg-main.44" target="_blank" >https://aclanthology.org/2025.inlg-main.44</a>
DOI - Digital Object Identifier
—
Alternative languages
Result language
angličtina
Original language name
When LLMs Can’t Help: Real-World Evaluation of LLMs in Nutrition
Original language description
The increasing trust in large language models (LLMs), especially in the form of chatbots, is often undermined by the lack of their extrinsic evaluation. This holds particularly true in nutrition, where randomised controlled trials (RCTs) are the gold standard, and experts demand them for evidence-based deployment. LLMs have shown promising results in this field, but these are limited to intrinsic setups. We address this gap by running the first RCT involving LLMs for nutrition. We augment a rule-based chatbot with two LLM-based features: (1) message rephrasing for conversational variety and engagement, and (2) nutritional counselling through a fine-tuned model. In our seven-week RCT (n=81), we compare chatbot variants with and without LLM integration. We measure effects on dietary outcome, emotional well-being, and engagement. Despite our LLM-based features performing well in intrinsic evaluation, we find that they did not yield consistent benefits in real-world deployment. These results highlight cri
Czech name
—
Czech description
—
Classification
Type
D - Article in proceedings
CEP classification
—
OECD FORD branch
10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
Result continuities
Project
—
Continuities
R - Projekt Ramcoveho programu EK
Others
Publication year
2025
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Data specific for result type
Article name in the collection
Proceedings of the 18th International Natural Language Generation Conference
ISBN
979-8-89176-321-0
ISSN
—
e-ISSN
—
Number of pages
27
Pages from-to
753-779
Publisher name
Association for Computational Linguistics
Place of publication
Kerrville, TX, USA
Event location
Hanoi, Vietnam
Event date
Oct 29, 2025
Type of event by nationality
WRD - Celosvětová akce
UT code for WoS article
—