Comparing traditional natural language processing and large language models for mental health status classification: a multi-model evaluation

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00159816%3A_____%2F25%3A00082405" target="_blank" >RIV/00159816:_____/25:00082405 - isvavai.cz</a>

Výsledek na webu

<a href="https://www.nature.com/articles/s41598-025-08031-0" target="_blank" >https://www.nature.com/articles/s41598-025-08031-0</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1038/s41598-025-08031-0" target="_blank" >10.1038/s41598-025-08031-0</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Comparing traditional natural language processing and large language models for mental health status classification: a multi-model evaluation

Popis výsledku v původním jazyce

The substantial increase in mental health disorders globally necessitates scalable, accurate tools for detecting and classifying these conditions in digital environments. This study addresses the critical challenge of automated mental health classification by comparing three distinct computational approaches: (1) Traditional Natural Language Processing (NLP) with advanced feature engineering, (2) Prompt-engineered large language models (LLMs), and (3) Fine-tuned LLMs. The dataset consisted of over 51,000 publicly available text statements from social media platforms, tagged with seven mental health conditions: Normal, Depression, Suicidal, Anxiety, Stress, Bipolar Disorder, and Personality Disorder. The dataset was stratified into training, validation, and test sets for model evaluation. The primary outcome was classification accuracy across these seven mental health conditions. Additional metrics like precision, recall, and F1-score were analyzed. We compared the results of the three computational approaches and overfitting was monitored through validation loss across epochs for the fine-tuned LLM. The NLP model with advanced feature engineering achieved an overall accuracy of 95%, surpassing both the prompt-engineered LLM (65%) and the fine-tuned LLM (91%). This model performed exceptionally well in terms of accuracy and precision. While fine-tuning for three epochs yielded optimal results, further training led to overfitting and decreased performance. This study demonstrates the significant benefits of applying advanced text preprocessing and feature engineering techniques to traditional NLP models, alongside fine-tuning LLMs, such as GPT-4o-mini, for mental health classification tasks. The results clearly indicate that off-the-shelf LLM chatbots using prompt engineering are inadequate for mental health classification, performing 30% points below specialized NLP approaches. Despite the popularity of general-purpose LLMs, specialized approaches remain superior for critical healthcare applications like mental health classification.

Název v anglickém jazyce

Comparing traditional natural language processing and large language models for mental health status classification: a multi-model evaluation

Popis výsledku anglicky

The substantial increase in mental health disorders globally necessitates scalable, accurate tools for detecting and classifying these conditions in digital environments. This study addresses the critical challenge of automated mental health classification by comparing three distinct computational approaches: (1) Traditional Natural Language Processing (NLP) with advanced feature engineering, (2) Prompt-engineered large language models (LLMs), and (3) Fine-tuned LLMs. The dataset consisted of over 51,000 publicly available text statements from social media platforms, tagged with seven mental health conditions: Normal, Depression, Suicidal, Anxiety, Stress, Bipolar Disorder, and Personality Disorder. The dataset was stratified into training, validation, and test sets for model evaluation. The primary outcome was classification accuracy across these seven mental health conditions. Additional metrics like precision, recall, and F1-score were analyzed. We compared the results of the three computational approaches and overfitting was monitored through validation loss across epochs for the fine-tuned LLM. The NLP model with advanced feature engineering achieved an overall accuracy of 95%, surpassing both the prompt-engineered LLM (65%) and the fine-tuned LLM (91%). This model performed exceptionally well in terms of accuracy and precision. While fine-tuning for three epochs yielded optimal results, further training led to overfitting and decreased performance. This study demonstrates the significant benefits of applying advanced text preprocessing and feature engineering techniques to traditional NLP models, alongside fine-tuning LLMs, such as GPT-4o-mini, for mental health classification tasks. The results clearly indicate that off-the-shelf LLM chatbots using prompt engineering are inadequate for mental health classification, performing 30% points below specialized NLP approaches. Despite the popularity of general-purpose LLMs, specialized approaches remain superior for critical healthcare applications like mental health classification.

Druh

J_imp - Článek v periodiku v databázi Web of Science

CEP obor

—

OECD FORD obor

10200 - Computer and information sciences

Projekt

—

Návaznosti

I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace

Rok uplatnění

2025

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název periodika

Scientific Reports

ISSN

2045-2322

e-ISSN

—

Svazek periodika

15

Číslo periodika v rámci svazku

1

Stát vydavatele periodika

DE - Spolková republika Německo

Počet stran výsledku

13

Strana od-do

24102

Kód UT WoS článku

001523298400024

EID výsledku v databázi Scopus

—