From human artefact to machine output: automating the “art” of psychological measurement

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F61988987%3A17250%2F25%3AA2603BMW" target="_blank" >RIV/61988987:17250/25:A2603BMW - isvavai.cz</a>

Výsledek na webu

<a href="https://www.tandfonline.com/doi/full/10.1080/29974100.2025.2561692" target="_blank" >https://www.tandfonline.com/doi/full/10.1080/29974100.2025.2561692</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1080/29974100.2025.2561692" target="_blank" >10.1080/29974100.2025.2561692</a>

Jazyk výsledku

angličtina

Název v původním jazyce

From human artefact to machine output: automating the “art” of psychological measurement

Popis výsledku v původním jazyce

Creating psychological assessment tools is crucial for research but traditionally expensive and time-consuming. While Large Language Models (LLMs) show promise for automating this process, existing approaches lack systematic, user-friendly methodologies grounded in psychometric principles. This study presents an enhanced Psychometric Item Generator (PIG) method using conversational LLMs with Problem-Solving Plans (PSP) and Chain-of-Thought (CoT) prompting. Three demonstrations validated the approach: Gemini 1.5 Flash generated 20 “propensity to trust AI” items with strong semantic coherence; Claude 3 Opus created 20 “AI anxiety” items that outperformed human-generated versions linguistically; and a 6-item “AI adoption in online learning” scale was developed and validated with 1,233 participants using multiverse analysis. Results demonstrate that LLMs can produce psychometrically sound items. The AI-generated anxiety scale showed superior linguistic properties compared to human alternatives, while the learning scale exhibited good internal consistency, item homogeneity, and clear two-factor structure across multiple analytical teams. The study establishes a PSP-CoT framework that improves LLM output quality, offering researchers a cost-effective, accessible scale development methodology. However, findings emphasize that human oversight, rigorous validation, and ethical considerations remain essential components of the process.

Název v anglickém jazyce

From human artefact to machine output: automating the “art” of psychological measurement

Popis výsledku anglicky

Creating psychological assessment tools is crucial for research but traditionally expensive and time-consuming. While Large Language Models (LLMs) show promise for automating this process, existing approaches lack systematic, user-friendly methodologies grounded in psychometric principles. This study presents an enhanced Psychometric Item Generator (PIG) method using conversational LLMs with Problem-Solving Plans (PSP) and Chain-of-Thought (CoT) prompting. Three demonstrations validated the approach: Gemini 1.5 Flash generated 20 “propensity to trust AI” items with strong semantic coherence; Claude 3 Opus created 20 “AI anxiety” items that outperformed human-generated versions linguistically; and a 6-item “AI adoption in online learning” scale was developed and validated with 1,233 participants using multiverse analysis. Results demonstrate that LLMs can produce psychometrically sound items. The AI-generated anxiety scale showed superior linguistic properties compared to human alternatives, while the learning scale exhibited good internal consistency, item homogeneity, and clear two-factor structure across multiple analytical teams. The study establishes a PSP-CoT framework that improves LLM output quality, offering researchers a cost-effective, accessible scale development methodology. However, findings emphasize that human oversight, rigorous validation, and ethical considerations remain essential components of the process.

Druh

J_ost - Ostatní články v recenzovaných periodicích

CEP obor

—

OECD FORD obor

50101 - Psychology (including human - machine relations)

Projekt

<a href="/cs/project/EH23_025%2F0008724" target="_blank" >EH23_025/0008724: Biografie dezinformace s přívlastkem AI: Rizikový fenomén prizmatem moderních věd o člověku</a><br>

Návaznosti

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Rok uplatnění

2025

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název periodika

Journal of Psychology and AI

ISSN

2997-4100

e-ISSN

2997-4100

Svazek periodika

—

Číslo periodika v rámci svazku

1

Stát vydavatele periodika

GB - Spojené království Velké Británie a Severního Irska

Počet stran výsledku

18

Strana od-do

—

Kód UT WoS článku

—

EID výsledku v databázi Scopus

—