From human artefact to machine output: automating the “art” of psychological measurement

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F61988987%3A17250%2F25%3AA2603BMW" target="_blank" >RIV/61988987:17250/25:A2603BMW - isvavai.cz</a>

Result on the web

<a href="https://www.tandfonline.com/doi/full/10.1080/29974100.2025.2561692" target="_blank" >https://www.tandfonline.com/doi/full/10.1080/29974100.2025.2561692</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1080/29974100.2025.2561692" target="_blank" >10.1080/29974100.2025.2561692</a>

Result language

angličtina

Original language name

From human artefact to machine output: automating the “art” of psychological measurement

Original language description

Creating psychological assessment tools is crucial for research but traditionally expensive and time-consuming. While Large Language Models (LLMs) show promise for automating this process, existing approaches lack systematic, user-friendly methodologies grounded in psychometric principles. This study presents an enhanced Psychometric Item Generator (PIG) method using conversational LLMs with Problem-Solving Plans (PSP) and Chain-of-Thought (CoT) prompting. Three demonstrations validated the approach: Gemini 1.5 Flash generated 20 “propensity to trust AI” items with strong semantic coherence; Claude 3 Opus created 20 “AI anxiety” items that outperformed human-generated versions linguistically; and a 6-item “AI adoption in online learning” scale was developed and validated with 1,233 participants using multiverse analysis. Results demonstrate that LLMs can produce psychometrically sound items. The AI-generated anxiety scale showed superior linguistic properties compared to human alternatives, while the learning scale exhibited good internal consistency, item homogeneity, and clear two-factor structure across multiple analytical teams. The study establishes a PSP-CoT framework that improves LLM output quality, offering researchers a cost-effective, accessible scale development methodology. However, findings emphasize that human oversight, rigorous validation, and ethical considerations remain essential components of the process.

Czech name

—

Czech description

—

Type

J_ost - Miscellaneous article in a specialist periodical

CEP classification

—

OECD FORD branch

50101 - Psychology (including human - machine relations)

Project

<a href="/en/project/EH23_025%2F0008724" target="_blank" >EH23_025/0008724: Biography of Fake News with a Touch of AI: Dangerous Phenomenon through the Prism of Modern Human Sciences</a><br>

Continuities

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Publication year

2025

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Name of the periodical

Journal of Psychology and AI

ISSN

2997-4100

e-ISSN

2997-4100

Volume of the periodical

—

Issue of the periodical within the volume

1

Country of publishing house

GB - UNITED KINGDOM

Number of pages

18

Pages from-to

—

UT code for WoS article

—

EID of the result in the Scopus database

—