Automation of cleaning and ensembles for outliers detection in questionnaire data

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F61989592%3A15510%2F22%3A73613216" target="_blank" >RIV/61989592:15510/22:73613216 - isvavai.cz</a>

Nalezeny alternativní kódy

RIV/61989100:27240/22:10250037

Výsledek na webu

<a href="https://www.sciencedirect.com/science/article/pii/S0957417422010727" target="_blank" >https://www.sciencedirect.com/science/article/pii/S0957417422010727</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1016/j.eswa.2022.117809" target="_blank" >10.1016/j.eswa.2022.117809</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Automation of cleaning and ensembles for outliers detection in questionnaire data

Popis výsledku v původním jazyce

This article is focused on the automatic detection of the corrupted or inappropriate responses in questionnaire data using unsupervised outliers detection. The questionnaire surveys are often used in psychology research to collect self-report data and their preprocessing takes a lot of manual effort. Unlike with numerical data where the distance-based outliers prevail, the records in questionnaires have to be assessed from various perspectives that do not relate so much. We identify the most frequent types of errors in questionnaires. For each of them, we suggest different outliers detection methods ranking the records with the usage of normalized scores. Considering the similarity between pairs of outlier scores (some are highly uncorrelated), we propose an ensemble based on the union of outliers detected by different methods. Our outlier detection framework consists of some well-known algorithms but we also propose novel approaches addressing the typical issues of questionnaires. The selected methods are based on distance, entropy, and probability. The experimental section describes the process of assembling the methods and selecting their parameters for the final model detecting significant outliers in the real-world HBSC dataset.

Název v anglickém jazyce

Automation of cleaning and ensembles for outliers detection in questionnaire data

Popis výsledku anglicky

This article is focused on the automatic detection of the corrupted or inappropriate responses in questionnaire data using unsupervised outliers detection. The questionnaire surveys are often used in psychology research to collect self-report data and their preprocessing takes a lot of manual effort. Unlike with numerical data where the distance-based outliers prevail, the records in questionnaires have to be assessed from various perspectives that do not relate so much. We identify the most frequent types of errors in questionnaires. For each of them, we suggest different outliers detection methods ranking the records with the usage of normalized scores. Considering the similarity between pairs of outlier scores (some are highly uncorrelated), we propose an ensemble based on the union of outliers detected by different methods. Our outlier detection framework consists of some well-known algorithms but we also propose novel approaches addressing the typical issues of questionnaires. The selected methods are based on distance, entropy, and probability. The experimental section describes the process of assembling the methods and selecting their parameters for the final model detecting significant outliers in the real-world HBSC dataset.

Druh

J_imp - Článek v periodiku v databázi Web of Science

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

Výsledek vznikl pri realizaci vícero projektů. Více informací v záložce Projekty.

Návaznosti

P - Projekt vyzkumu a vyvoje financovany z verejnych zdroju (s odkazem do CEP)

Rok uplatnění

2022

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název periodika

EXPERT SYSTEMS WITH APPLICATIONS

ISSN

0957-4174

e-ISSN

1873-6793

Svazek periodika

15

Číslo periodika v rámci svazku

117809

Stát vydavatele periodika

US - Spojené státy americké

Počet stran výsledku

14

Strana od-do

1-14

Kód UT WoS článku

000841013700010

EID výsledku v databázi Scopus

2-s2.0-85133853052