Using Large Language Models to Support Thematic Analysis in Empirical Legal Studies

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68378122%3A_____%2F23%3A00579735" target="_blank" >RIV/68378122:_____/23:00579735 - isvavai.cz</a>

Výsledek na webu

<a href="http://dx.doi.org/10.3233/FAIA230965" target="_blank" >http://dx.doi.org/10.3233/FAIA230965</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.3233/FAIA230965" target="_blank" >10.3233/FAIA230965</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Using Large Language Models to Support Thematic Analysis in Empirical Legal Studies

Popis výsledku v původním jazyce

Thematic analysis and other variants of inductive coding are widely used qualitative analytic methods within empirical legal studies (ELS). We propose a novel framework facilitating effective collaboration of a legal expert with a large language model (LLM) for generating initial codes (phase 2 of thematic analysis), searching for themes (phase 3), and classifying the data in terms of the themes (to kick-start phase 4). We employed the framework for an analysis of a dataset (n = 785) of facts descriptions from criminal court opinions regarding thefts. The goal of the analysis was to discover classes of typical thefts. Our results show that the LLM, namely OpenAI’s GPT-4, generated reasonable initial codes, and it was capable of improving the quality of the codes based on expert feedback. They also suggest that the model performed well in zero-shot classification of facts descriptions in terms of the themes. Finally, the themes autonomously discovered by the LLM appear to map fairly well to the themes arrived at by legal experts. These findings can be leveraged by legal researchers to guide their decisions in integrating LLMs into their thematic analyses, as well as other inductive coding projects.

Název v anglickém jazyce

Using Large Language Models to Support Thematic Analysis in Empirical Legal Studies

Popis výsledku anglicky

Thematic analysis and other variants of inductive coding are widely used qualitative analytic methods within empirical legal studies (ELS). We propose a novel framework facilitating effective collaboration of a legal expert with a large language model (LLM) for generating initial codes (phase 2 of thematic analysis), searching for themes (phase 3), and classifying the data in terms of the themes (to kick-start phase 4). We employed the framework for an analysis of a dataset (n = 785) of facts descriptions from criminal court opinions regarding thefts. The goal of the analysis was to discover classes of typical thefts. Our results show that the LLM, namely OpenAI’s GPT-4, generated reasonable initial codes, and it was capable of improving the quality of the codes based on expert feedback. They also suggest that the model performed well in zero-shot classification of facts descriptions in terms of the themes. Finally, the themes autonomously discovered by the LLM appear to map fairly well to the themes arrived at by legal experts. These findings can be leveraged by legal researchers to guide their decisions in integrating LLMs into their thematic analyses, as well as other inductive coding projects.

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

50501 - Law

Projekt

<a href="/cs/project/GA19-15077S" target="_blank" >GA19-15077S: Rozdíly při ukládání trestů v postkomunistických kontinentálních právních systémech</a><br>

Návaznosti

I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace

Rok uplatnění

2023

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

Legal Knowledge and Information Systems

ISBN

978-1-64368-364-5

ISSN

—

e-ISSN

—

Počet stran výsledku

10

Strana od-do

197-206

Název nakladatele

IOS Press

Místo vydání

Amsterdam

Místo konání akce

Maastricht

Datum konání akce

18. 12. 2023

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

—