Interpreting Workflow Architectures by LLMs

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F25%3A10499966" target="_blank" >RIV/00216208:11320/25:10499966 - isvavai.cz</a>

Výsledek na webu

<a href="https://doi.org/10.5220/0013358000003928" target="_blank" >https://doi.org/10.5220/0013358000003928</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.5220/0013358000003928" target="_blank" >10.5220/0013358000003928</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Interpreting Workflow Architectures by LLMs

Popis výsledku v původním jazyce

In this paper, we focus on how reliably can a Large Lanuage Model (LLM) interpret a software architecture, namely a workflow architecture (WA). Even though our initial experiments show that an LLM can answer specific questions about a WA, it is unclear how correct its answers are. To this end, we propose a methodology to assess whether an LLM can correctly interpret a WA specification. Based on the conjecture that the LLM needs to correctly answer low-abstraction level questions to answer questions at a higher abstraction level properly, we define a set of test patterns, each of them providing a template for low-abstraction level questions, together with a metric for evaluating the correctness of LLM&apos;s answers. We posit that having this metric will allow us not only to establish which LLM model works the best with WAs, but also to determine what their concrete syntax and concepts are suitable to strengthen the correctness of LLM&apos;s interpretability of WA specifications. We demonstrate the methodology on the workflow specification language developed for a currently running Horizon Europe project.

Název v anglickém jazyce

Interpreting Workflow Architectures by LLMs

Popis výsledku anglicky

In this paper, we focus on how reliably can a Large Lanuage Model (LLM) interpret a software architecture, namely a workflow architecture (WA). Even though our initial experiments show that an LLM can answer specific questions about a WA, it is unclear how correct its answers are. To this end, we propose a methodology to assess whether an LLM can correctly interpret a WA specification. Based on the conjecture that the LLM needs to correctly answer low-abstraction level questions to answer questions at a higher abstraction level properly, we define a set of test patterns, each of them providing a template for low-abstraction level questions, together with a metric for evaluating the correctness of LLM&apos;s answers. We posit that having this metric will allow us not only to establish which LLM model works the best with WAs, but also to determine what their concrete syntax and concepts are suitable to strengthen the correctness of LLM&apos;s interpretability of WA specifications. We demonstrate the methodology on the workflow specification language developed for a currently running Horizon Europe project.

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

S - Specificky vyzkum na vysokych skolach

Rok uplatnění

2025

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

Proceedings of the 20th International Conference on Evaluation of Novel Approaches to Software Engineering (ENASE 2025)

ISBN

978-989-758-742-9

ISSN

2184-4895

e-ISSN

—

Počet stran výsledku

10

Strana od-do

608-617

Název nakladatele

SciTePress

Místo vydání

Neuveden

Místo konání akce

Porto, Portugalsko

Datum konání akce

4. 4. 2025

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

—