Interpreting Workflow Architectures by LLMs

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F25%3A10499966" target="_blank" >RIV/00216208:11320/25:10499966 - isvavai.cz</a>

Result on the web

<a href="https://doi.org/10.5220/0013358000003928" target="_blank" >https://doi.org/10.5220/0013358000003928</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.5220/0013358000003928" target="_blank" >10.5220/0013358000003928</a>

Result language

angličtina

Original language name

Interpreting Workflow Architectures by LLMs

Original language description

In this paper, we focus on how reliably can a Large Lanuage Model (LLM) interpret a software architecture, namely a workflow architecture (WA). Even though our initial experiments show that an LLM can answer specific questions about a WA, it is unclear how correct its answers are. To this end, we propose a methodology to assess whether an LLM can correctly interpret a WA specification. Based on the conjecture that the LLM needs to correctly answer low-abstraction level questions to answer questions at a higher abstraction level properly, we define a set of test patterns, each of them providing a template for low-abstraction level questions, together with a metric for evaluating the correctness of LLM&apos;s answers. We posit that having this metric will allow us not only to establish which LLM model works the best with WAs, but also to determine what their concrete syntax and concepts are suitable to strengthen the correctness of LLM&apos;s interpretability of WA specifications. We demonstrate the methodology on the workflow specification language developed for a currently running Horizon Europe project.

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

S - Specificky vyzkum na vysokych skolach

Publication year

2025

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

Proceedings of the 20th International Conference on Evaluation of Novel Approaches to Software Engineering (ENASE 2025)

ISBN

978-989-758-742-9

ISSN

2184-4895

e-ISSN

—

Number of pages

10

Pages from-to

608-617

Publisher name

SciTePress

Place of publication

Neuveden

Event location

Porto, Portugalsko

Event date

Apr 4, 2025

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

—