Effectiveness of Combinatorial Interaction Testing in Test Automation - An Industrial Case Study

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F68407700%3A21230%2F25%3A00388886" target="_blank" >RIV/68407700:21230/25:00388886 - isvavai.cz</a>

Výsledek na webu

<a href="https://doi.org/10.1145/3756681.3757016" target="_blank" >https://doi.org/10.1145/3756681.3757016</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1145/3756681.3757016" target="_blank" >10.1145/3756681.3757016</a>

Jazyk výsledku

angličtina

Název v původním jazyce

Effectiveness of Combinatorial Interaction Testing in Test Automation - An Industrial Case Study

Popis výsledku v původním jazyce

Combinatorial Interaction Testing (CIT) is an established software testing method having a wide application area. While many studies have been conducted to investigate the effectiveness of CIT through various simulations and mutation testing, there needs to be more evidence regarding its effectiveness in real industrial testing processes. In response to this certain gap, this work investigates the practical implications of CIT application in a real software project with real historical defects as well as artificial defects created to resemble historical defects. The evidence presented encompasses two distinct studies: firstly, the optimization of test data input into user interface forms, and secondly, the management of system configurations for testing purposes. These studies employ the Tricentis Test Automation for ServiceNow (TTA-SNOW) outcome as the system under evaluation. We cautiously evaluated the impact of CIT tool on various metrics, such as time spent by test design, test automation and test execution, test suite size, and defect detection compared to an intuitive approach taken by test engineers without proper application of CIT. Our findings demonstrate that CIT has the potential to reduce test suite size and execution time while maintaining or improving defect detection rates. Specifically, in Study 1, CIT reduced average defect detection time from 5.11 hours to 3.39 hours. In Study 2, focusing on system configurations, this time was reduced from 4.38 hours to 2.28 hours. These findings provide valuable insights into CIT's practical benefits in industrial contexts.

Název v anglickém jazyce

Effectiveness of Combinatorial Interaction Testing in Test Automation - An Industrial Case Study

Popis výsledku anglicky

Combinatorial Interaction Testing (CIT) is an established software testing method having a wide application area. While many studies have been conducted to investigate the effectiveness of CIT through various simulations and mutation testing, there needs to be more evidence regarding its effectiveness in real industrial testing processes. In response to this certain gap, this work investigates the practical implications of CIT application in a real software project with real historical defects as well as artificial defects created to resemble historical defects. The evidence presented encompasses two distinct studies: firstly, the optimization of test data input into user interface forms, and secondly, the management of system configurations for testing purposes. These studies employ the Tricentis Test Automation for ServiceNow (TTA-SNOW) outcome as the system under evaluation. We cautiously evaluated the impact of CIT tool on various metrics, such as time spent by test design, test automation and test execution, test suite size, and defect detection compared to an intuitive approach taken by test engineers without proper application of CIT. Our findings demonstrate that CIT has the potential to reduce test suite size and execution time while maintaining or improving defect detection rates. Specifically, in Study 1, CIT reduced average defect detection time from 5.11 hours to 3.39 hours. In Study 2, focusing on system configurations, this time was reduced from 4.38 hours to 2.28 hours. These findings provide valuable insights into CIT's practical benefits in industrial contexts.

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace

Rok uplatnění

2025

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

EASE '25: Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering

ISBN

979-8-4007-1385-9

ISSN

—

e-ISSN

—

Počet stran výsledku

7

Strana od-do

659-665

Název nakladatele

ACM Press

Místo vydání

New York

Místo konání akce

Istanbul

Datum konání akce

17. 6. 2025

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

001668832700062