Early Stopping of Non-productive Performance Testing Experiments Using Measurement Mutations

Result code in IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F23%3A10474396" target="_blank" >RIV/00216208:11320/23:10474396 - isvavai.cz</a>

Result on the web

<a href="https://doi.org/10.1109/SEAA60479.2023.00022" target="_blank" >https://doi.org/10.1109/SEAA60479.2023.00022</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1109/SEAA60479.2023.00022" target="_blank" >10.1109/SEAA60479.2023.00022</a>

Result language

angličtina

Original language name

Early Stopping of Non-productive Performance Testing Experiments Using Measurement Mutations

Original language description

Modern software projects often incorporate some form of performance testing into their development cycle, intending to detect changes in performance between commits or releases. Performance testing generally relies on experimental evaluation using various benchmark workloads. To detect performance changes reliably, benchmarks must be executed many times to account for variability in the measurement results. While considered best practice, this approach can become prohibitively expensive when the number of versions and benchmark workloads increases. To alleviate the cost of performance testing, we propose an approach for the early stopping of non-productive experiments that are unlikely to detect a performance bug in a particular benchmark. The stopping conditions are based on benchmark-specific thresholds determined from historical data modified to emulate the potential effects of software changes on benchmark performance. We evaluate the approach on the GraalVM benchmarking project and show that it can eliminate about 50% of the experiments if we can afford to ignore about 15% of the least significant performance changes.

Czech name

—

Czech description

—

Type

D - Article in proceedings

CEP classification

—

OECD FORD branch

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Project

—

Continuities

S - Specificky vyzkum na vysokych skolach<br>I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace

Publication year

2023

Confidentiality

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Article name in the collection

2023 49th Euromicro Conference on Software Engineering and Advanced Applications (SEAA)

ISBN

979-8-3503-4235-2

ISSN

2640-592X

e-ISSN

2376-9521

Number of pages

8

Pages from-to

86-93

Publisher name

IEEE

Place of publication

Los Alamitos

Event location

Durres, Albania

Event date

Sep 6, 2023

Type of event by nationality

WRD - Celosvětová akce

UT code for WoS article

—