A Combined Approach to Performance Regression Testing Resource Usage Reduction

Kód výsledku v IS VaVaI

<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F00216208%3A11320%2F25%3A10507953" target="_blank" >RIV/00216208:11320/25:10507953 - isvavai.cz</a>

Výsledek na webu

<a href="https://doi.org/10.1145/3727582.3728690" target="_blank" >https://doi.org/10.1145/3727582.3728690</a>

DOI - Digital Object Identifier

<a href="http://dx.doi.org/10.1145/3727582.3728690" target="_blank" >10.1145/3727582.3728690</a>

Jazyk výsledku

angličtina

Název v původním jazyce

A Combined Approach to Performance Regression Testing Resource Usage Reduction

Popis výsledku v původním jazyce

Performance regression testing is often seen as a natural part of the continuous integration pipeline. The underpinning layers, such as just-in-time compilation, memory mapping, and operating system characteristics, often influence performance measurement samples. To reduce such non-deterministic factors, the usual practice includes restarting the measured workload, performing warmups, and controlling environmental variability. These need to be parameterized, among others, by run count, warm-up iterations, and iteration count. Importantly, performance testing that detects performance regressions of any scale is computationally expensive due to the need to collect samples that can detect performance changes with statistical significance. To reduce the costs of performance testing, different methods for code analysis and experiment parameterization can be used. In this work, we address the challenge of identifying the optimal parameters for performance testing. Especially in environments that use just-in-time compilation, determining the required run count is non-trivial. The run count needed depends on the workload and non-deterministic factors. To address these challenges, we have developed an approach that combines several methods for parameter selection in performance testing automation. We created a simulation where these methods work together interactively, providing a dynamic environment to evaluate their effectiveness. We evaluated three controller methods on a public dataset from the GraalVM compiler. Based on our evaluation, find that the Peass method is most efficient if the change effect size of the training set mirrors the change effect size of the test set, and that the Mutations method has constant accuracy regardless of the training set data.

Název v anglickém jazyce

A Combined Approach to Performance Regression Testing Resource Usage Reduction

Popis výsledku anglicky

Performance regression testing is often seen as a natural part of the continuous integration pipeline. The underpinning layers, such as just-in-time compilation, memory mapping, and operating system characteristics, often influence performance measurement samples. To reduce such non-deterministic factors, the usual practice includes restarting the measured workload, performing warmups, and controlling environmental variability. These need to be parameterized, among others, by run count, warm-up iterations, and iteration count. Importantly, performance testing that detects performance regressions of any scale is computationally expensive due to the need to collect samples that can detect performance changes with statistical significance. To reduce the costs of performance testing, different methods for code analysis and experiment parameterization can be used. In this work, we address the challenge of identifying the optimal parameters for performance testing. Especially in environments that use just-in-time compilation, determining the required run count is non-trivial. The run count needed depends on the workload and non-deterministic factors. To address these challenges, we have developed an approach that combines several methods for parameter selection in performance testing automation. We created a simulation where these methods work together interactively, providing a dynamic environment to evaluate their effectiveness. We evaluated three controller methods on a public dataset from the GraalVM compiler. Based on our evaluation, find that the Peass method is most efficient if the change effect size of the training set mirrors the change effect size of the test set, and that the Mutations method has constant accuracy regardless of the training set data.

Druh

D - Stať ve sborníku

CEP obor

—

OECD FORD obor

10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Projekt

—

Návaznosti

S - Specificky vyzkum na vysokych skolach

Rok uplatnění

2025

Kód důvěrnosti údajů

S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů

Název statě ve sborníku

PROCEEDINGS OF THE 2025 21ST INTERNATIONAL CONFERENCE ON PREDICTIVE MODELS AND DATA ANALYTICS IN SOFTWARE ENGINEERING, PROMISE 2025

ISBN

979-8-4007-1594-5

ISSN

—

e-ISSN

—

Počet stran výsledku

10

Strana od-do

75-84

Název nakladatele

ASSOC COMPUTING MACHINERY

Místo vydání

NEW YORK

Místo konání akce

Trondheim

Datum konání akce

26. 6. 2025

Typ akce podle státní příslušnosti

WRD - Celosvětová akce

Kód UT WoS článku

001543698200009