Anomaly detection in the CERN cloud infrastructure
The result's identifiers
Result code in IS VaVaI
<a href="https://www.isvavai.cz/riv?ss=detail&h=RIV%2F61389005%3A_____%2F21%3A00604044" target="_blank" >RIV/61389005:_____/21:00604044 - isvavai.cz</a>
Result on the web
<a href="https://doi.org/10.1051/epjconf/202125102011" target="_blank" >https://doi.org/10.1051/epjconf/202125102011</a>
DOI - Digital Object Identifier
<a href="http://dx.doi.org/10.1051/epjconf/202125102011" target="_blank" >10.1051/epjconf/202125102011</a>
Alternative languages
Result language
angličtina
Original language name
Anomaly detection in the CERN cloud infrastructure
Original language description
Anomaly detection in the CERN OpenStack cloud is a challenging task due to the large scale of the computing infrastructure and, consequently, the large volume of monitoring data to analyse. The current solution to spot anomalous servers in the cloud infrastructure relies on a threshold-based alarming system carefully set by the system managers on the performance metrics of each infrastructure’s component. This contribution explores fully automated, unsupervised machine learning solutions in the anomaly detection field for time series metrics, by adapting both traditional and deep learning approaches. The paper describes a novel end-to-end data analytics pipeline implemented to digest the large amount of monitoring data and to expose anomalies to the system managers. The pipeline relies solely on open-source tools and frameworks, such as Spark, Apache Airflow, Kubernetes, Grafana, Elasticsearch. In addition, an approach to build annotated datasets from the CERN cloud monitoring data is reported. Finally, a preliminary performance of a number of anomaly detection algorithms is evaluated by using the aforementioned annotated datasets.
Czech name
—
Czech description
—
Classification
Type
D - Article in proceedings
CEP classification
—
OECD FORD branch
10201 - Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
Result continuities
Project
—
Continuities
I - Institucionalni podpora na dlouhodoby koncepcni rozvoj vyzkumne organizace
Others
Publication year
2021
Confidentiality
S - Úplné a pravdivé údaje o projektu nepodléhají ochraně podle zvláštních právních předpisů
Data specific for result type
Article name in the collection
EPJ Web of Conferences
ISBN
—
ISSN
2100-014X
e-ISSN
2100-014X
Number of pages
10
Pages from-to
02011
Publisher name
EDP Sciences
Place of publication
Les Ulis
Event location
Virtual Event
Event date
May 17, 2021
Type of event by nationality
WRD - Celosvětová akce
UT code for WoS article
001329391600019