Publications

Journal and Magazine Articles

A literature review on optimization techniques for adaptation planning in adaptive systems: State of the art and research directions Henrichs, Elia; Lesch, Veronika; Straesser, Martin; Kounev, Samuel; Krupitzer, Christian; in Information and Software Technology (2022). 149 106940.
- [ Abstract ]
- [ BibTeX ]
- [ URL ]
- [ Download ]
Context: Recent developments in modern IT systems including internet of things, edge/fog computing, or cyber–physical systems support intelligent and seamless interaction between users and systems. This requires a reaction to changes in their environment or the system. Adaptive systems provide mechanisms for these reactions. Objective: To implement this functionality, several approaches for the planning of adaptations exist that rely on rules, utility functions, or advanced techniques, such as machine learning. As the adaptation space with possible options is often extensively huge, optimization techniques might support efficient determination of the adaptation space and identify the system’s optimal configuration. With this paper, we provide a systematic review of adaptation planning as the optimization target. Method: In this paper, we review which optimization techniques are applied for adaptation planning in adaptive systems using a systematic literature review approach. Results: We reviewed 115 paper in detail out of an initial search set of 9,588 papers. Our analysis reveals that learning techniques and genetic algorithms are by far dominant; in total, heuristics (anytime learning) are more frequently applied as exact algorithms. We observed that around 57% of the approaches target multi-objectiveness and around 30% integrate distributed optimization. As last dimension, we focused on situation-awareness, which is only supported by two approaches. Conclusion: In this paper, we provide an overview of the current state of the art of approaches that rely on optimization techniques for planning adaptations in adaptive systems and further derive open research challenges, in particular regarding the integration of distributed optimization and situation-awareness.

@article{HENRICHS2022106940, abstract = {Context: Recent developments in modern IT systems including internet of things, edge/fog computing, or cyber–physical systems support intelligent and seamless interaction between users and systems. This requires a reaction to changes in their environment or the system. Adaptive systems provide mechanisms for these reactions. Objective: To implement this functionality, several approaches for the planning of adaptations exist that rely on rules, utility functions, or advanced techniques, such as machine learning. As the adaptation space with possible options is often extensively huge, optimization techniques might support efficient determination of the adaptation space and identify the system’s optimal configuration. With this paper, we provide a systematic review of adaptation planning as the optimization target. Method: In this paper, we review which optimization techniques are applied for adaptation planning in adaptive systems using a systematic literature review approach. Results: We reviewed 115 paper in detail out of an initial search set of 9,588 papers. Our analysis reveals that learning techniques and genetic algorithms are by far dominant; in total, heuristics (anytime learning) are more frequently applied as exact algorithms. We observed that around 57% of the approaches target multi-objectiveness and around 30% integrate distributed optimization. As last dimension, we focused on situation-awareness, which is only supported by two approaches. Conclusion: In this paper, we provide an overview of the current state of the art of approaches that rely on optimization techniques for planning adaptations in adaptive systems and further derive open research challenges, in particular regarding the integration of distributed optimization and situation-awareness.}, author = {Henrichs, Elia and Lesch, Veronika and Straesser, Martin and Kounev, Samuel and Krupitzer, Christian}, journal = {Information and Software Technology}, keywords = {myown}, pages = 106940, title = {A literature review on optimization techniques for adaptation planning in adaptive systems: State of the art and research directions}, volume = 149, year = 2022 }

Full Papers

An Empirical Study of Container Image Configurations and Their Impact on Start Times Straesser, Martin; Bauer, André; Leppich, Robert; Herbst, Nikolas; Chard, Kyle; Foster, Ian; Kounev, Samuel; in 2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet Computing (CCGrid) (2023). IEEE.

In print.
- [ BibTeX ]
- [ Download ]
@inproceedings{straesser2023starttime, author = {Straesser, Martin and Bauer, André and Leppich, Robert and Herbst, Nikolas and Chard, Kyle and Foster, Ian and Kounev, Samuel}, booktitle = {2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet Computing (CCGrid)}, keywords = {myown}, note = {In print.}, publisher = {IEEE}, title = {An Empirical Study of Container Image Configurations and Their Impact on Start Times}, year = 2023 }
Autoscaler Evaluation and Configuration: A Practitioner’s Guideline Straesser, Martin; Eismann, Simon; von Kistowski, Jóakim; Bauer, André; Kounev, Samuel; in Proceedings of the 2023 ACM/SPEC International Conference on Performance Engineering (2023). Association for Computing Machinery, New York, NY, USA.

In print.
- [ BibTeX ]
- [ Download ]
@inproceedings{straesser2023guideline, address = {New York, NY, USA}, author = {Straesser, Martin and Eismann, Simon and von Kistowski, J\'{o}akim and Bauer, Andr\'{e} and Kounev, Samuel}, booktitle = {Proceedings of the 2023 ACM/SPEC International Conference on Performance Engineering}, keywords = {myown}, note = {In print.}, publisher = {Association for Computing Machinery}, series = {ICPE '23}, title = {Autoscaler Evaluation and Configuration: A Practitioner's Guideline}, year = 2023 }
A Systematic Approach for Benchmarking of Container Orchestration Frameworks Straesser, Martin; Mathiasch, Jonas; Bauer, André; Kounev, Samuel; in Proceedings of the 2023 ACM/SPEC International Conference on Performance Engineering (2023). 187–198. Association for Computing Machinery, New York, NY, USA.
- [ Abstract ]
- [ BibTeX ]
- [ URL ]
- [ Download ]
Container orchestration frameworks play a critical role in modern cloud computing paradigms such as cloud-native or serverless computing. They significantly impact the quality and cost of service deployment as they manage many performance-critical tasks such as container provisioning, scheduling, scaling, and networking. Consequently, a comprehensive performance assessment of container orchestration frameworks is essential. However, until now, there is no benchmarking approach that covers the many different tasks implemented in such platforms and supports evaluating different technology stacks. In this paper, we present a systematic approach that enables benchmarking of container orchestrators. Based on a definition of container orchestration, we define the core requirements and benchmarking scope for such platforms. Each requirement is then linked to metrics and measurement methods, and a benchmark architecture is proposed. With COFFEE, we introduce a benchmarking tool supporting the definition of complex test campaigns for container orchestration frameworks. We demonstrate the potential of our approach with case studies of the frameworks Kubernetes and Nomad in a self-hosted environment and on the Google Cloud Platform. The presented case studies focus on container startup times, crash recovery, rolling updates, and more.

@inproceedings{straesser2023orchestration, abstract = {Container orchestration frameworks play a critical role in modern cloud computing paradigms such as cloud-native or serverless computing. They significantly impact the quality and cost of service deployment as they manage many performance-critical tasks such as container provisioning, scheduling, scaling, and networking. Consequently, a comprehensive performance assessment of container orchestration frameworks is essential. However, until now, there is no benchmarking approach that covers the many different tasks implemented in such platforms and supports evaluating different technology stacks. In this paper, we present a systematic approach that enables benchmarking of container orchestrators. Based on a definition of container orchestration, we define the core requirements and benchmarking scope for such platforms. Each requirement is then linked to metrics and measurement methods, and a benchmark architecture is proposed. With COFFEE, we introduce a benchmarking tool supporting the definition of complex test campaigns for container orchestration frameworks. We demonstrate the potential of our approach with case studies of the frameworks Kubernetes and Nomad in a self-hosted environment and on the Google Cloud Platform. The presented case studies focus on container startup times, crash recovery, rolling updates, and more.}, address = {New York, NY, USA}, author = {Straesser, Martin and Mathiasch, Jonas and Bauer, Andr\'{e} and Kounev, Samuel}, booktitle = {Proceedings of the 2023 ACM/SPEC International Conference on Performance Engineering}, keywords = {myown}, pages = {187-198}, publisher = {Association for Computing Machinery}, series = {ICPE '23}, title = {A Systematic Approach for Benchmarking of Container Orchestration Frameworks}, year = 2023 }
MiSim: A Simulator for Resilience Assessment of Microservice-based Architectures Frank, Sebastian; Wagner, Lion; Hakamian, Alireza; Straesser, Martin; van Hoorn, Andre; in 2022 IEEE 22nd International Conference on Software Quality, Reliability and Security (QRS) (2022). 1014–1025. IEEE.
- [ Abstract ]
- [ BibTeX ]
- [ URL ]
- [ Download ]
Increased resilience compared to monolithic architectures is both one of the key promises of microservice-based architectures and a big challenge, e.g., due to the systems’ distributed nature. Resilience assessment through simulation requires fewer resources than the measurement-based techniques used in practice. However, there is no existing simulation approach that is suitable for a holistic resilience assessment of microservices comprised of (i) representative fault injections, (ii) common resilience mechanisms, and (iii) time-varying workloads. This paper presents MiSim — an extensible simulator for resilience assessment of microservice-based architectures. It overcomes the stated limitations of related work. MiSim fits resilience engineering practices by supporting scenario-based experiments and requiring only lightweight input models. We demonstrate how MiSim simulates (1) common resilience mechanisms — i.e., circuit breaker, connection limiter, retry, load balancer, and autoscaler — and (2) fault injections — i.e., instance/service killing and latency injections. In addition, we use TeaStore, a reference microservice-based architecture, aiming to reproduce scaling behavior from an experiment by using simulation. Our results show that MiSim allows for quantitative insights into microservice-based systems’ complex transient behavior by providing up to 25 metrics.

@inproceedings{FrankWagnerHakamianStraesserVanHoorn2022MiSim, abstract = {Increased resilience compared to monolithic architectures is both one of the key promises of microservice-based architectures and a big challenge, e.g., due to the systems’ distributed nature. Resilience assessment through simulation requires fewer resources than the measurement-based techniques used in practice. However, there is no existing simulation approach that is suitable for a holistic resilience assessment of microservices comprised of (i) representative fault injections, (ii) common resilience mechanisms, and (iii) time-varying workloads. This paper presents MiSim — an extensible simulator for resilience assessment of microservice-based architectures. It overcomes the stated limitations of related work. MiSim fits resilience engineering practices by supporting scenario-based experiments and requiring only lightweight input models. We demonstrate how MiSim simulates (1) common resilience mechanisms — i.e., circuit breaker, connection limiter, retry, load balancer, and autoscaler — and (2) fault injections — i.e., instance/service killing and latency injections. In addition, we use TeaStore, a reference microservice-based architecture, aiming to reproduce scaling behavior from an experiment by using simulation. Our results show that MiSim allows for quantitative insights into microservice-based systems’ complex transient behavior by providing up to 25 metrics.}, author = {Frank, Sebastian and Wagner, Lion and Hakamian, Alireza and Straesser, Martin and van Hoorn, Andre}, booktitle = {2022 IEEE 22nd International Conference on Software Quality, Reliability and Security (QRS)}, keywords = {myown}, pages = {1014-1025}, publisher = {IEEE}, title = {MiSim: A Simulator for Resilience Assessment of Microservice-based Architectures}, year = 2022 }
Why Is It Not Solved Yet? Challenges for Production-Ready Autoscaling Straesser, Martin; Grohmann, Johannes; von Kistowski, Jóakim; Eismann, Simon; Bauer, André; Kounev, Samuel; in Proceedings of the 2022 ACM/SPEC on International Conference on Performance Engineering (2022). 105–115. Association for Computing Machinery, New York, NY, USA.
- [ Abstract ]
- [ BibTeX ]
- [ URL ]
- [ Download ]
Autoscaling is a task of major importance in the cloud computing domain as it directly affects both operating costs and customer experience. Although there has been active research in this area for over ten years now, there is still a significant gap between the proposed methods in the literature and the deployed autoscalers in practice. Hence, many research autoscalers do not find their way into production deployments. This paper describes six core challenges that arise in production systems that are still not solved by most research autoscalers. We illustrate these problems through experiments in a realistic cloud environment with a real-world multi-service business application and show that commonly used autoscalers have various shortcomings. In addition, we analyze the behavior of overloaded services and show that these can be problematic for existing autoscalers. Generally, we analyze that these challenges are only insufficiently addressed in the literature and conclude that future scaling approaches should focus on the needs of production systems.

@inproceedings{10.1145/3489525.3511680, abstract = {Autoscaling is a task of major importance in the cloud computing domain as it directly affects both operating costs and customer experience. Although there has been active research in this area for over ten years now, there is still a significant gap between the proposed methods in the literature and the deployed autoscalers in practice. Hence, many research autoscalers do not find their way into production deployments. This paper describes six core challenges that arise in production systems that are still not solved by most research autoscalers. We illustrate these problems through experiments in a realistic cloud environment with a real-world multi-service business application and show that commonly used autoscalers have various shortcomings. In addition, we analyze the behavior of overloaded services and show that these can be problematic for existing autoscalers. Generally, we analyze that these challenges are only insufficiently addressed in the literature and conclude that future scaling approaches should focus on the needs of production systems.}, address = {New York, NY, USA}, author = {Straesser, Martin and Grohmann, Johannes and von Kistowski, J\'{o}akim and Eismann, Simon and Bauer, Andr\'{e} and Kounev, Samuel}, booktitle = {Proceedings of the 2022 ACM/SPEC on International Conference on Performance Engineering}, keywords = {myown}, pages = {105–115}, publisher = {Association for Computing Machinery}, series = {ICPE '22}, title = {Why Is It Not Solved Yet? Challenges for Production-Ready Autoscaling}, year = 2022 }
SuanMing: Explainable Prediction of Performance Degradations in Microservice Applications Grohmann, Johannes; Straesser, Martin; Chalbani, Avi; Eismann, Simon; Arian, Yair; Herbst, Nikolas; Peretz, Noam; Kounev, Samuel; in Proceedings of the 12th ACM/SPEC International Conference on Performance Engineering (ICPE) (2021). ACM, New York, NY, USA.

Acceptance Rate: 29%
- [ Abstract ]
- [ BibTeX ]
- [ URL ]
- [ DOI ]
- [ Download ]
Application performance management (APM) tools are useful to observe the performance properties of an application during production. However, APM is normally purely reactive, that is, it can only report about current or past performance degradation. Although some approaches capable of predictive application monitoring have been proposed, they can only report a predicted degradation but cannot explain its root-cause, making it hard to prevent the expected degradation. In this paper, we present SuanMing---a framework for predicting performance degradation of microservice applications running in cloud environments. SuanMing is able to predict future root causes for anticipated performance degradations and therefore aims at preventing performance degradations before they actually occur. We evaluate SuanMing on two realistic microservice applications, TeaStore and TrainTicket, and we show that our approach is able to predict and pinpoint performance degradations with an accuracy of over 90%.

@inproceedings{GrStChEiArHePeKo2021-ICPE, abstract = {Application performance management (APM) tools are useful to observe the performance properties of an application during production. However, APM is normally purely reactive, that is, it can only report about current or past performance degradation. Although some approaches capable of predictive application monitoring have been proposed, they can only report a predicted degradation but cannot explain its root-cause, making it hard to prevent the expected degradation. In this paper, we present SuanMing—a framework for predicting performance degradation of microservice applications running in cloud environments. SuanMing is able to predict future root causes for anticipated performance degradations and therefore aims at preventing performance degradations before they actually occur. We evaluate SuanMing on two realistic microservice applications, TeaStore and TrainTicket, and we show that our approach is able to predict and pinpoint performance degradations with an accuracy of over 90\%.}, address = {New York, NY, USA}, author = {Grohmann, Johannes and Straesser, Martin and Chalbani, Avi and Eismann, Simon and Arian, Yair and Herbst, Nikolas and Peretz, Noam and Kounev, Samuel}, booktitle = {Proceedings of the 12th ACM/SPEC International Conference on Performance Engineering (ICPE)}, keywords = {myown}, month = {April}, note = {Acceptance Rate: 29%}, organization = {ACM}, title = {SuanMing: Explainable Prediction of Performance Degradations in Microservice Applications}, year = 2021 }

Short Papers

Searching for the Ground Truth: Assessing the Similarity of Benchmarking Runs Bauer, André; Straesser, Martin; Leznik, Mark; Beierlieb, Lukas; Hadry, Marius; Hudson, Nathaniel; Chard, Kyle; Kounev, Samuel; Foster, Ian; in Companion of the 2023 ACM/SPEC International Conference on Performance Engineering (2023). 95–99. Association for Computing Machinery, New York, NY, USA.
- [ Abstract ]
- [ BibTeX ]
- [ URL ]
- [ Download ]
Stable and repeatable measurements are essential for comparing the performance of different systems or applications, and benchmarks are used to ensure accuracy and replication. However, if the corresponding measurements are not stable and repeatable, wrong conclusions can be drawn. To facilitate the task of determining whether the measurements are similar, we used a data set of 586 micro-benchmarks to (i) analyze the data set itself, (ii) examine our previous approach, and (iii) propose and evaluate a heuristic. To evaluate the different approaches, we perform a peer review to assess the dissimilarity of the benchmark runs. Our results show that this task is challenging even for humans and that our heuristic exhibits a sensitivity of 92%.

@inproceedings{10.1145/3578245.3584693, abstract = {Stable and repeatable measurements are essential for comparing the performance of different systems or applications, and benchmarks are used to ensure accuracy and replication. However, if the corresponding measurements are not stable and repeatable, wrong conclusions can be drawn. To facilitate the task of determining whether the measurements are similar, we used a data set of 586 micro-benchmarks to (i) analyze the data set itself, (ii) examine our previous approach, and (iii) propose and evaluate a heuristic. To evaluate the different approaches, we perform a peer review to assess the dissimilarity of the benchmark runs. Our results show that this task is challenging even for humans and that our heuristic exhibits a sensitivity of 92%.}, address = {New York, NY, USA}, author = {Bauer, Andr\'{e} and Straesser, Martin and Leznik, Mark and Beierlieb, Lukas and Hadry, Marius and Hudson, Nathaniel and Chard, Kyle and Kounev, Samuel and Foster, Ian}, booktitle = {Companion of the 2023 ACM/SPEC International Conference on Performance Engineering}, keywords = {myown}, pages = {95–99}, publisher = {Association for Computing Machinery}, series = {ICPE '23 Companion}, title = {Searching for the Ground Truth: Assessing the Similarity of Benchmarking Runs}, year = 2023 }
Automated Triage of Performance Change Points Using Time Series Analysis and Machine Learning Bauer, André; Straesser, Martin; Beierlieb, Lukas; Meissner, Maximilian; Kounev, Samuel; in Proceedings of the 2022 ACM/SPEC on International Conference on Performance Engineering (2022). ACM.
- [ BibTeX ]
- [ Download ]
@conference{bauer2022automated, author = {Bauer, André and Straesser, Martin and Beierlieb, Lukas and Meissner, Maximilian and Kounev, Samuel}, booktitle = {Proceedings of the 2022 ACM/SPEC on International Conference on Performance Engineering}, keywords = {timeseries}, month = 4, publisher = {ACM}, title = {Automated Triage of Performance Change Points Using Time Series Analysis and Machine Learning}, year = 2022 }

Poster Papers

Power to the Applications: The Vision of Continuous Decentralized Autoscaling Straesser, Martin; Geißler, Stefan; Hoßfeld, Tobias; Kounev, Samuel; in 2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet Computing Workshops (CCGridW) (2023). IEEE.

In print.
- [ BibTeX ]
- [ Download ]
@inproceedings{straesser2023decentral, author = {Straesser, Martin and Geißler, Stefan and Hoßfeld, Tobias and Kounev, Samuel}, booktitle = {2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet Computing Workshops (CCGridW)}, keywords = {myown}, note = {In print.}, publisher = {IEEE}, title = {Power to the Applications: The Vision of Continuous Decentralized Autoscaling}, year = 2023 }

Talks

Leveraging Kubernetes Source Code for Performance Simulation Straesser, Martin; Haas, Patrick; (2022).
- [ BibTeX ]
- [ URL ]
- [ Download ]
@presentation{straesser2022leveraging, author = {Straesser, Martin and Haas, Patrick}, keywords = {myown}, title = {Leveraging Kubernetes Source Code for Performance Simulation}, year = 2022 }
Container Start Times: Empirical Analysis and Predictability Straesser, Martin; (2021).
- [ BibTeX ]
- [ URL ]
- [ Download ]
@presentation{straesser2021container, author = {Straesser, Martin}, keywords = {myown}, title = {Container Start Times: Empirical Analysis and Predictability}, year = 2021 }
Predicting Performance Degradations of Black-box Microservice Applications Straesser, Martin; (2020).
- [ BibTeX ]
- [ URL ]
- [ Download ]
@presentation{straesser2020talk, author = {Straesser, Martin}, keywords = {t_talk}, title = {Predicting Performance Degradations of Black-box Microservice Applications}, year = 2020 }

Master Thesis

Predicting Performance Degradations of Microservice Applications Straesser, Martin; (2020, May). University of Wuerzburg.

Wuerzburg Software Engineering Award 2020: Best Master Thesis
- [ BibTeX ]
- [ Download ]
@mastersthesis{straesser2020predicting, author = {Straesser, Martin}, keywords = {myown}, month = {may}, note = {Wuerzburg Software Engineering Award 2020: Best Master Thesis}, school = {University of Wuerzburg}, title = {Predicting Performance Degradations of Microservice Applications}, year = 2020 }

Miscellaneous

SPEC Research Group Newsletter, Issue 12 Kounev, Samuel; van Hoorn, Andre; Straesser, Martin; Bauer, André; (2023, April).

Published by Standard Performance Evaluation Corporation (SPEC)
- [ BibTeX ]
- [ URL ]
- [ Download ]
@misc{SPEC-RG-NEWSLETTER-12, author = {Kounev, Samuel and van Hoorn, Andre and Straesser, Martin and Bauer, Andr{é}}, keywords = {myown}, month = {April}, note = {Published by Standard Performance Evaluation Corporation (SPEC)}, title = {SPEC Research Group Newsletter, Issue 12}, year = 2023 }
SPEC Research Group Newsletter, Issue 11 Kounev, Samuel; van Hoorn, Andre; Straesser, Martin; Bauer, André; (2022, April).

Published by Standard Performance Evaluation Corporation (SPEC)
- [ BibTeX ]
- [ Download ]
@misc{SPEC-RG-NEWSLETTER-11, author = {Kounev, Samuel and van Hoorn, Andre and Straesser, Martin and Bauer, Andr{é}}, keywords = {myown}, month = {April}, note = {Published by Standard Performance Evaluation Corporation (SPEC)}, title = {SPEC Research Group Newsletter, Issue 11}, year = 2022 }

Press Coverage

Universität Würzburg (Press Release): Damit der Webshop rundläuft. August 2021. [ Link ]

Journal and Magazine Articles

Full Papers

Short Papers

Poster Papers

Talks

Master Thesis

Miscellaneous

Press Coverage

Data privacy protection

Data privacy protection

Picture credits