Publications - Chair of Computer Science II

Full Papers

Generating Executable Microservice Applications for Performance Benchmarking. Lubas, Yannik; Straesser, Martin; Bauer, André; Kounev, Samuel; in Proceedings of the 16th ACM/SPEC International Conference on Performance Engineering (2025). 31–44. Association for Computing Machinery, Toronto ON, Canada.
- [ Abstract ]
- [ BibTeX ]
- [ URL ]
- [ DOI ]
- [ Download ]
Microservice applications are the building blocks of modern cloud applications. As such, their performance aspects have been receiving increasing attention in the software engineering community. However, many microservice performance studies use only a small set of popular microservice test applications for experiments, questioning the applicability of their approaches in practice. Researchers currently lack the opportunity to collect large and diverse datasets containing performance metrics of microservices. This is because popular test applications only represent specific technology stacks and often come with custom benchmark tooling (e.g., load generation and monitoring). In this paper, we present Creo, a framework for generating microservice applications that (1) are fully executable, (2) have configurable properties and resource usage profiles, and (3) have built-in support for standardized monitoring, load generation, and deployment. Our approach enables researchers to run experiments with diverse microservice applications with minimal effort. We demonstrate the value of our approach in the context of two use cases. First, we show that using generated applications when training machine learning models for predicting performance degradation can improve the prediction accuracy. Second, we evaluate a recent approach for performance anomaly classification on a set of generated applications highlighting strengths and weaknesses not discussed in the original work.

@inproceedings{10.1145/3676151.3719361, abstract = {Microservice applications are the building blocks of modern cloud applications. As such, their performance aspects have been receiving increasing attention in the software engineering community. However, many microservice performance studies use only a small set of popular microservice test applications for experiments, questioning the applicability of their approaches in practice. Researchers currently lack the opportunity to collect large and diverse datasets containing performance metrics of microservices. This is because popular test applications only represent specific technology stacks and often come with custom benchmark tooling (e.g., load generation and monitoring). In this paper, we present Creo, a framework for generating microservice applications that (1) are fully executable, (2) have configurable properties and resource usage profiles, and (3) have built-in support for standardized monitoring, load generation, and deployment. Our approach enables researchers to run experiments with diverse microservice applications with minimal effort. We demonstrate the value of our approach in the context of two use cases. First, we show that using generated applications when training machine learning models for predicting performance degradation can improve the prediction accuracy. Second, we evaluate a recent approach for performance anomaly classification on a set of generated applications highlighting strengths and weaknesses not discussed in the original work.}, address = {New York, NY, USA}, author = {Lubas, Yannik and Straesser, Martin and Bauer, Andr\'{e} and Kounev, Samuel}, booktitle = {Proceedings of the 16th ACM/SPEC International Conference on Performance Engineering}, keywords = {microservices}, pages = {31–44}, publisher = {Association for Computing Machinery}, series = {ICPE '25}, title = {Generating Executable Microservice Applications for Performance Benchmarking}, year = 2025 }

Short Papers

Unveiling Temporal Performance Deviation: Leveraging Clustering in Microservices Performance Analysis. Bauer, André; Dittus, Timo; Straesser, Martin; Kamatar, Alok; Baughman, Matt; Beierlieb, Lukas; Hadry, Marius; Grillmeyer, Daniel; Lubas, Yannik; Kounev, Samuel; Foster, Ian; Chard, Kyle; in Companion of the 15th ACM/SPEC International Conference on Performance Engineering (2024). 72–76. Association for Computing Machinery, New York, NY, USA.
- [ Abstract ]
- [ BibTeX ]
- [ URL ]
- [ Download ]
As the market for cloud computing continues to grow, an increasing number of users are deploying applications as microservices. The shift introduces unique challenges in identifying and addressing performance issues, particularly within large and complex infrastructures. To address this challenge, we propose a methodology that unveils temporal performance deviations in microservices by clustering containers based on their performance characteristics at different time intervals. Showcasing our methodology on the Alibaba dataset, we found both stable and dynamic performance patterns, providing a valuable tool for enhancing overall performance and reliability in modern application landscapes.

@inproceedings{10.1145/3629527.3651843, abstract = {As the market for cloud computing continues to grow, an increasing number of users are deploying applications as microservices. The shift introduces unique challenges in identifying and addressing performance issues, particularly within large and complex infrastructures. To address this challenge, we propose a methodology that unveils temporal performance deviations in microservices by clustering containers based on their performance characteristics at different time intervals. Showcasing our methodology on the Alibaba dataset, we found both stable and dynamic performance patterns, providing a valuable tool for enhancing overall performance and reliability in modern application landscapes.}, address = {New York, NY, USA}, author = {Bauer, André and Dittus, Timo and Straesser, Martin and Kamatar, Alok and Baughman, Matt and Beierlieb, Lukas and Hadry, Marius and Grillmeyer, Daniel and Lubas, Yannik and Kounev, Samuel and Foster, Ian and Chard, Kyle}, booktitle = {Companion of the 15th ACM/SPEC International Conference on Performance Engineering}, keywords = {descartes}, pages = {72–76}, publisher = {Association for Computing Machinery}, series = {ICPE '24 Companion}, title = {Unveiling Temporal Performance Deviation: Leveraging Clustering in Microservices Performance Analysis}, year = 2024 }

Workshop Papers

Microservice Applications and Their Workloads on GitHub. Lubas, Yannik; Straesser, Martin; Rohwer, Ivo; Kounev, Samuel; Bauer, André; in Companion of the 16th ACM/SPEC International Conference on Performance Engineering (2025). 199–205. Association for Computing Machinery, Toronto ON, Canada.
- [ Abstract ]
- [ BibTeX ]
- [ URL ]
- [ DOI ]
- [ Download ]
Many cloud applications employ a microservice architecture to reap its various benefits, such as independent scalability and development. However, microservices' distributed nature comes with its own challenges, which the literature tries to combat with various approaches in domains, such as autoscaling or service placement. Researchers, especially performance engineers, need representative microservice applications as a basis to develop and evaluate their approaches. While the literature proposes several reference and benchmark applications, the set of commonly used applications is rather limited. Further, the representativeness of reference applications for industry practices has been challenged, raising questions about the applicability of existing approaches. To investigate possible alternatives to and inspirations for microservice testbeds, we conduct an exploratory search of microservice applications and their workloads by mining GitHub repositories. Our work provides two datasets containing 553 applications and 8 workload dataset repositories, respectively. In addition, we provide a first analysis of the collected data that future work can build upon.

@inproceedings{10.1145/3680256.3721321, abstract = {Many cloud applications employ a microservice architecture to reap its various benefits, such as independent scalability and development. However, microservices' distributed nature comes with its own challenges, which the literature tries to combat with various approaches in domains, such as autoscaling or service placement. Researchers, especially performance engineers, need representative microservice applications as a basis to develop and evaluate their approaches. While the literature proposes several reference and benchmark applications, the set of commonly used applications is rather limited. Further, the representativeness of reference applications for industry practices has been challenged, raising questions about the applicability of existing approaches. To investigate possible alternatives to and inspirations for microservice testbeds, we conduct an exploratory search of microservice applications and their workloads by mining GitHub repositories. Our work provides two datasets containing 553 applications and 8 workload dataset repositories, respectively. In addition, we provide a first analysis of the collected data that future work can build upon.}, address = {New York, NY, USA}, author = {Lubas, Yannik and Straesser, Martin and Rohwer, Ivo and Kounev, Samuel and Bauer, Andr\'{e}}, booktitle = {Companion of the 16th ACM/SPEC International Conference on Performance Engineering}, keywords = {t_workshop}, pages = {199–205}, publisher = {Association for Computing Machinery}, series = {ICPE '25}, title = {Microservice Applications and Their Workloads on GitHub}, year = 2025 }

Book Chapters

Microservice Benchmark Applications. von Kistowski, Jóakim; Eismann, Simon; Lubas, Yannik; Schmitt, Norbert; Bauer, André; Grohmann, Johannes; Kounev, Samuel; in Systems Benchmarking: For Scientists and Engineers (2025). 305–321. Springer Nature Switzerland, Cham.
- [ Abstract ]
- [ BibTeX ]
- [ URL ]
- [ DOI ]
This chapter provides an overview of well-established microservice benchmark applications used in research. Further, it describes TeaStore as a selected example in more detail. Based on this description, we also showcase two case studies using TeaStore as an example of the usage of a microservice benchmark in practice.

@inbook{vonKistowski2025, abstract = {This chapter provides an overview of well-established microservice benchmark applications used in research. Further, it describes TeaStore as a selected example in more detail. Based on this description, we also showcase two case studies using TeaStore as an example of the usage of a microservice benchmark in practice.}, address = {Cham}, author = {von Kistowski, J{\'o}akim and Eismann, Simon and Lubas, Yannik and Schmitt, Norbert and Bauer, Andr{\'e} and Grohmann, Johannes and Kounev, Samuel}, booktitle = {Systems Benchmarking: For Scientists and Engineers}, keywords = {t_bookchapter}, pages = {305–321}, publisher = {Springer Nature Switzerland}, title = {Microservice Benchmark Applications}, year = 2025 }

Talks

Generating Microservice Applications for Performance Benchmarking Lubas, Yannik; Straesser, Martin; Kounev, Samuel; (2023).
- [ BibTeX ]
- [ Download ]
@presentation{lubas2023generating, author = {Lubas, Yannik and Straesser, Martin and Kounev, Samuel}, keywords = {microservices}, title = {Generating Microservice Applications for Performance Benchmarking}, year = 2023 }

Picture credits