Publications

• [ Article ]
• [ Incollection ]
• [ Inproceedings ]
• [ Misc ]
• [ PhD thesis ]
• [ Proceedings ]
• [ Techreport ]

• 
  The State of Serverless Applications: Collection, Characterization, and Community Consensus Eismann, Simon; Scheuner, Joel; van Eyk, Erwin; Schwinger, Maximilian; Grohmann, Johannes; Herbst, Nikolas; Abad, Cristina; Iosup, Alexandru; in Transactions on Software Engineering (2022). 48(10) 4152–4166.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Over the last five years, all major cloud platform providers have increased their serverless offerings. Many early adopters report significant benefits for serverless-based over traditional applications, and many companies are considering moving to serverless themselves. However, currently there exist only few, scattered, and sometimes even conflicting reports on when serverless applications are well suited and what the best practices for their implementation are. We address this problem in the present study about the state of serverless applications. We collect descriptions of 89 serverless applications from open-source projects, academic literature, industrial literature, and domain-specific feedback. We analyze 16 characteristics that describe why and when successful adopters are using serverless applications, and how they are building them. We further compare the results of our characterization study to 10 existing, mostly industrial, studies and datasets; this allows us to identify points of consensus across multiple studies, investigate points of disagreement, and overall confirm the validity of our results. The results of this study can help managers to decide if they should adopt serverless technology, engineers to learn about current practices of building serverless applications, and researchers and platform providers to better understand the current landscape of serverless applications.

  @article{eismann2021state, abstract = {Over the last five years, all major cloud platform providers have increased their serverless offerings. Many early adopters report significant benefits for serverless-based over traditional applications, and many companies are considering moving to serverless themselves. However, currently there exist only few, scattered, and sometimes even conflicting reports on when serverless applications are well suited and what the best practices for their implementation are. We address this problem in the present study about the state of serverless applications. We collect descriptions of 89 serverless applications from open-source projects, academic literature, industrial literature, and domain-specific feedback. We analyze 16 characteristics that describe why and when successful adopters are using serverless applications, and how they are building them. We further compare the results of our characterization study to 10 existing, mostly industrial, studies and datasets; this allows us to identify points of consensus across multiple studies, investigate points of disagreement, and overall confirm the validity of our results. The results of this study can help managers to decide if they should adopt serverless technology, engineers to learn about current practices of building serverless applications, and researchers and platform providers to better understand the current landscape of serverless applications.}, author = {Eismann, Simon and Scheuner, Joel and van Eyk, Erwin and Schwinger, Maximilian and Grohmann, Johannes and Herbst, Nikolas and Abad, Cristina and Iosup, Alexandru}, journal = {Transactions on Software Engineering}, keywords = {myown}, number = 10, pages = {4152- 4166}, title = {The State of Serverless Applications: Collection, Characterization, and Community Consensus}, volume = 48, year = 2022 }

  %0 Journal Article %1 eismann2021state %A Eismann, Simon %A Scheuner, Joel %A van Eyk, Erwin %A Schwinger, Maximilian %A Grohmann, Johannes %A Herbst, Nikolas %A Abad, Cristina %A Iosup, Alexandru %D 2022 %J Transactions on Software Engineering %N 10 %P 4152- 4166 %T The State of Serverless Applications: Collection, Characterization, and Community Consensus %U /brokenurl#https://doi.org/10.1109/TSE.2021.3113940 %V 48 %X Over the last five years, all major cloud platform providers have increased their serverless offerings. Many early adopters report significant benefits for serverless-based over traditional applications, and many companies are considering moving to serverless themselves. However, currently there exist only few, scattered, and sometimes even conflicting reports on when serverless applications are well suited and what the best practices for their implementation are. We address this problem in the present study about the state of serverless applications. We collect descriptions of 89 serverless applications from open-source projects, academic literature, industrial literature, and domain-specific feedback. We analyze 16 characteristics that describe why and when successful adopters are using serverless applications, and how they are building them. We further compare the results of our characterization study to 10 existing, mostly industrial, studies and datasets; this allows us to identify points of consensus across multiple studies, investigate points of disagreement, and overall confirm the validity of our results. The results of this study can help managers to decide if they should adopt serverless technology, engineers to learn about current practices of building serverless applications, and researchers and platform providers to better understand the current landscape of serverless applications.
• 
  Serverless Computing (Dagstuhl Seminar 21201) Abad, Cristina; Foster, Ian T.; Herbst, Nikolas; Iosup, Alexandru; in Dagstuhl Reports, (C. Abad; I. T. Foster; N. Herbst; A. Iosup, eds.) (2021). 11(4) 34–93.
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  @article{abad_et_al:DagRep.11.4.34, address = {Dagstuhl, Germany}, annote = {Keywords: Cloud computing, Cloud continuum, data-driven, design patterns, DevOps, experimentation, model-driven, serverless computing, simulation, software architecture, systems management, vision}, author = {Abad, Cristina and Foster, Ian T. and Herbst, Nikolas and Iosup, Alexandru}, editor = {Abad, Cristina and Foster, Ian T. and Herbst, Nikolas and Iosup, Alexandru}, journal = {Dagstuhl Reports}, keywords = {myown}, number = 4, pages = {34–93}, publisher = {Schloss Dagstuhl – Leibniz-Zentrum f{\"u}r Informatik}, title = {{Serverless Computing (Dagstuhl Seminar 21201)}}, volume = 11, year = 2021 }

  %0 Journal Article %1 abad_et_al:DagRep.11.4.34 %A Abad, Cristina %A Foster, Ian T. %A Herbst, Nikolas %A Iosup, Alexandru %C Dagstuhl, Germany %D 2021 %E Abad, Cristina %E Foster, Ian T. %E Herbst, Nikolas %E Iosup, Alexandru %I Schloss Dagstuhl -- Leibniz-Zentrum für Informatik %J Dagstuhl Reports %N 4 %P 34--93 %T Serverless Computing (Dagstuhl Seminar 21201) %U https://drops.dagstuhl.de/opus/volltexte/2021/14798 %V 11 %Z Keywords: Cloud computing, Cloud continuum, data-driven, design patterns, DevOps, experimentation, model-driven, serverless computing, simulation, software architecture, systems management, vision
• 
  SARDE: A Framework for Continuous and Self-Adaptive Resource Demand Estimation Grohmann, Johannes; Eismann, Simon; Bauer, André; Spinner, Simon; Blum, Johannes; Herbst, Nikolas; Kounev, Samuel; in ACM Transactions on Autonomous and Adaptive Systems (2021). 15(2)
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Resource demands are crucial parameters for modeling and predicting the performance of software systems. Currently, resource demand estimators are usually executed once for system analysis. However, the monitored system, as well as the resource demand itself, are subject to constant change in runtime environments. These changes additionally impact the applicability, the required parametrization as well as the resulting accuracy of individual estimation approaches. Over time, this leads to invalid or outdated estimates, which in turn negatively influence the decision-making of adaptive systems. In this article, we present SARDE, a framework for self-adaptive resource demand estimation in continuous environments. SARDE dynamically and continuously tunes, selects, and executes an ensemble of resource demand estimation approaches to adapt to changes in the environment. This creates an autonomous and unsupervised ensemble estimation technique, providing reliable resource demand estimations in dynamic environments. We evaluate SARDE using two realistic datasets. One set of different micro-benchmarks reflecting different possible system states and one dataset consisting of a continuously running application in a changing environment. Our results show that by continuously applying online optimization, selection and estimation, SARDE is able to efficiently adapt to the online trace and reduce the model error using the resulting ensemble technique.

  @article{GrEiBaSpBlHeKo2021-TaaS-SARDE, abstract = {Resource demands are crucial parameters for modeling and predicting the performance of software systems. Currently, resource demand estimators are usually executed once for system analysis. However, the monitored system, as well as the resource demand itself, are subject to constant change in runtime environments. These changes additionally impact the applicability, the required parametrization as well as the resulting accuracy of individual estimation approaches. Over time, this leads to invalid or outdated estimates, which in turn negatively influence the decision-making of adaptive systems. In this article, we present SARDE, a framework for self-adaptive resource demand estimation in continuous environments. SARDE dynamically and continuously tunes, selects, and executes an ensemble of resource demand estimation approaches to adapt to changes in the environment. This creates an autonomous and unsupervised ensemble estimation technique, providing reliable resource demand estimations in dynamic environments. We evaluate SARDE using two realistic datasets. One set of different micro-benchmarks reflecting different possible system states and one dataset consisting of a continuously running application in a changing environment. Our results show that by continuously applying online optimization, selection and estimation, SARDE is able to efficiently adapt to the online trace and reduce the model error using the resulting ensemble technique.}, address = {New York, NY, USA}, author = {Grohmann, Johannes and Eismann, Simon and Bauer, Andr\'{e} and Spinner, Simon and Blum, Johannes and Herbst, Nikolas and Kounev, Samuel}, journal = {ACM Transactions on Autonomous and Adaptive Systems}, keywords = {myown}, month = {jun}, number = 2, publisher = {Association for Computing Machinery}, title = {SARDE: A Framework for Continuous and Self-Adaptive Resource Demand Estimation}, volume = 15, year = 2021 }

  %0 Journal Article %1 GrEiBaSpBlHeKo2021-TaaS-SARDE %A Grohmann, Johannes %A Eismann, Simon %A Bauer, André %A Spinner, Simon %A Blum, Johannes %A Herbst, Nikolas %A Kounev, Samuel %C New York, NY, USA %D 2021 %I Association for Computing Machinery %J ACM Transactions on Autonomous and Adaptive Systems %N 2 %T SARDE: A Framework for Continuous and Self-Adaptive Resource Demand Estimation %U https://doi.org/10.1145/3463369 %V 15 %X Resource demands are crucial parameters for modeling and predicting the performance of software systems. Currently, resource demand estimators are usually executed once for system analysis. However, the monitored system, as well as the resource demand itself, are subject to constant change in runtime environments. These changes additionally impact the applicability, the required parametrization as well as the resulting accuracy of individual estimation approaches. Over time, this leads to invalid or outdated estimates, which in turn negatively influence the decision-making of adaptive systems. In this article, we present SARDE, a framework for self-adaptive resource demand estimation in continuous environments. SARDE dynamically and continuously tunes, selects, and executes an ensemble of resource demand estimation approaches to adapt to changes in the environment. This creates an autonomous and unsupervised ensemble estimation technique, providing reliable resource demand estimations in dynamic environments. We evaluate SARDE using two realistic datasets. One set of different micro-benchmarks reflecting different possible system states and one dataset consisting of a continuously running application in a changing environment. Our results show that by continuously applying online optimization, selection and estimation, SARDE is able to efficiently adapt to the online trace and reduce the model error using the resulting ensemble technique.
• 
  Serverless Applications:Why, When, and How? Eismann, Simon; Joel, Scheuner; van Eyk, Erwin; Schwinger, Maximilian; Grohmann, Johannes; Herbst, Nikolas; Abad, Cristina; Iosup, Alexandru; in IEEE Software (2021). 38(1) 32–39.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  • [ Download ]
  Serverless computing shows good promise for efficiency and ease-of-use. Yet, there are only a few, scattered and sometimes conflicting reports on questions such as Why do so many companies adopt serverless?, When are serverless applications well suited?, and How are serverless applications currently implemented? To address these questions, we analyze 89 serverless applications from open-source projects, industrial sources, academic literature, and scientific computing—the most extensive study to date.

  @article{eismann2020serverless, abstract = {Serverless computing shows good promise for efficiency and ease-of-use. Yet, there are only a few, scattered and sometimes conflicting reports on questions such as Why do so many companies adopt serverless?, When are serverless applications well suited?, and How are serverless applications currently implemented? To address these questions, we analyze 89 serverless applications from open-source projects, industrial sources, academic literature, and scientific computing—the most extensive study to date.}, author = {Eismann, Simon and Joel, Scheuner and van Eyk, Erwin and Schwinger, Maximilian and Grohmann, Johannes and Herbst, Nikolas and Abad, Cristina and Iosup, Alexandru}, journal = {IEEE Software}, keywords = {t_journalmagazine}, number = 1, pages = {32–39}, title = {Serverless Applications:Why, When, and How?}, volume = 38, year = 2021 }

  %0 Journal Article %1 eismann2020serverless %A Eismann, Simon %A Joel, Scheuner %A van Eyk, Erwin %A Schwinger, Maximilian %A Grohmann, Johannes %A Herbst, Nikolas %A Abad, Cristina %A Iosup, Alexandru %D 2021 %J IEEE Software %N 1 %P 32--39 %R 10.1109/MS.2020.3023302 %T Serverless Applications:Why, When, and How? %U https://doi.org/10.1109/MS.2020.3023302 %V 38 %X Serverless computing shows good promise for efficiency and ease-of-use. Yet, there are only a few, scattered and sometimes conflicting reports on questions such as Why do so many companies adopt serverless?, When are serverless applications well suited?, and How are serverless applications currently implemented? To address these questions, we analyze 89 serverless applications from open-source projects, industrial sources, academic literature, and scientific computing—the most extensive study to date.
• 
  A Taxonomy of Techniques for SLO Failure Prediction in Software Systems Grohmann, Johannes; Herbst, Nikolas; Chalbani, Avi; Arian, Yair; Peretz, Noam; Kounev, Samuel; in Computers (2020). 9(1) 10.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  Failure prediction is an important aspect of self-aware computing systems. Therefore, a multitude of different approaches has been proposed in the literature over the past few years. In this work, we propose a taxonomy for organizing works focusing on the prediction of Service Level Objective (SLO) failures. Our taxonomy classifies related work along the dimensions of the prediction target (e.g., anomaly detection, performance prediction, or failure prediction), the time horizon (e.g., detection or prediction, online or offline application), and the applied modeling type (e.g., time series forecasting, machine learning, or queueing theory). The classification is derived based on a systematic mapping of relevant papers in the area. Additionally, we give an overview of different techniques in each sub-group and address remaining challenges in order to guide future research.

  @article{grohmann2020-MDPI-taxonomy, abstract = {Failure prediction is an important aspect of self-aware computing systems. Therefore, a multitude of different approaches has been proposed in the literature over the past few years. In this work, we propose a taxonomy for organizing works focusing on the prediction of Service Level Objective (SLO) failures. Our taxonomy classifies related work along the dimensions of the prediction target (e.g., anomaly detection, performance prediction, or failure prediction), the time horizon (e.g., detection or prediction, online or offline application), and the applied modeling type (e.g., time series forecasting, machine learning, or queueing theory). The classification is derived based on a systematic mapping of relevant papers in the area. Additionally, we give an overview of different techniques in each sub-group and address remaining challenges in order to guide future research.}, author = {Grohmann, Johannes and Herbst, Nikolas and Chalbani, Avi and Arian, Yair and Peretz, Noam and Kounev, Samuel}, journal = {Computers}, keywords = {myown}, number = 1, pages = 10, publisher = {Multidisciplinary Digital Publishing Institute (MDPI)}, title = {A Taxonomy of Techniques for SLO Failure Prediction in Software Systems}, volume = 9, year = 2020 }

  %0 Journal Article %1 grohmann2020-MDPI-taxonomy %A Grohmann, Johannes %A Herbst, Nikolas %A Chalbani, Avi %A Arian, Yair %A Peretz, Noam %A Kounev, Samuel %D 2020 %I Multidisciplinary Digital Publishing Institute (MDPI) %J Computers %N 1 %P 10 %R https://doi.org/10.3390/computers9010010 %T A Taxonomy of Techniques for SLO Failure Prediction in Software Systems %U https://doi.org/10.3390/computers9010010 %V 9 %X Failure prediction is an important aspect of self-aware computing systems. Therefore, a multitude of different approaches has been proposed in the literature over the past few years. In this work, we propose a taxonomy for organizing works focusing on the prediction of Service Level Objective (SLO) failures. Our taxonomy classifies related work along the dimensions of the prediction target (e.g., anomaly detection, performance prediction, or failure prediction), the time horizon (e.g., detection or prediction, online or offline application), and the applied modeling type (e.g., time series forecasting, machine learning, or queueing theory). The classification is derived based on a systematic mapping of relevant papers in the area. Additionally, we give an overview of different techniques in each sub-group and address remaining challenges in order to guide future research.
• 
  Time Series Forecasting for Self-Aware Systems Bauer, André; Züfle, Marwin; Herbst, Nikolas; Zehe, Albin; Hotho, Andreas; Kounev, Samuel; in Proceedings of the IEEE (2020). 108(7) 1068–1093.
  • [ BibTeX ]
  • [ EndNote ]
  • [ DOI ]
  @article{BaZuHeZeHoKo-PIEEE-Forecasting, author = {Bauer, Andr{\'e} and Z{\"u}fle, Marwin and Herbst, Nikolas and Zehe, Albin and Hotho, Andreas and Kounev, Samuel}, journal = {Proceedings of the IEEE}, keywords = {myown}, month = 7, number = 7, pages = {1068 - 1093}, title = {{Time Series Forecasting for Self-Aware Systems}}, volume = 108, year = 2020 }

  %0 Journal Article %1 BaZuHeZeHoKo-PIEEE-Forecasting %A Bauer, André %A Züfle, Marwin %A Herbst, Nikolas %A Zehe, Albin %A Hotho, Andreas %A Kounev, Samuel %D 2020 %J Proceedings of the IEEE %N 7 %P 1068 - 1093 %R 10.1109/JPROC.2020.2983857 %T Time Series Forecasting for Self-Aware Systems %V 108
• 
  Agile Scalability Engineering: The ScrumScale Method Brataas, Gunnar; Hanssen, Geir; Herbst, Nikolas; van Hoorn, André; in IEEE Software (2019). 37(5) 77–85.

  to appear
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  Scalability is a property that must be carefully designed into a system. A case study in the largest Norwegian public portal, Altinn, illustrates how developers and scalability experts improved scalability and spent less time during scalability testing. With minor adjustments to an agile development process, stakeholders spend more up-front time together.

  @article{ScrumScale2019IEEESW, abstract = {Scalability is a property that must be carefully designed into a system. A case study in the largest Norwegian public portal, Altinn, illustrates how developers and scalability experts improved scalability and spent less time during scalability testing. With minor adjustments to an agile development process, stakeholders spend more up-front time together.}, author = {Brataas, Gunnar and Hanssen, Geir and Herbst, Nikolas and van Hoorn, Andr{\'e}}, journal = {IEEE Software}, keywords = {t_journalmagazine}, month = {June}, note = {to appear}, number = 5, pages = {77-85}, publisher = {IEEE}, title = {Agile Scalability Engineering: The ScrumScale Method}, volume = 37, year = 2019 }

  %0 Journal Article %1 ScrumScale2019IEEESW %A Brataas, Gunnar %A Hanssen, Geir %A Herbst, Nikolas %A van Hoorn, André %D 2019 %I IEEE %J IEEE Software %N 5 %P 77-85 %R 10.1109/MS.2019.2923184 %T Agile Scalability Engineering: The ScrumScale Method %U https://ieeexplore.ieee.org/document/8736723 %V 37 %X Scalability is a property that must be carefully designed into a system. A case study in the largest Norwegian public portal, Altinn, illustrates how developers and scalability experts improved scalability and spent less time during scalability testing. With minor adjustments to an agile development process, stakeholders spend more up-front time together.
• 
  Chameleon: A Hybrid, Proactive Auto-Scaling Mechanism on a Level-Playing Field Bauer, André; Herbst, Nikolas; Spinner, Simon; Ali-Eldin, Ahmed; Kounev, Samuel; in IEEE Transactions on Parallel and Distributed Systems (2019). 30(4) 800–813.
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @article{BaHeSpAEKo2019-TPDS-ChameleonAutoscaler, author = {Bauer, Andr{\'e} and Herbst, Nikolas and Spinner, Simon and Ali-Eldin, Ahmed and Kounev, Samuel}, journal = {IEEE Transactions on Parallel and Distributed Systems}, keywords = {myown}, month = {April}, number = 4, pages = {800–813}, publisher = {IEEE}, title = {{Chameleon: A Hybrid, Proactive Auto-Scaling Mechanism on a Level-Playing Field}}, volume = 30, year = 2019 }

  %0 Journal Article %1 BaHeSpAEKo2019-TPDS-ChameleonAutoscaler %A Bauer, André %A Herbst, Nikolas %A Spinner, Simon %A Ali-Eldin, Ahmed %A Kounev, Samuel %D 2019 %I IEEE %J IEEE Transactions on Parallel and Distributed Systems %N 4 %P 800--813 %T Chameleon: A Hybrid, Proactive Auto-Scaling Mechanism on a Level-Playing Field %V 30
• 
  The SPEC-RG Reference Architecture for FaaS: From Microservices and Containers to Serverless Platforms van Eyk, Erwin; Grohmann, Johannes; Eismann, Simon; Bauer, André; Versluis, Laurens; Toader, Lucian; Schmitt, Norbert; Herbst, Nikolas; Abad, Cristina L.; Iosup, Alexandru; in IEEE Internet Computing (2019). 23(6) 7–18.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  • [ Download ]
  Microservices, containers, and serverless computing belong to a trend toward applications composed of many small, self-contained, and automatically managed components. Core to serverless computing, Function-as-a-Service (FaaS) platforms employ state-of-the-art container technology and microservices-based architectures to enable users to manage complex applications without the need for systems-level expertise. Victim of its own success, and partially due to proprietary technology, currently the community has a limited overview of these platforms. To address this, we propose a reference architecture and ecosystem for FaaS platforms. Based on a year-long survey of real-world platforms conducted within the SPEC-RG Cloud Group, we highlight specific components and identify common operational patterns.

  @article{vEGrEiBaVeToScHeAbIo-IC-FaaS, abstract = {Microservices, containers, and serverless computing belong to a trend toward applications composed of many small, self-contained, and automatically managed components. Core to serverless computing, Function-as-a-Service (FaaS) platforms employ state-of-the-art container technology and microservices-based architectures to enable users to manage complex applications without the need for systems-level expertise. Victim of its own success, and partially due to proprietary technology, currently the community has a limited overview of these platforms. To address this, we propose a reference architecture and ecosystem for FaaS platforms. Based on a year-long survey of real-world platforms conducted within the SPEC-RG Cloud Group, we highlight specific components and identify common operational patterns.}, author = {van Eyk, Erwin and Grohmann, Johannes and Eismann, Simon and Bauer, Andr{\'e} and Versluis, Laurens and Toader, Lucian and Schmitt, Norbert and Herbst, Nikolas and Abad, Cristina L. and Iosup, Alexandru}, journal = {IEEE Internet Computing}, keywords = {myown}, month = {nov}, number = 6, pages = {7–18}, publisher = {IEEE}, title = {The SPEC-RG Reference Architecture for FaaS: From Microservices and Containers to Serverless Platforms}, volume = 23, year = 2019 }

  %0 Journal Article %1 vEGrEiBaVeToScHeAbIo-IC-FaaS %A van Eyk, Erwin %A Grohmann, Johannes %A Eismann, Simon %A Bauer, André %A Versluis, Laurens %A Toader, Lucian %A Schmitt, Norbert %A Herbst, Nikolas %A Abad, Cristina L. %A Iosup, Alexandru %D 2019 %I IEEE %J IEEE Internet Computing %N 6 %P 7--18 %R 10.1109/MIC.2019.2952061 %T The SPEC-RG Reference Architecture for FaaS: From Microservices and Containers to Serverless Platforms %U https://doi.org/10.1109/MIC.2019.2952061 %V 23 %X Microservices, containers, and serverless computing belong to a trend toward applications composed of many small, self-contained, and automatically managed components. Core to serverless computing, Function-as-a-Service (FaaS) platforms employ state-of-the-art container technology and microservices-based architectures to enable users to manage complex applications without the need for systems-level expertise. Victim of its own success, and partially due to proprietary technology, currently the community has a limited overview of these platforms. To address this, we propose a reference architecture and ecosystem for FaaS platforms. Based on a year-long survey of real-world platforms conducted within the SPEC-RG Cloud Group, we highlight specific components and identify common operational patterns.
• 
  Methodological Principles for Reproducible Performance Evaluation in Cloud Computing Papadopoulos, Alessandro Vittorio; Versluis, Laurens; Bauer, André; Herbst, Nikolas; von Kistowski, Jóakim; Ali-Eldin, Ahmed; Abad, Cristina; Amaral, J. Nelson; Tuma, Petr; Iosup, Alexandru; in IEEE Transactions on Software Engineering (2019). 47(8) 1528–1543.
  • [ BibTeX ]
  • [ EndNote ]
  • [ DOI ]
  • [ Download ]
  @article{PaVeBaHeKiAEAbAmTuIo2019-TSE-MethodologicalPrinciples, author = {Papadopoulos, Alessandro Vittorio and Versluis, Laurens and Bauer, Andr{\'e} and Herbst, Nikolas and von Kistowski, J{\'o}akim and Ali-Eldin, Ahmed and Abad, Cristina and Amaral, J. Nelson and Tuma, Petr and Iosup, Alexandru}, journal = {IEEE Transactions on Software Engineering}, keywords = {myown}, month = 7, number = 8, pages = {1528-1543}, title = {Methodological Principles for Reproducible Performance Evaluation in Cloud Computing}, volume = 47, year = 2019 }

  %0 Journal Article %1 PaVeBaHeKiAEAbAmTuIo2019-TSE-MethodologicalPrinciples %A Papadopoulos, Alessandro Vittorio %A Versluis, Laurens %A Bauer, André %A Herbst, Nikolas %A von Kistowski, Jóakim %A Ali-Eldin, Ahmed %A Abad, Cristina %A Amaral, J. Nelson %A Tuma, Petr %A Iosup, Alexandru %D 2019 %J IEEE Transactions on Software Engineering %N 8 %P 1528-1543 %R 10.1109/TSE.2019.2927908 %T Methodological Principles for Reproducible Performance Evaluation in Cloud Computing %V 47
• 
  Quantifying Cloud Performance and Dependability: Taxonomy, Metric Design, and Emerging Challenges Herbst, Nikolas; Bauer, André; Kounev, Samuel; Oikonomou, Giorgos; van Eyk, Erwin; Kousiouris, George; Evangelinou, Athanasia; Krebs, Rouven; Brecht, Tim; Abad, Cristina L.; Iosup, Alexandru; in ACM Transactions on Modeling and Performance Evaluation of Computing Systems (ToMPECS) (2018). 3(4) 19:1–19:36.
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  @article{HeBaKoOiEyKoEvKrBrAbIo2018-TOMPECS-CloudMetrics, address = {New York, NY, USA}, author = {Herbst, Nikolas and Bauer, Andr{\'e} and Kounev, Samuel and Oikonomou, Giorgos and van Eyk, Erwin and Kousiouris, George and Evangelinou, Athanasia and Krebs, Rouven and Brecht, Tim and Abad, Cristina L. and Iosup, Alexandru}, journal = {ACM Transactions on Modeling and Performance Evaluation of Computing Systems (ToMPECS)}, keywords = {myown}, month = 8, number = 4, pages = {19:1–19:36}, publisher = {ACM}, title = {{Quantifying Cloud Performance and Dependability: Taxonomy, Metric Design, and Emerging Challenges}}, volume = 3, year = 2018 }

  %0 Journal Article %1 HeBaKoOiEyKoEvKrBrAbIo2018-TOMPECS-CloudMetrics %A Herbst, Nikolas %A Bauer, André %A Kounev, Samuel %A Oikonomou, Giorgos %A van Eyk, Erwin %A Kousiouris, George %A Evangelinou, Athanasia %A Krebs, Rouven %A Brecht, Tim %A Abad, Cristina L. %A Iosup, Alexandru %C New York, NY, USA %D 2018 %I ACM %J ACM Transactions on Modeling and Performance Evaluation of Computing Systems (ToMPECS) %N 4 %P 19:1--19:36 %T Quantifying Cloud Performance and Dependability: Taxonomy, Metric Design, and Emerging Challenges %U http://doi.acm.org/10.1145/3236332 %V 3
• 
  An Experimental Performance Evaluation of Autoscalers for Complex Workflows Ilyushkin, Alexey; Ali-Eldin, Ahmed; Herbst, Nikolas; Bauer, André; Papadopoulos, Alessandro V.; Epema, Dick; Iosup, Alexandru; in ACM Transactions on Modeling and Performance Evaluation of Computing Systems (ToMPECS) (2018). 3(2) 8:1–8:32.
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  @article{IlAEHeBaPaEpIo2018-TOMPECS-Workflow, address = {New York, NY, USA}, author = {Ilyushkin, Alexey and Ali-Eldin, Ahmed and Herbst, Nikolas and Bauer, Andr{\'e} and Papadopoulos, Alessandro V. and Epema, Dick and Iosup, Alexandru}, journal = {ACM Transactions on Modeling and Performance Evaluation of Computing Systems (ToMPECS)}, keywords = {myown}, month = {April}, number = 2, pages = {8:1–8:32}, publisher = {ACM}, title = {An Experimental Performance Evaluation of Autoscalers for Complex Workflows}, volume = 3, year = 2018 }

  %0 Journal Article %1 IlAEHeBaPaEpIo2018-TOMPECS-Workflow %A Ilyushkin, Alexey %A Ali-Eldin, Ahmed %A Herbst, Nikolas %A Bauer, André %A Papadopoulos, Alessandro V. %A Epema, Dick %A Iosup, Alexandru %C New York, NY, USA %D 2018 %I ACM %J ACM Transactions on Modeling and Performance Evaluation of Computing Systems (ToMPECS) %N 2 %P 8:1--8:32 %T An Experimental Performance Evaluation of Autoscalers for Complex Workflows %U http://doi.acm.org/10.1145/3164537 %V 3
• 
  Modeling and Extracting Load Intensity Profiles von Kistowski, Jóakim; Herbst, Nikolas; Kounev, Samuel; Groenda, Henning; Stier, Christian; Lehrig, Sebastian; in ACM Transactions on Autonomous and Adaptive Systems (TAAS) (2017). 11(4) 23:1–23:28.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Today's system developers and operators face the challenge of creating software systems that make efficient use of dynamically allocated resources under highly variable and dynamic load profiles, while at the same time delivering reliable performance. Autonomic controllers, e.g., an advanced auto-scaling mechanism in a cloud computing context, can benefit from an abstracted load model as knowledge to reconfigure on time and precisely. Existing workload characterization approaches have limited support to capture variations the inter-arrival times of incoming work units over time (i.e., a variable load profile). For example, industrial and scientific benchmarks support constant or stepwise increasing load, or inter-arrival times defined by statistical distributions or recorded traces. These options show shortcomings either in representative character of load variation patterns or in abstraction and flexibility of their format. In this article, we present the Descartes Load Intensity Model (DLIM) approach addressing these issues. DLIM provides a modeling formalism for describing load intensity variations over time. A DLIM instance is a compact formal description of a load intensity trace. DLIM-based tools provide features for benchmarking, performance and recorded load intensity trace analysis. As manually obtaining and maintaining DLIM instances becomes time consuming, we contribute three automated extraction methods and devised metrics for comparison and method selection. We discuss how these features are used to enhance system management approaches for adaptations during run-time, and how they are integrated into simulation contexts and enable benchmarking of elastic or adaptive behavior. We show that automatically extracted DLIM instances exhibit an average modeling error of 15.2% over ten different real-world traces that cover between two weeks and seven months. These results underline DLIM model expressiveness. In terms of accuracy and processing speed, our proposed extraction methods for the descriptive models are comparable to existing time series decomposition methods. Additionally, we illustrate DLIM applicability by outlining approaches of workload modeling in systems engineering that employ or rely on our proposed load intensity modeling formalism.

  @article{KiHeKo-TAAS17-ModelExtractLoadProfiles, abstract = {Today's system developers and operators face the challenge of creating software systems that make efficient use of dynamically allocated resources under highly variable and dynamic load profiles, while at the same time delivering reliable performance. Autonomic controllers, e.g., an advanced auto-scaling mechanism in a cloud computing context, can benefit from an abstracted load model as knowledge to reconfigure on time and precisely. Existing workload characterization approaches have limited support to capture variations the inter-arrival times of incoming work units over time (i.e., a variable load profile). For example, industrial and scientific benchmarks support constant or stepwise increasing load, or inter-arrival times defined by statistical distributions or recorded traces. These options show shortcomings either in representative character of load variation patterns or in abstraction and flexibility of their format. In this article, we present the Descartes Load Intensity Model (DLIM) approach addressing these issues. DLIM provides a modeling formalism for describing load intensity variations over time. A DLIM instance is a compact formal description of a load intensity trace. DLIM-based tools provide features for benchmarking, performance and recorded load intensity trace analysis. As manually obtaining and maintaining DLIM instances becomes time consuming, we contribute three automated extraction methods and devised metrics for comparison and method selection. We discuss how these features are used to enhance system management approaches for adaptations during run-time, and how they are integrated into simulation contexts and enable benchmarking of elastic or adaptive behavior. We show that automatically extracted DLIM instances exhibit an average modeling error of 15.2\% over ten different real-world traces that cover between two weeks and seven months. These results underline DLIM model expressiveness. In terms of accuracy and processing speed, our proposed extraction methods for the descriptive models are comparable to existing time series decomposition methods. Additionally, we illustrate DLIM applicability by outlining approaches of workload modeling in systems engineering that employ or rely on our proposed load intensity modeling formalism.}, address = {New York, NY, USA}, author = {von Kistowski, J{\'o}akim and Herbst, Nikolas and Kounev, Samuel and Groenda, Henning and Stier, Christian and Lehrig, Sebastian}, journal = {ACM Transactions on Autonomous and Adaptive Systems (TAAS)}, keywords = {myown}, month = {January}, number = 4, pages = {23:1–23:28}, publisher = {ACM}, title = {{Modeling and Extracting Load Intensity Profiles}}, volume = 11, year = 2017 }

  %0 Journal Article %1 KiHeKo-TAAS17-ModelExtractLoadProfiles %A von Kistowski, Jóakim %A Herbst, Nikolas %A Kounev, Samuel %A Groenda, Henning %A Stier, Christian %A Lehrig, Sebastian %C New York, NY, USA %D 2017 %I ACM %J ACM Transactions on Autonomous and Adaptive Systems (TAAS) %N 4 %P 23:1--23:28 %T Modeling and Extracting Load Intensity Profiles %U http://doi.acm.org/10.1145/3019596 %V 11 %X Today's system developers and operators face the challenge of creating software systems that make efficient use of dynamically allocated resources under highly variable and dynamic load profiles, while at the same time delivering reliable performance. Autonomic controllers, e.g., an advanced auto-scaling mechanism in a cloud computing context, can benefit from an abstracted load model as knowledge to reconfigure on time and precisely. Existing workload characterization approaches have limited support to capture variations the inter-arrival times of incoming work units over time (i.e., a variable load profile). For example, industrial and scientific benchmarks support constant or stepwise increasing load, or inter-arrival times defined by statistical distributions or recorded traces. These options show shortcomings either in representative character of load variation patterns or in abstraction and flexibility of their format. In this article, we present the Descartes Load Intensity Model (DLIM) approach addressing these issues. DLIM provides a modeling formalism for describing load intensity variations over time. A DLIM instance is a compact formal description of a load intensity trace. DLIM-based tools provide features for benchmarking, performance and recorded load intensity trace analysis. As manually obtaining and maintaining DLIM instances becomes time consuming, we contribute three automated extraction methods and devised metrics for comparison and method selection. We discuss how these features are used to enhance system management approaches for adaptations during run-time, and how they are integrated into simulation contexts and enable benchmarking of elastic or adaptive behavior. We show that automatically extracted DLIM instances exhibit an average modeling error of 15.2% over ten different real-world traces that cover between two weeks and seven months. These results underline DLIM model expressiveness. In terms of accuracy and processing speed, our proposed extraction methods for the descriptive models are comparable to existing time series decomposition methods. Additionally, we illustrate DLIM applicability by outlining approaches of workload modeling in systems engineering that employ or rely on our proposed load intensity modeling formalism.

• 
  Online Workload Forecasting Herbst, Nikolas; Amin, Ayman; Andrzejak, Artur; Grunske, Lars; Kounev, Samuel; Mengshoel, Ole J.; Sundararajan, Priya; in Self-Aware Computing Systems, S. Kounev, J. O. Kephart, X. Zhu, A. Milenkoski (eds.) (2017).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  This chapter gives a summary of the state-of-the-art approaches from different research fields that can be applied to continuously forecast future developments of time series data streams. More specifically, the input time series data contains continuously monitored metrics that quantify the amount of incoming workload units to a self-aware system. It is the goal of this chapter to identify and present approaches for online workload forecasting that are required for a self-aware system to act proactively in terms of problem prevention and optimization inferred from likely changes in their usage. The research fields covered are machine learning and time series analysis. We describe explicit limitations and advantages for each forecasting method.

  @incollection{HeAmAnGrMeSuKo2016-self, abstract = {{This chapter gives a summary of the state-of-the-art approaches from different research fields that can be applied to continuously forecast future developments of time series data streams. More specifically, the input time series data contains continuously monitored metrics that quantify the amount of incoming workload units to a self-aware system. It is the goal of this chapter to identify and present approaches for online workload forecasting that are required for a self-aware system to act proactively in terms of problem prevention and optimization inferred from likely changes in their usage. The research fields covered are machine learning and time series analysis. We describe explicit limitations and advantages for each forecasting method.}}, address = {{Berlin Heidelberg, Germany}}, author = {Herbst, Nikolas and Amin, Ayman and Andrzejak, Artur and Grunske, Lars and Kounev, Samuel and Mengshoel, Ole J. and Sundararajan, Priya}, booktitle = {{Self-Aware Computing Systems}}, editor = {Kounev, Samuel and Kephart, Jeffrey O. and Zhu, Xiaoyun and Milenkoski, Aleksandar}, keywords = {myown}, publisher = {{Springer Verlag}}, title = {{Online Workload Forecasting}}, year = 2017 }

  %0 Book Section %1 HeAmAnGrMeSuKo2016-self %A Herbst, Nikolas %A Amin, Ayman %A Andrzejak, Artur %A Grunske, Lars %A Kounev, Samuel %A Mengshoel, Ole J. %A Sundararajan, Priya %B Self-Aware Computing Systems %C Berlin Heidelberg, Germany %D 2017 %E Kounev, Samuel %E Kephart, Jeffrey O. %E Zhu, Xiaoyun %E Milenkoski, Aleksandar %I Springer Verlag %T Online Workload Forecasting %X This chapter gives a summary of the state-of-the-art approaches from different research fields that can be applied to continuously forecast future developments of time series data streams. More specifically, the input time series data contains continuously monitored metrics that quantify the amount of incoming workload units to a self-aware system. It is the goal of this chapter to identify and present approaches for online workload forecasting that are required for a self-aware system to act proactively in terms of problem prevention and optimization inferred from likely changes in their usage. The research fields covered are machine learning and time series analysis. We describe explicit limitations and advantages for each forecasting method.
• 
  Metrics and Benchmarks for Self-Aware Computing Systems Herbst, Nikolas; Becker, Steffen; Kounev, Samuel; Koziolek, Heiko; Maggio, Martina; Milenkoski, Aleksandar; Smirni, Evgenia; in Self-Aware Computing Systems, S. Kounev, J. O. Kephart, A. Milenkoski, X. Zhu (eds.) (2017).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  In this chapter, we propose a list of metrics grouped by the MAPE-K paradigm for quantifying properties of self-aware computing systems. This set of metrics can be seen as a starting point toward benchmarking and comparing self-aware computing systems on a level-playing field. We discuss state-of-the art approaches in the related fields of self-adaptation and self-protection to identify commonalities in metrics for self-aware computing. We illustrate the need for benchmarking self-aware computing systems with the help of an approach that uncovers real-time characteristics of operating systems. Gained insights of this approach can be seen as a way of enhancing self-awareness by a measurement methodology on an ongoing basis. At the end of this chapter, we address new challenges in reference workload definition for benchmarking self-aware computing systems, namely load intensity patterns and burstiness modeling.

  @incollection{HeBeKoKoMaMiSm2017-self, abstract = {{In this chapter, we propose a list of metrics grouped by the MAPE-K paradigm for quantifying properties of self-aware computing systems. This set of metrics can be seen as a starting point toward benchmarking and comparing self-aware computing systems on a level-playing field. We discuss state-of-the art approaches in the related fields of self-adaptation and self-protection to identify commonalities in metrics for self-aware computing. We illustrate the need for benchmarking self-aware computing systems with the help of an approach that uncovers real-time characteristics of operating systems. Gained insights of this approach can be seen as a way of enhancing self-awareness by a measurement methodology on an ongoing basis. At the end of this chapter, we address new challenges in reference workload definition for benchmarking self-aware computing systems, namely load intensity patterns and burstiness modeling.}}, address = {{Berlin Heidelberg, Germany}}, author = {Herbst, Nikolas and Becker, Steffen and Kounev, Samuel and Koziolek, Heiko and Maggio, Martina and Milenkoski, Aleksandar and Smirni, Evgenia}, booktitle = {{Self-Aware Computing Systems}}, editor = {Kounev, Samuel and Kephart, Jeffrey O. and Milenkoski, Aleksandar and Zhu, Xiaoyun}, keywords = {myown}, publisher = {{Springer Verlag}}, title = {{Metrics and Benchmarks for Self-Aware Computing Systems}}, year = 2017 }

  %0 Book Section %1 HeBeKoKoMaMiSm2017-self %A Herbst, Nikolas %A Becker, Steffen %A Kounev, Samuel %A Koziolek, Heiko %A Maggio, Martina %A Milenkoski, Aleksandar %A Smirni, Evgenia %B Self-Aware Computing Systems %C Berlin Heidelberg, Germany %D 2017 %E Kounev, Samuel %E Kephart, Jeffrey O. %E Milenkoski, Aleksandar %E Zhu, Xiaoyun %I Springer Verlag %T Metrics and Benchmarks for Self-Aware Computing Systems %U https://link.springer.com/chapter/10.1007%2F978-3-319-47474-8_14 %X In this chapter, we propose a list of metrics grouped by the MAPE-K paradigm for quantifying properties of self-aware computing systems. This set of metrics can be seen as a starting point toward benchmarking and comparing self-aware computing systems on a level-playing field. We discuss state-of-the art approaches in the related fields of self-adaptation and self-protection to identify commonalities in metrics for self-aware computing. We illustrate the need for benchmarking self-aware computing systems with the help of an approach that uncovers real-time characteristics of operating systems. Gained insights of this approach can be seen as a way of enhancing self-awareness by a measurement methodology on an ongoing basis. At the end of this chapter, we address new challenges in reference workload definition for benchmarking self-aware computing systems, namely load intensity patterns and burstiness modeling.

• 
  A Trace-driven Performance Evaluation of Hash-based Task Placement Algorithms for Cache-enabled Serverless Computing Talluri, Sacheendra; Herbst, Nikolas; Abad, Cristina; Trivedi, Animesh; Iosup, Alexandru; in 2023 ACM International Conference on Computing Frontiers (CF ’23) (2023).
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @inproceedings{2023-cf-hash-task-placement, author = {Talluri, Sacheendra and Herbst, Nikolas and Abad, Cristina and Trivedi, Animesh and Iosup, Alexandru}, booktitle = {2023 ACM International Conference on Computing Frontiers (CF '23)}, keywords = {myown}, title = {A Trace-driven Performance Evaluation of Hash-based Task Placement Algorithms for Cache-enabled Serverless Computing}, year = 2023 }

  %0 Conference Paper %1 2023-cf-hash-task-placement %A Talluri, Sacheendra %A Herbst, Nikolas %A Abad, Cristina %A Trivedi, Animesh %A Iosup, Alexandru %B 2023 ACM International Conference on Computing Frontiers (CF '23) %D 2023 %T A Trace-driven Performance Evaluation of Hash-based Task Placement Algorithms for Cache-enabled Serverless Computing
• 
  An Empirical Study of Container Image Configurations and Their Impact on Start Times Straesser, Martin; Bauer, André; Leppich, Robert; Herbst, Nikolas; Chard, Kyle; Foster, Ian; Kounev, Samuel; in 2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet Computing (CCGrid) (2023).

  In print.
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @inproceedings{straesser2023starttime, author = {Straesser, Martin and Bauer, André and Leppich, Robert and Herbst, Nikolas and Chard, Kyle and Foster, Ian and Kounev, Samuel}, booktitle = {2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet Computing (CCGrid)}, keywords = {myown}, note = {In print.}, publisher = {IEEE}, title = {An Empirical Study of Container Image Configurations and Their Impact on Start Times}, year = 2023 }

  %0 Conference Paper %1 straesser2023starttime %A Straesser, Martin %A Bauer, André %A Leppich, Robert %A Herbst, Nikolas %A Chard, Kyle %A Foster, Ian %A Kounev, Samuel %B 2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet Computing (CCGrid) %D 2023 %I IEEE %T An Empirical Study of Container Image Configurations and Their Impact on Start Times
• 
  SuanMing: Explainable Prediction of Performance Degradations in Microservice Applications Grohmann, Johannes; Straesser, Martin; Chalbani, Avi; Eismann, Simon; Arian, Yair; Herbst, Nikolas; Peretz, Noam; Kounev, Samuel; in Proceedings of the 12th ACM/SPEC International Conference on Performance Engineering (ICPE) (2021).

  Acceptance Rate: 29%
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  • [ Download ]
  Application performance management (APM) tools are useful to observe the performance properties of an application during production. However, APM is normally purely reactive, that is, it can only report about current or past performance degradation. Although some approaches capable of predictive application monitoring have been proposed, they can only report a predicted degradation but cannot explain its root-cause, making it hard to prevent the expected degradation. In this paper, we present SuanMing---a framework for predicting performance degradation of microservice applications running in cloud environments. SuanMing is able to predict future root causes for anticipated performance degradations and therefore aims at preventing performance degradations before they actually occur. We evaluate SuanMing on two realistic microservice applications, TeaStore and TrainTicket, and we show that our approach is able to predict and pinpoint performance degradations with an accuracy of over 90%.

  @inproceedings{GrStChEiArHePeKo2021-ICPE, abstract = {Application performance management (APM) tools are useful to observe the performance properties of an application during production. However, APM is normally purely reactive, that is, it can only report about current or past performance degradation. Although some approaches capable of predictive application monitoring have been proposed, they can only report a predicted degradation but cannot explain its root-cause, making it hard to prevent the expected degradation. In this paper, we present SuanMing—a framework for predicting performance degradation of microservice applications running in cloud environments. SuanMing is able to predict future root causes for anticipated performance degradations and therefore aims at preventing performance degradations before they actually occur. We evaluate SuanMing on two realistic microservice applications, TeaStore and TrainTicket, and we show that our approach is able to predict and pinpoint performance degradations with an accuracy of over 90\%.}, address = {New York, NY, USA}, author = {Grohmann, Johannes and Straesser, Martin and Chalbani, Avi and Eismann, Simon and Arian, Yair and Herbst, Nikolas and Peretz, Noam and Kounev, Samuel}, booktitle = {Proceedings of the 12th ACM/SPEC International Conference on Performance Engineering (ICPE)}, keywords = {myown}, month = {April}, note = {Acceptance Rate: 29%}, organization = {ACM}, title = {SuanMing: Explainable Prediction of Performance Degradations in Microservice Applications}, year = 2021 }

  %0 Conference Paper %1 GrStChEiArHePeKo2021-ICPE %A Grohmann, Johannes %A Straesser, Martin %A Chalbani, Avi %A Eismann, Simon %A Arian, Yair %A Herbst, Nikolas %A Peretz, Noam %A Kounev, Samuel %B Proceedings of the 12th ACM/SPEC International Conference on Performance Engineering (ICPE) %C New York, NY, USA %D 2021 %R https://doi.org/10.1145/3427921.3450248 %T SuanMing: Explainable Prediction of Performance Degradations in Microservice Applications %U https://doi.org/10.1145/3427921.3450248 %X Application performance management (APM) tools are useful to observe the performance properties of an application during production. However, APM is normally purely reactive, that is, it can only report about current or past performance degradation. Although some approaches capable of predictive application monitoring have been proposed, they can only report a predicted degradation but cannot explain its root-cause, making it hard to prevent the expected degradation. In this paper, we present SuanMing---a framework for predicting performance degradation of microservice applications running in cloud environments. SuanMing is able to predict future root causes for anticipated performance degradations and therefore aims at preventing performance degradations before they actually occur. We evaluate SuanMing on two realistic microservice applications, TeaStore and TrainTicket, and we show that our approach is able to predict and pinpoint performance degradations with an accuracy of over 90%.
• 
  Libra: A Benchmark for Time Series Forecasting Methods Bauer, André; Züfle, Marwin; Eismann, Simon; Grohmann, Johannes; Herbst, Nikolas; Kounev, Samuel; in Proceedings of the 12th ACM/SPEC International Conference on Performance Engineering (ICPE) (2021).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  • [ Download ]
  In many areas of decision making, forecasting is an essential pillar. Consequently, there are many different forecasting methods. According to the "No-Free-Lunch Theorem", there is no single forecasting method that performs best for all time series. In other words, each method has its advantages and disadvantages depending on the specific use case. Therefore, the choice of the forecasting method remains a mandatory expert task. However, expert knowledge cannot be fully automated. To establish a level playing field for evaluating the performance of time series forecasting methods in a broad setting, we propose Libra, a forecasting benchmark that automatically evaluates and ranks forecasting methods based on their performance in a diverse set of evaluation scenarios. The benchmark comprises four different use cases, each covering 100 heterogeneous time series taken from different domains. The data set was assembled from publicly available time series and was designed to exhibit much higher diversity than existing forecasting competitions. Based on this benchmark, we perform a comprehensive evaluation to compare different existing time series forecasting methods.

  @inproceedings{bauer2021benchmark, abstract = {In many areas of decision making, forecasting is an essential pillar. Consequently, there are many different forecasting methods. According to the "No-Free-Lunch Theorem", there is no single forecasting method that performs best for all time series. In other words, each method has its advantages and disadvantages depending on the specific use case. Therefore, the choice of the forecasting method remains a mandatory expert task. However, expert knowledge cannot be fully automated. To establish a level playing field for evaluating the performance of time series forecasting methods in a broad setting, we propose Libra, a forecasting benchmark that automatically evaluates and ranks forecasting methods based on their performance in a diverse set of evaluation scenarios. The benchmark comprises four different use cases, each covering 100 heterogeneous time series taken from different domains. The data set was assembled from publicly available time series and was designed to exhibit much higher diversity than existing forecasting competitions. Based on this benchmark, we perform a comprehensive evaluation to compare different existing time series forecasting methods.}, address = {New York, NY, USA}, author = {Bauer, Andr{\'e} and Z{\"u}fle, Marwin and Eismann, Simon and Grohmann, Johannes and Herbst, Nikolas and Kounev, Samuel}, booktitle = {Proceedings of the 12th ACM/SPEC International Conference on Performance Engineering (ICPE)}, keywords = {myown}, month = {April}, organization = {ACM}, title = {Libra: A Benchmark for Time Series Forecasting Methods}, year = 2021 }

  %0 Conference Paper %1 bauer2021benchmark %A Bauer, André %A Züfle, Marwin %A Eismann, Simon %A Grohmann, Johannes %A Herbst, Nikolas %A Kounev, Samuel %B Proceedings of the 12th ACM/SPEC International Conference on Performance Engineering (ICPE) %C New York, NY, USA %D 2021 %R 10.1145/3427921.3450241 %T Libra: A Benchmark for Time Series Forecasting Methods %U https://dl.acm.org/doi/10.1145/3427921.3450241 %X In many areas of decision making, forecasting is an essential pillar. Consequently, there are many different forecasting methods. According to the "No-Free-Lunch Theorem", there is no single forecasting method that performs best for all time series. In other words, each method has its advantages and disadvantages depending on the specific use case. Therefore, the choice of the forecasting method remains a mandatory expert task. However, expert knowledge cannot be fully automated. To establish a level playing field for evaluating the performance of time series forecasting methods in a broad setting, we propose Libra, a forecasting benchmark that automatically evaluates and ranks forecasting methods based on their performance in a diverse set of evaluation scenarios. The benchmark comprises four different use cases, each covering 100 heterogeneous time series taken from different domains. The data set was assembled from publicly available time series and was designed to exhibit much higher diversity than existing forecasting competitions. Based on this benchmark, we perform a comprehensive evaluation to compare different existing time series forecasting methods.
• 
  Sizeless: Predicting the Optimal Size of Serverless Functions Eismann, Simon; Bui, Long; Grohmann, Johannes; Abad, Cristina; Herbst, Nikolas; Kounev, Samuel; in Proceedings of the 22nd International MIDDLEWARE Conference (2021). 248–259.

  Best Student Paper Award, ACM Artifacts Evaluated — Functional
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  • [ Download ]
  Serverless functions are an emerging cloud computing paradigm that is being rapidly adopted by both industry and academia. In this cloud computing model, the provider opaquely handles resource management tasks such as resource provisioning, deployment, and auto-scaling. The only resource management task that developers are still in charge of is selecting how much resources are allocated to each worker instance. However, selecting the optimal size of serverless functions is quite challenging, so developers often neglect it despite its significant cost and performance benefits. Existing approaches aiming to automate serverless functions resource sizing require dedicated performance tests, which are time-consuming to implement and maintain. In this paper, we introduce an approach to predict the optimal resource size of a serverless function using monitoring data from a single resource size. As our approach does not require dedicated performance tests, it enables cloud providers to implement resource sizing on a platform level and automate the last resource management task associated with serverless functions. We evaluate our approach on four different serverless applications on AWS, where it predicts the execution time of the other memory sizes based on monitoring data for a single memory size with an average prediction error of 15.3%. Based on these predictions, it selects the optimal memory size for 79.0% of the serverless functions and the secondbest memory size for 12.3% of the serverless functions, which results in an average speedup of 39.7% while also decreasing average costs by 2.6%.

  @inproceedings{eismann2021sizeless, abstract = {Serverless functions are an emerging cloud computing paradigm that is being rapidly adopted by both industry and academia. In this cloud computing model, the provider opaquely handles resource management tasks such as resource provisioning, deployment, and auto-scaling. The only resource management task that developers are still in charge of is selecting how much resources are allocated to each worker instance. However, selecting the optimal size of serverless functions is quite challenging, so developers often neglect it despite its significant cost and performance benefits. Existing approaches aiming to automate serverless functions resource sizing require dedicated performance tests, which are time-consuming to implement and maintain. In this paper, we introduce an approach to predict the optimal resource size of a serverless function using monitoring data from a single resource size. As our approach does not require dedicated performance tests, it enables cloud providers to implement resource sizing on a platform level and automate the last resource management task associated with serverless functions. We evaluate our approach on four different serverless applications on AWS, where it predicts the execution time of the other memory sizes based on monitoring data for a single memory size with an average prediction error of 15.3%. Based on these predictions, it selects the optimal memory size for 79.0% of the serverless functions and the secondbest memory size for 12.3% of the serverless functions, which results in an average speedup of 39.7% while also decreasing average costs by 2.6%.}, author = {Eismann, Simon and Bui, Long and Grohmann, Johannes and Abad, Cristina and Herbst, Nikolas and Kounev, Samuel}, booktitle = {Proceedings of the 22nd International MIDDLEWARE Conference}, keywords = {myown}, note = {Best Student Paper Award, ACM Artifacts Evaluated — Functional}, pages = {248–259}, title = {Sizeless: Predicting the Optimal Size of Serverless Functions}, year = 2021 }

  %0 Conference Paper %1 eismann2021sizeless %A Eismann, Simon %A Bui, Long %A Grohmann, Johannes %A Abad, Cristina %A Herbst, Nikolas %A Kounev, Samuel %B Proceedings of the 22nd International MIDDLEWARE Conference %D 2021 %P 248–259 %R 10.1145/3464298.3493398 %T Sizeless: Predicting the Optimal Size of Serverless Functions %U https://dl.acm.org/doi/abs/10.1145/3464298.3493398 %X Serverless functions are an emerging cloud computing paradigm that is being rapidly adopted by both industry and academia. In this cloud computing model, the provider opaquely handles resource management tasks such as resource provisioning, deployment, and auto-scaling. The only resource management task that developers are still in charge of is selecting how much resources are allocated to each worker instance. However, selecting the optimal size of serverless functions is quite challenging, so developers often neglect it despite its significant cost and performance benefits. Existing approaches aiming to automate serverless functions resource sizing require dedicated performance tests, which are time-consuming to implement and maintain. In this paper, we introduce an approach to predict the optimal resource size of a serverless function using monitoring data from a single resource size. As our approach does not require dedicated performance tests, it enables cloud providers to implement resource sizing on a platform level and automate the last resource management task associated with serverless functions. We evaluate our approach on four different serverless applications on AWS, where it predicts the execution time of the other memory sizes based on monitoring data for a single memory size with an average prediction error of 15.3%. Based on these predictions, it selects the optimal memory size for 79.0% of the serverless functions and the secondbest memory size for 12.3% of the serverless functions, which results in an average speedup of 39.7% while also decreasing average costs by 2.6%.
• 
  An Automated Forecasting Framework based on Method Recommendation for Seasonal Time Series Bauer, André; Züfle, Marwin; Grohmann, Johannes; Schmitt, Norbert; Herbst, Nikolas; Kounev, Samuel; in Proceedings of the ACM/SPEC International Conference on Performance Engineering (2020). 48–55.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  • [ Download ]
  Due to the fast-paced and changing demands of their users, computing systems require autonomic resource management. To enable proactive and accurate decision-making for changes causing a particular overhead, reliable forecasts are needed. In fact, choosing the best performing forecasting method for a given time series scenario is a crucial task. Taking the "No-Free-Lunch Theorem" into account, there exists no forecasting method that performs best on all types of time series. To this end, we propose an automated approach that (i) extracts characteristics from a given time series, (ii) selects the best-suited machine learning method based on recommendation, and finally, (iii) performs the forecast. Our approach offers the benefit of not relying on a single method with its possibly inaccurate forecasts. In an extensive evaluation, our approach achieves the best forecasting accuracy.

  @inproceedings{BaZuGrScHeKo-ICPE20-Seasonal-Forecast, abstract = {Due to the fast-paced and changing demands of their users, computing systems require autonomic resource management. To enable proactive and accurate decision-making for changes causing a particular overhead, reliable forecasts are needed. In fact, choosing the best performing forecasting method for a given time series scenario is a crucial task. Taking the "No-Free-Lunch Theorem" into account, there exists no forecasting method that performs best on all types of time series. To this end, we propose an automated approach that (i) extracts characteristics from a given time series, (ii) selects the best-suited machine learning method based on recommendation, and finally, (iii) performs the forecast. Our approach offers the benefit of not relying on a single method with its possibly inaccurate forecasts. In an extensive evaluation, our approach achieves the best forecasting accuracy.}, address = {New York, NY, USA}, author = {Bauer, Andr{\'e} and Z{\"u}fle, Marwin and Grohmann, Johannes and Schmitt, Norbert and Herbst, Nikolas and Kounev, Samuel}, booktitle = {Proceedings of the ACM/SPEC International Conference on Performance Engineering}, keywords = {myown}, month = {April}, pages = {48–55}, publisher = {Association for Computing Machinery (ACM)}, series = {ICPE '20}, title = {{An Automated Forecasting Framework based on Method Recommendation for Seasonal Time Series}}, year = 2020 }

  %0 Conference Paper %1 BaZuGrScHeKo-ICPE20-Seasonal-Forecast %A Bauer, André %A Züfle, Marwin %A Grohmann, Johannes %A Schmitt, Norbert %A Herbst, Nikolas %A Kounev, Samuel %B Proceedings of the ACM/SPEC International Conference on Performance Engineering %C New York, NY, USA %D 2020 %I Association for Computing Machinery (ACM) %P 48--55 %R 10.1145/3358960.3379123 %T An Automated Forecasting Framework based on Method Recommendation for Seasonal Time Series %U https://doi.org/10.1145/3358960.3379123 %X Due to the fast-paced and changing demands of their users, computing systems require autonomic resource management. To enable proactive and accurate decision-making for changes causing a particular overhead, reliable forecasts are needed. In fact, choosing the best performing forecasting method for a given time series scenario is a crucial task. Taking the "No-Free-Lunch Theorem" into account, there exists no forecasting method that performs best on all types of time series. To this end, we propose an automated approach that (i) extracts characteristics from a given time series, (ii) selects the best-suited machine learning method based on recommendation, and finally, (iii) performs the forecast. Our approach offers the benefit of not relying on a single method with its possibly inaccurate forecasts. In an extensive evaluation, our approach achieves the best forecasting accuracy. %@ 9781450369916
• 
  Predicting the Costs of Serverless Workflows Eismann, Simon; Grohmann, Johannes; van Eyk, Erwin; Herbst, Nikolas; Kounev, Samuel; in Proceedings of the 2020 ACM/SPEC International Conference on Performance Engineering (ICPE) (2020). 265–276.

  Acceptance Rate: 23.4% (15/64)
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  • [ Download ]
  Function-as-a-Service (FaaS) platforms enable users to run arbitrary functions without being concerned about operational issues, while only paying for the consumed resources. Individual functions are often composed into workflows for complex tasks. However, the pay-per-use model and nontransparent reporting by cloud providers make it challenging to estimate the expected cost of a workflow, which prevents informed business decisions. Existing cost-estimation approaches assume a static response time for the serverless functions, without taking input parameters into account. In this paper, we propose a methodology for the cost prediction of serverless workflows consisting of input-parameter sensitive function models and a monte-carlo simulation of an abstract workflow model. Our approach enables workflow designers to predict, compare, and optimize the expected costs and performance of a planned workflow, which currently requires time-intensive experimentation. In our evaluation, we show that our approach can predict the response time and output parameters of a function based on its input parameters with an accuracy of 96.1%. In a case study with two audio-processing workflows, our approach predicts the costs of the two workflows with an accuracy of 96.2%.

  @inproceedings{EiGrEyHeKo2020-ICPE-ServerlessWorkflows, abstract = {Function-as-a-Service (FaaS) platforms enable users to run arbitrary functions without being concerned about operational issues, while only paying for the consumed resources. Individual functions are often composed into workflows for complex tasks. However, the pay-per-use model and nontransparent reporting by cloud providers make it challenging to estimate the expected cost of a workflow, which prevents informed business decisions. Existing cost-estimation approaches assume a static response time for the serverless functions, without taking input parameters into account. In this paper, we propose a methodology for the cost prediction of serverless workflows consisting of input-parameter sensitive function models and a monte-carlo simulation of an abstract workflow model. Our approach enables workflow designers to predict, compare, and optimize the expected costs and performance of a planned workflow, which currently requires time-intensive experimentation. In our evaluation, we show that our approach can predict the response time and output parameters of a function based on its input parameters with an accuracy of 96.1%. In a case study with two audio-processing workflows, our approach predicts the costs of the two workflows with an accuracy of 96.2%.}, address = {New York, NY, USA}, author = {Eismann, Simon and Grohmann, Johannes and van Eyk, Erwin and Herbst, Nikolas and Kounev, Samuel}, booktitle = {Proceedings of the 2020 ACM/SPEC International Conference on Performance Engineering (ICPE)}, keywords = {myown}, month = {April}, note = {{Acceptance Rate: 23.4% (15/64)}}, pages = {265–276}, publisher = {Association for Computing Machinery (ACM)}, series = {ICPE '20}, title = {Predicting the Costs of Serverless Workflows}, year = 2020 }

  %0 Conference Paper %1 EiGrEyHeKo2020-ICPE-ServerlessWorkflows %A Eismann, Simon %A Grohmann, Johannes %A van Eyk, Erwin %A Herbst, Nikolas %A Kounev, Samuel %B Proceedings of the 2020 ACM/SPEC International Conference on Performance Engineering (ICPE) %C New York, NY, USA %D 2020 %I Association for Computing Machinery (ACM) %P 265--276 %R 10.1145/3358960.3379133 %T Predicting the Costs of Serverless Workflows %U https://doi.org/10.1145/3358960.3379133 %X Function-as-a-Service (FaaS) platforms enable users to run arbitrary functions without being concerned about operational issues, while only paying for the consumed resources. Individual functions are often composed into workflows for complex tasks. However, the pay-per-use model and nontransparent reporting by cloud providers make it challenging to estimate the expected cost of a workflow, which prevents informed business decisions. Existing cost-estimation approaches assume a static response time for the serverless functions, without taking input parameters into account. In this paper, we propose a methodology for the cost prediction of serverless workflows consisting of input-parameter sensitive function models and a monte-carlo simulation of an abstract workflow model. Our approach enables workflow designers to predict, compare, and optimize the expected costs and performance of a planned workflow, which currently requires time-intensive experimentation. In our evaluation, we show that our approach can predict the response time and output parameters of a function based on its input parameters with an accuracy of 96.1%. In a case study with two audio-processing workflows, our approach predicts the costs of the two workflows with an accuracy of 96.2%. %@ 9781450369916
• 
  Telescope: An Automatic Feature Extraction and Transformation Approach for Time Series Forecasting on a Level-Playing Field Bauer, André; Züfle, Marwin; Herbst, Nikolas; Kounev, Samuel; Curtef, Valentin; in Proceedings of the 36th International Conference on Data Engineering (ICDE) (2020).
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @inproceedings{BaZuHeKoCu-ICDE-Telescope, author = {Bauer, Andr{\'e} and Z{\"u}fle, Marwin and Herbst, Nikolas and Kounev, Samuel and Curtef, Valentin}, booktitle = {Proceedings of the 36th International Conference on Data Engineering (ICDE)}, keywords = {myown}, month = {April}, title = {Telescope: An Automatic Feature Extraction and Transformation Approach for Time Series Forecasting on a Level-Playing Field}, year = 2020 }

  %0 Conference Paper %1 BaZuHeKoCu-ICDE-Telescope %A Bauer, André %A Züfle, Marwin %A Herbst, Nikolas %A Kounev, Samuel %A Curtef, Valentin %B Proceedings of the 36th International Conference on Data Engineering (ICDE) %D 2020 %T Telescope: An Automatic Feature Extraction and Transformation Approach for Time Series Forecasting on a Level-Playing Field
• 
  Elasticity of Cloud Platforms Herbst, Nikolas; Bauer, André; Kounev, Samuel; in Systems Benchmarking: For Scientists and Engineers, S. Kounev, K.-D. Lange, J. von Kistowski (eds.) (2020). 319–340.
  • [ BibTeX ]
  • [ EndNote ]
  @inproceedings{bauer2020bookelasticity, author = {Herbst, Nikolas and Bauer, Andr{\'e} and Kounev, Samuel}, booktitle = {{Systems Benchmarking: For Scientists and Engineers}}, editor = {Kounev, Samuel and Lange, Klaus-Dieter and von Kistowski, J{\'o}akim}, keywords = {myown}, month = 1, organization = {Springer}, pages = {319-340}, title = {{Elasticity of Cloud Platforms}}, year = 2020 }

  %0 Conference Paper %1 bauer2020bookelasticity %A Herbst, Nikolas %A Bauer, André %A Kounev, Samuel %B Systems Benchmarking: For Scientists and Engineers %D 2020 %E Kounev, Samuel %E Lange, Klaus-Dieter %E von Kistowski, Jóakim %P 319-340 %T Elasticity of Cloud Platforms
• 
  Methodological Principles for Reproducible Performance Evaluation in Cloud Computing Papadopoulos, Alessandro Vittorio; Versluis, Laurens; Bauer, André; Herbst, Nikolas; von Kistowski, Jóakim; Ali-Eldin, Ahmed; Abad, Cristina L.; Amaral, José Nelson; Tuma, Petr; Iosup, Alexandru; in Software Engineering 2020, Fachtagung des GI-Fachbereichs Softwaretechnik, 24.-28. Februar 2020, Innsbruck, Austria, M. Felderer, W. Hasselbring, R. Rabiser, R. Jung (eds.) (2020). (Vol. P-300) 93–94.
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  @inproceedings{DBLP:conf/se/PapadopoulosVBH20, author = {Papadopoulos, Alessandro Vittorio and Versluis, Laurens and Bauer, Andr{{\'e}} and Herbst, Nikolas and von Kistowski, J{{\'o}}akim and Ali{-}Eldin, Ahmed and Abad, Cristina L. and Amaral, Jos{{\'e}} Nelson and Tuma, Petr and Iosup, Alexandru}, booktitle = {Software Engineering 2020, Fachtagung des GI-Fachbereichs Softwaretechnik, 24.-28. Februar 2020, Innsbruck, Austria}, editor = {Felderer, Michael and Hasselbring, Wilhelm and Rabiser, Rick and Jung, Reiner}, keywords = {myown}, pages = {93–94}, publisher = {Gesellschaft f{{\"u}}r Informatik e.V.}, series = {{LNI}}, title = {{Methodological Principles for Reproducible Performance Evaluation in Cloud Computing}}, volume = {{P-300}}, year = 2020 }

  %0 Conference Paper %1 DBLP:conf/se/PapadopoulosVBH20 %A Papadopoulos, Alessandro Vittorio %A Versluis, Laurens %A Bauer, André %A Herbst, Nikolas %A von Kistowski, Jóakim %A Ali-Eldin, Ahmed %A Abad, Cristina L. %A Amaral, José Nelson %A Tuma, Petr %A Iosup, Alexandru %B Software Engineering 2020, Fachtagung des GI-Fachbereichs Softwaretechnik, 24.-28. Februar 2020, Innsbruck, Austria %D 2020 %E Felderer, Michael %E Hasselbring, Wilhelm %E Rabiser, Rick %E Jung, Reiner %I Gesellschaft für Informatik e.V. %P 93--94 %T Methodological Principles for Reproducible Performance Evaluation in Cloud Computing %U https://doi.org/10.18420/SE2020_27 %V P-300
• 
  Systematic Search for Optimal Resource Configurations of Distributed Applications Bauer, André; Eismann, Simon; Grohmann, Johannes; Herbst, Nikolas; Kounev, Samuel; in 2019 IEEE 4th International Workshops on Foundations and Applications of Self* Systems (FAS*W) (2019). 120–125.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  • [ Download ]
  With the advent of the micro-service paradigm, applications are divided into small, distributed parts. Knowledge of optimal resource configurations of such applications is required both for autonomic resource management as well as its assessment. Due to the high-dimensional search space of all possible configurations, the systematic measuring of the optimal configurations is challenging. To this end, we introduce a search algorithm based on hill-climbing for finding all optimal configurations in a feasible time and integrate it in an existing measuring framework. This approach enables the assessment, comparison and optimization of autonomic resource management approaches for micro-service applications. The evaluation shows that our approach is able to find all optimal configurations in the considered scenarios, while state-of-the-art multi-objective search algorithms do not.

  @inproceedings{BaEiGrHeKo-ICAC-BUNGEE, abstract = {With the advent of the micro-service paradigm, applications are divided into small, distributed parts. Knowledge of optimal resource configurations of such applications is required both for autonomic resource management as well as its assessment. Due to the high-dimensional search space of all possible configurations, the systematic measuring of the optimal configurations is challenging. To this end, we introduce a search algorithm based on hill-climbing for finding all optimal configurations in a feasible time and integrate it in an existing measuring framework. This approach enables the assessment, comparison and optimization of autonomic resource management approaches for micro-service applications. The evaluation shows that our approach is able to find all optimal configurations in the considered scenarios, while state-of-the-art multi-objective search algorithms do not.}, address = {Los Alamitos, CA, USA}, author = {Bauer, Andr{\'e} and Eismann, Simon and Grohmann, Johannes and Herbst, Nikolas and Kounev, Samuel}, booktitle = {2019 IEEE 4th International Workshops on Foundations and Applications of Self* Systems (FAS*W)}, keywords = {myown}, month = {jun}, pages = {120–125}, publisher = {IEEE Computer Society}, title = {{Systematic Search for Optimal Resource Configurations of Distributed Applications}}, year = 2019 }

  %0 Conference Paper %1 BaEiGrHeKo-ICAC-BUNGEE %A Bauer, André %A Eismann, Simon %A Grohmann, Johannes %A Herbst, Nikolas %A Kounev, Samuel %B 2019 IEEE 4th International Workshops on Foundations and Applications of Self* Systems (FAS*W) %C Los Alamitos, CA, USA %D 2019 %I IEEE Computer Society %P 120--125 %R 10.1109/FAS-W.2019.00040 %T Systematic Search for Optimal Resource Configurations of Distributed Applications %U https://doi.ieeecomputersociety.org/10.1109/FAS-W.2019.00040 %X With the advent of the micro-service paradigm, applications are divided into small, distributed parts. Knowledge of optimal resource configurations of such applications is required both for autonomic resource management as well as its assessment. Due to the high-dimensional search space of all possible configurations, the systematic measuring of the optimal configurations is challenging. To this end, we introduce a search algorithm based on hill-climbing for finding all optimal configurations in a feasible time and integrate it in an existing measuring framework. This approach enables the assessment, comparison and optimization of autonomic resource management approaches for micro-service applications. The evaluation shows that our approach is able to find all optimal configurations in the considered scenarios, while state-of-the-art multi-objective search algorithms do not.
• 
  Kaa: Evaluating Elasticity of Cloud-hosted DBMS Seybold, Daniel; Volpert, Simon; Wesner, Stefan; Bauer, André; Herbst, Nikolas; Domaschka, Jörg; in Proceedings of the 11th IEEE International Conference on Cloud Computing (CloudCom) (2019).
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @inproceedings{SeVoWeBaHeDo-CloudCom-Kaa, author = {Seybold, Daniel and Volpert, Simon and Wesner, Stefan and Bauer, Andr{\'e} and Herbst, Nikolas and Domaschka, J{\"o}rg}, booktitle = {Proceedings of the 11th IEEE International Conference on Cloud Computing (CloudCom)}, keywords = {myown}, month = {December}, title = {Kaa: Evaluating Elasticity of Cloud-hosted DBMS}, year = 2019 }

  %0 Conference Paper %1 SeVoWeBaHeDo-CloudCom-Kaa %A Seybold, Daniel %A Volpert, Simon %A Wesner, Stefan %A Bauer, André %A Herbst, Nikolas %A Domaschka, Jörg %B Proceedings of the 11th IEEE International Conference on Cloud Computing (CloudCom) %D 2019 %T Kaa: Evaluating Elasticity of Cloud-hosted DBMS
• 
  Best Practices for Time Series Forecasting Bauer, André; Züfle, Marwin; Herbst, Nikolas; Kounev, Samuel; in 2019 IEEE 4th International Workshops on Foundations and Applications of Self* Systems (FAS*W) (2019).
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @inproceedings{BaZuHeKo2019-ICAC-ForecastingTutorial, author = {Bauer, Andr{\'e} and Z{\"u}fle, Marwin and Herbst, Nikolas and Kounev, Samuel}, booktitle = {{2019 IEEE 4th International Workshops on Foundations and Applications of Self* Systems (FAS*W)}}, keywords = {descartes}, month = {June}, title = {{Best Practices for Time Series Forecasting}}, year = 2019 }

  %0 Conference Paper %1 BaZuHeKo2019-ICAC-ForecastingTutorial %A Bauer, André %A Züfle, Marwin %A Herbst, Nikolas %A Kounev, Samuel %B 2019 IEEE 4th International Workshops on Foundations and Applications of Self* Systems (FAS*W) %D 2019 %T Best Practices for Time Series Forecasting
• 
  Chamulteon: Coordinated Auto-Scaling of Micro-Services Bauer, André; Lesch, Veronika; Versluis, Laurens; Ilyushkin, Alexey; Herbst, Nikolas; Kounev, Samuel; in Proceedings of the 39th IEEE International Conference on Distributed Computing Systems (ICDCS) (2019).
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @inproceedings{BaLeVeIlHeKo2019-ICDCS-Chamulteon, author = {Bauer, Andr{\'e} and Lesch, Veronika and Versluis, Laurens and Ilyushkin, Alexey and Herbst, Nikolas and Kounev, Samuel}, booktitle = {Proceedings of the 39th IEEE International Conference on Distributed Computing Systems (ICDCS)}, keywords = {myown}, month = {July}, title = {Chamulteon: Coordinated Auto-Scaling of Micro-Services}, year = 2019 }

  %0 Conference Paper %1 BaLeVeIlHeKo2019-ICDCS-Chamulteon %A Bauer, André %A Lesch, Veronika %A Versluis, Laurens %A Ilyushkin, Alexey %A Herbst, Nikolas %A Kounev, Samuel %B Proceedings of the 39th IEEE International Conference on Distributed Computing Systems (ICDCS) %D 2019 %T Chamulteon: Coordinated Auto-Scaling of Micro-Services
• 
  Autonomic Forecasting Method Selection: Examination and Ways Ahead Züfle, Marwin; Bauer, André; Lesch, Veronika; Krupitzer, Christian; Herbst, Nikolas; Kounev, Samuel; Curtef, Valentin; in Proceedings of the 16th IEEE International Conference on Autonomic Computing (ICAC) (2019).
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @inproceedings{ZuBaLeKrHeKoCu2019-ICAC-Autonomic, author = {Z{\"u}fle, Marwin and Bauer, Andr{\'e} and Lesch, Veronika and Krupitzer, Christian and Herbst, Nikolas and Kounev, Samuel and Curtef, Valentin}, booktitle = {Proceedings of the 16th {IEEE} International Conference on Autonomic Computing (ICAC)}, keywords = {myown}, month = {June}, publisher = {IEEE}, title = {{Autonomic Forecasting Method Selection: Examination and Ways Ahead}}, year = 2019 }

  %0 Conference Paper %1 ZuBaLeKrHeKoCu2019-ICAC-Autonomic %A Züfle, Marwin %A Bauer, André %A Lesch, Veronika %A Krupitzer, Christian %A Herbst, Nikolas %A Kounev, Samuel %A Curtef, Valentin %B Proceedings of the 16th IEEE International Conference on Autonomic Computing (ICAC) %D 2019 %I IEEE %T Autonomic Forecasting Method Selection: Examination and Ways Ahead
• 
  Utilizing Clustering to Optimize Resource Demand Estimation Approaches Grohmann, Johannes; Eismann, Simon; Bauer, Andre; Zuefle, Marwin; Herbst, Nikolas; Kounev, Samuel; in 2019 IEEE 4th International Workshops on Foundations and Applications of Self* Systems (FAS*W) (2019). 134–139.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  • [ Download ]
  Resource demands are crucial parameters for modeling and predicting the performance of software systems. Direct measurement of these resource demands is usually infeasible due to instrumentation overheads causing measurement interferences and perturbation in production environments. Thus, a number of statistical estimation approaches (e.g., based on optimization, regression or Kalman filters) have been proposed in the literature. Most of these approaches are parameterized. These parameters influence the estimation quality and the required computation time. Existing work uses historical data as training sets to optimize those parameters and to minimize the estimation error of those approaches. However, if the data traces are fundamentally different, the optimal parameter settings are different as well. In this paper, we propose to use automated clustering in order to group training sets into groups of similar optimization behavior. This way, optimization can be specifically tailored to certain groups of traces in a self-aware manner. During run-time, every trace is first sorted into a cluster, where the respective cluster-wide parameter optimum can be applied. A preliminary case study shows that clustering can provide promising improvements.

  @inproceedings{GrEiBaZuHeKo2019-ICAC-RDEClustering, abstract = {Resource demands are crucial parameters for modeling and predicting the performance of software systems. Direct measurement of these resource demands is usually infeasible due to instrumentation overheads causing measurement interferences and perturbation in production environments. Thus, a number of statistical estimation approaches (e.g., based on optimization, regression or Kalman filters) have been proposed in the literature. Most of these approaches are parameterized. These parameters influence the estimation quality and the required computation time. Existing work uses historical data as training sets to optimize those parameters and to minimize the estimation error of those approaches. However, if the data traces are fundamentally different, the optimal parameter settings are different as well. In this paper, we propose to use automated clustering in order to group training sets into groups of similar optimization behavior. This way, optimization can be specifically tailored to certain groups of traces in a self-aware manner. During run-time, every trace is first sorted into a cluster, where the respective cluster-wide parameter optimum can be applied. A preliminary case study shows that clustering can provide promising improvements.}, author = {Grohmann, Johannes and Eismann, Simon and Bauer, Andre and Zuefle, Marwin and Herbst, Nikolas and Kounev, Samuel}, booktitle = {2019 IEEE 4th International Workshops on Foundations and Applications of Self* Systems (FAS*W)}, keywords = {myown}, month = {June}, pages = {134–139}, publisher = {IEEE}, title = {{Utilizing Clustering to Optimize Resource Demand Estimation Approaches}}, year = 2019 }

  %0 Conference Paper %1 GrEiBaZuHeKo2019-ICAC-RDEClustering %A Grohmann, Johannes %A Eismann, Simon %A Bauer, Andre %A Zuefle, Marwin %A Herbst, Nikolas %A Kounev, Samuel %B 2019 IEEE 4th International Workshops on Foundations and Applications of Self* Systems (FAS*W) %D 2019 %I IEEE %P 134--139 %R 10.1109/FAS-W.2019.00043 %T Utilizing Clustering to Optimize Resource Demand Estimation Approaches %U https://doi.org/10.1109/FAS-W.2019.00043 %X Resource demands are crucial parameters for modeling and predicting the performance of software systems. Direct measurement of these resource demands is usually infeasible due to instrumentation overheads causing measurement interferences and perturbation in production environments. Thus, a number of statistical estimation approaches (e.g., based on optimization, regression or Kalman filters) have been proposed in the literature. Most of these approaches are parameterized. These parameters influence the estimation quality and the required computation time. Existing work uses historical data as training sets to optimize those parameters and to minimize the estimation error of those approaches. However, if the data traces are fundamentally different, the optimal parameter settings are different as well. In this paper, we propose to use automated clustering in order to group training sets into groups of similar optimization behavior. This way, optimization can be specifically tailored to certain groups of traces in a self-aware manner. During run-time, every trace is first sorted into a cluster, where the respective cluster-wide parameter optimum can be applied. A preliminary case study shows that clustering can provide promising improvements.
• 
  FOX: Cost-Awareness for Autonomic Resource Management in Public Clouds Lesch, Veronika; Bauer, André; Herbst, Nikolas; Kounev, Samuel; in Proceedings of the 9th ACM/SPEC International Conference on Performance Engineering (ICPE 2018) (2018).

  Full Paper Acceptance Ratio: 23,7%
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  @inproceedings{LeBaHeKo-ICPE18-FOX-AS-COST, address = {New York, NY, USA}, author = {Lesch, Veronika and Bauer, Andr{\'e} and Herbst, Nikolas and Kounev, Samuel}, booktitle = {Proceedings of the 9th ACM/SPEC International Conference on Performance Engineering (ICPE 2018)}, keywords = {Prediction}, month = {April}, note = {Full Paper Acceptance Ratio: 23,7%}, publisher = {ACM}, title = {{FOX: Cost-Awareness for Autonomic Resource Management in Public Clouds}}, year = 2018 }

  %0 Conference Paper %1 LeBaHeKo-ICPE18-FOX-AS-COST %A Lesch, Veronika %A Bauer, André %A Herbst, Nikolas %A Kounev, Samuel %B Proceedings of the 9th ACM/SPEC International Conference on Performance Engineering (ICPE 2018) %C New York, NY, USA %D 2018 %I ACM %T FOX: Cost-Awareness for Autonomic Resource Management in Public Clouds %U https://doi.org/10.1145/3184407.3184415
• 
  On the Value of Service Demand Estimation for Auto-Scaling Bauer, André; Grohmann, Johannes; Herbst, Nikolas; Kounev, Samuel; in Proceedings of 19th International GI/ITG Conference on Measurement, Modelling and Evaluation of Computing Systems (MMB 2018) (2018). (Vol. 10740) 142–156.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  • [ Download ]
  In the context of performance models, service demands are key model parameters capturing the average time individual requests of different workload classes are actively processed. In a system under load, due to measurement interference, service demands normally cannot be measured directly, however, a number of estimation approaches exist based on high-level performance metrics. In this paper, we show that service demands provide significant benefits for implementing modern auto-scalers. Auto-scaling describes the process of dynamically adjusting the number of allocated virtual resources (e.g., virtual machines) in a data center according to the incoming workload. We demonstrate that even a simple auto-scaler that leverages information about service demands significantly outperforms auto-scalers solely based on CPU utilization measurements. This is shown by testing two approaches in three different scenarios. Our results show that the service demand-based auto-scaler outperforms the CPU utilization-based one in all scenarios. Our results encourage further research on the application of service demand estimates for resource management in data centers.

  @inproceedings{BaGrHeKo2018-MMB-ServiceDemand, abstract = {In the context of performance models, service demands are key model parameters capturing the average time individual requests of different workload classes are actively processed. In a system under load, due to measurement interference, service demands normally cannot be measured directly, however, a number of estimation approaches exist based on high-level performance metrics. In this paper, we show that service demands provide significant benefits for implementing modern auto-scalers. Auto-scaling describes the process of dynamically adjusting the number of allocated virtual resources (e.g., virtual machines) in a data center according to the incoming workload. We demonstrate that even a simple auto-scaler that leverages information about service demands significantly outperforms auto-scalers solely based on CPU utilization measurements. This is shown by testing two approaches in three different scenarios. Our results show that the service demand-based auto-scaler outperforms the CPU utilization-based one in all scenarios. Our results encourage further research on the application of service demand estimates for resource management in data centers.}, address = {Cham}, author = {Bauer, Andr{\'e} and Grohmann, Johannes and Herbst, Nikolas and Kounev, Samuel}, booktitle = {Proceedings of 19th International GI/ITG Conference on Measurement, Modelling and Evaluation of Computing Systems (MMB 2018)}, keywords = {myown}, month = {February}, pages = {142–156}, publisher = {Springer}, series = {Lecture Notes in Computer Science}, title = {{On the Value of Service Demand Estimation for Auto-Scaling}}, volume = 10740, year = 2018 }

  %0 Conference Paper %1 BaGrHeKo2018-MMB-ServiceDemand %A Bauer, André %A Grohmann, Johannes %A Herbst, Nikolas %A Kounev, Samuel %B Proceedings of 19th International GI/ITG Conference on Measurement, Modelling and Evaluation of Computing Systems (MMB 2018) %C Cham %D 2018 %I Springer %P 142--156 %R 10.1007/978-3-319-74947-1_10 %T On the Value of Service Demand Estimation for Auto-Scaling %U https://doi.org/10.1007/978-3-319-74947-1_10 %V 10740 %X In the context of performance models, service demands are key model parameters capturing the average time individual requests of different workload classes are actively processed. In a system under load, due to measurement interference, service demands normally cannot be measured directly, however, a number of estimation approaches exist based on high-level performance metrics. In this paper, we show that service demands provide significant benefits for implementing modern auto-scalers. Auto-scaling describes the process of dynamically adjusting the number of allocated virtual resources (e.g., virtual machines) in a data center according to the incoming workload. We demonstrate that even a simple auto-scaler that leverages information about service demands significantly outperforms auto-scalers solely based on CPU utilization measurements. This is shown by testing two approaches in three different scenarios. Our results show that the service demand-based auto-scaler outperforms the CPU utilization-based one in all scenarios. Our results encourage further research on the application of service demand estimates for resource management in data centers. %@ 978-3-319-74947-1
• 
  TeaStore: A Micro-Service Reference Application for Cloud Researchers Eismann, Simon; v. Kistowski, Jóakim; Grohmann, Johannes; Bauer, André; Schmitt, Norbert; Herbst, Nikolas; Kounev, Samuel; in Proceedings of 2018 IEEE/ACM International Conference on Utility and Cloud Computing Companion (UCC Companion) (2018). 11–12.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  • [ Download ]
  Researchers propose and employ various methods to analyze, model, optimize and manage modern distributed cloud applications. In order to demonstrate and evaluate these methods in realistic scenarios, researchers rely on reference applications. These applications should offer a range of different behaviors, degrees of freedom allowing for customization and should use a modern and representative technology stack. Existing testing and benchmarking applications are either outdated, designed for specific testing scenarios, or do not offer the necessary degrees of freedom. Further, most cloud reference applications are difficult to deploy and run. In this paper, we present the TeaStore, a micro-service-based test and reference cloud application. TeaStore offers services with various performance characteristics and a high degree of freedom regarding its deployment and configuration to be used as a cloud reference application for researchers. The TeaStore is designed for the evaluation of performance modeling and resource management techniques. We invite cloud researchers to use the TeaStore and provide it open-source, extendable, easily deployable and monitorable.

  @inproceedings{EiKiGrBaScHeKo2018-UCC-TeastorePoster, abstract = {Researchers propose and employ various methods to analyze, model, optimize and manage modern distributed cloud applications. In order to demonstrate and evaluate these methods in realistic scenarios, researchers rely on reference applications. These applications should offer a range of different behaviors, degrees of freedom allowing for customization and should use a modern and representative technology stack. Existing testing and benchmarking applications are either outdated, designed for specific testing scenarios, or do not offer the necessary degrees of freedom. Further, most cloud reference applications are difficult to deploy and run. In this paper, we present the TeaStore, a micro-service-based test and reference cloud application. TeaStore offers services with various performance characteristics and a high degree of freedom regarding its deployment and configuration to be used as a cloud reference application for researchers. The TeaStore is designed for the evaluation of performance modeling and resource management techniques. We invite cloud researchers to use the TeaStore and provide it open-source, extendable, easily deployable and monitorable.}, author = {Eismann, Simon and v. Kistowski, J{\'o}akim and Grohmann, Johannes and Bauer, Andr{\'e} and Schmitt, Norbert and Herbst, Nikolas and Kounev, Samuel}, booktitle = {Proceedings of 2018 IEEE/ACM International Conference on Utility and Cloud Computing Companion (UCC Companion)}, keywords = {myown}, month = {dec}, pages = {11–12}, publisher = {IEEE}, title = {TeaStore: A Micro-Service Reference Application for Cloud Researchers}, year = 2018 }

  %0 Conference Paper %1 EiKiGrBaScHeKo2018-UCC-TeastorePoster %A Eismann, Simon %A v. Kistowski, Jóakim %A Grohmann, Johannes %A Bauer, André %A Schmitt, Norbert %A Herbst, Nikolas %A Kounev, Samuel %B Proceedings of 2018 IEEE/ACM International Conference on Utility and Cloud Computing Companion (UCC Companion) %D 2018 %I IEEE %P 11--12 %R 10.1109/UCC-Companion.2018.00021 %T TeaStore: A Micro-Service Reference Application for Cloud Researchers %U https://doi.org/10.1109/UCC-Companion.2018.00021 %X Researchers propose and employ various methods to analyze, model, optimize and manage modern distributed cloud applications. In order to demonstrate and evaluate these methods in realistic scenarios, researchers rely on reference applications. These applications should offer a range of different behaviors, degrees of freedom allowing for customization and should use a modern and representative technology stack. Existing testing and benchmarking applications are either outdated, designed for specific testing scenarios, or do not offer the necessary degrees of freedom. Further, most cloud reference applications are difficult to deploy and run. In this paper, we present the TeaStore, a micro-service-based test and reference cloud application. TeaStore offers services with various performance characteristics and a high degree of freedom regarding its deployment and configuration to be used as a cloud reference application for researchers. The TeaStore is designed for the evaluation of performance modeling and resource management techniques. We invite cloud researchers to use the TeaStore and provide it open-source, extendable, easily deployable and monitorable.
• 
  Using Machine Learning for Recommending Service Demand Estimation Approaches Grohmann, Johannes; Herbst, Nikolas; Spinner, Simon; Kounev, Samuel; in Proceedings of the 8th International Conference on Cloud Computing and Services Science (CLOSER 2018) (2018). 473–480.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  • [ Download ]
  Service demands are key parameters in service and performance modeling. Hence, a variety of different approaches to service demand estimation exist in the literature. However, given a specific scenario, it is not trivial to select the currently best approach, since deep expertise in statistical estimation techniques is required and the requirements and characteristics of the application scenario might change over time (e.g., by varying load patterns). To tackle this problem, we propose the use of machine learning techniques to automatically recommend the best suitable approach for the target scenario. The approach works in an online fashion and can incorporate new measurement data and changing characteristics on-the-fly. Preliminary results show that executing only the recommended estimation approach achieves 99.6% accuracy compared to executing all approaches available, while speeding up the estimation time by 57%.

  @inproceedings{GrHeSpKo-Closer18-RecServDemEst, abstract = {Service demands are key parameters in service and performance modeling. Hence, a variety of different approaches to service demand estimation exist in the literature. However, given a specific scenario, it is not trivial to select the currently best approach, since deep expertise in statistical estimation techniques is required and the requirements and characteristics of the application scenario might change over time (e.g., by varying load patterns). To tackle this problem, we propose the use of machine learning techniques to automatically recommend the best suitable approach for the target scenario. The approach works in an online fashion and can incorporate new measurement data and changing characteristics on-the-fly. Preliminary results show that executing only the recommended estimation approach achieves 99.6% accuracy compared to executing all approaches available, while speeding up the estimation time by 57%.}, author = {Grohmann, Johannes and Herbst, Nikolas and Spinner, Simon and Kounev, Samuel}, booktitle = {Proceedings of the 8th International Conference on Cloud Computing and Services Science (CLOSER 2018)}, keywords = {myown}, month = {March}, organization = {INSTICC}, pages = {473–480}, publisher = {SciTePress}, title = {Using Machine Learning for Recommending Service Demand Estimation Approaches}, year = 2018 }

  %0 Conference Paper %1 GrHeSpKo-Closer18-RecServDemEst %A Grohmann, Johannes %A Herbst, Nikolas %A Spinner, Simon %A Kounev, Samuel %B Proceedings of the 8th International Conference on Cloud Computing and Services Science (CLOSER 2018) %D 2018 %I SciTePress %P 473--480 %R 10.5220/0006761104730480 %T Using Machine Learning for Recommending Service Demand Estimation Approaches %U https://doi.org/10.5220/0006761104730480 %X Service demands are key parameters in service and performance modeling. Hence, a variety of different approaches to service demand estimation exist in the literature. However, given a specific scenario, it is not trivial to select the currently best approach, since deep expertise in statistical estimation techniques is required and the requirements and characteristics of the application scenario might change over time (e.g., by varying load patterns). To tackle this problem, we propose the use of machine learning techniques to automatically recommend the best suitable approach for the target scenario. The approach works in an online fashion and can incorporate new measurement data and changing characteristics on-the-fly. Preliminary results show that executing only the recommended estimation approach achieves 99.6% accuracy compared to executing all approaches available, while speeding up the estimation time by 57%. %@ 978-989-758-295-0
• 
  Scalability Analysis of Cloud Software Services Brataas, Gunnar; Herbst, Nikolas; Ivansek, Simon; Polutnik, Jure; in Companion Proceedings of the 14th IEEE International Conference on Autonomic Computing (ICAC 2017), Self Organizing Self Managing Clouds Workshop (SOSeMC 2017) (2017).
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @inproceedings{BrHe2017CloudScalability, author = {Brataas, Gunnar and Herbst, Nikolas and Ivansek, Simon and Polutnik, Jure}, booktitle = {Companion Proceedings of the 14th IEEE International Conference on Autonomic Computing (ICAC 2017), Self Organizing Self Managing Clouds Workshop (SOSeMC 2017)}, keywords = {descartes}, month = {July}, publisher = {IEEE}, title = {{Scalability Analysis of Cloud Software Services}}, year = 2017 }

  %0 Conference Paper %1 BrHe2017CloudScalability %A Brataas, Gunnar %A Herbst, Nikolas %A Ivansek, Simon %A Polutnik, Jure %B Companion Proceedings of the 14th IEEE International Conference on Autonomic Computing (ICAC 2017), Self Organizing Self Managing Clouds Workshop (SOSeMC 2017) %D 2017 %I IEEE %T Scalability Analysis of Cloud Software Services
• 
  An Experimental Performance Evaluation of Autoscaling Policies for Complex Workflows Ilyushkin, Alexey; Ali-Eldin, Ahmed; Herbst, Nikolas; Papadopoulos, Alessandro V.; Ghit, Bogdan; Epema, Dick; Iosup, Alexandru; in Proceedings of the 8th ACM/SPEC International Conference on Performance Engineering (ICPE 2017) (2017).

  Best Paper Candidate (1/4)
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Simplifying the task of resource management and scheduling for customers, while still delivering complex Quality-of-Service (QoS), is key to cloud computing. Many autoscaling policies have been proposed in the past decade to decide on behalf of cloud customers when and how to provision resources to a cloud application utilizing cloud elasticity features. However, in prior work, when a new policy is proposed, it is seldom compared to the state-of-the-art, and is often compared only to static provisioning using a predefined QoS target. This reduces the ability of cloud customers and of cloud operators to choose and deploy an autoscaling policy. In our work, we conduct an experimental performance evaluation of autoscaling policies, using as application model workflows, a commonly used formalism for automating resource management for applications with well-defined yet complex structure. We present a detailed comparative study of general, state-of-the-art, generic autoscaling policies, along with two new workflow-specific policies. To understand the performance differences between the 7 policies, we conduct various forms of pairwise and group comparisons. We report both individual and aggregated metrics. Our results highlights the trade-offs between the suggested policies, and thus enable a better understanding of the current state-of-the-art.

  @inproceedings{IlAlHePa-ICPE17-AutoScalerWorkflowEval, abstract = {Simplifying the task of resource management and scheduling for customers, while still delivering complex Quality-of-Service (QoS), is key to cloud computing. Many autoscaling policies have been proposed in the past decade to decide on behalf of cloud customers when and how to provision resources to a cloud application utilizing cloud elasticity features. However, in prior work, when a new policy is proposed, it is seldom compared to the state-of-the-art, and is often compared only to static provisioning using a predefined QoS target. This reduces the ability of cloud customers and of cloud operators to choose and deploy an autoscaling policy. In our work, we conduct an experimental performance evaluation of autoscaling policies, using as application model workflows, a commonly used formalism for automating resource management for applications with well-defined yet complex structure. We present a detailed comparative study of general, state-of-the-art, generic autoscaling policies, along with two new workflow-specific policies. To understand the performance differences between the 7 policies, we conduct various forms of pairwise and group comparisons. We report both individual and aggregated metrics. Our results highlights the trade-offs between the suggested policies, and thus enable a better understanding of the current state-of-the-art.}, address = {New York, NY, USA}, author = {Ilyushkin, Alexey and Ali-Eldin, Ahmed and Herbst, Nikolas and Papadopoulos, Alessandro V. and Ghit, Bogdan and Epema, Dick and Iosup, Alexandru}, booktitle = {Proceedings of the 8th ACM/SPEC International Conference on Performance Engineering (ICPE 2017)}, keywords = {myown}, month = {April}, note = {<b>Best Paper Candidate (1/4)</b>}, publisher = {ACM}, title = {{An Experimental Performance Evaluation of Autoscaling Policies for Complex Workflows}}, year = 2017 }

  %0 Conference Paper %1 IlAlHePa-ICPE17-AutoScalerWorkflowEval %A Ilyushkin, Alexey %A Ali-Eldin, Ahmed %A Herbst, Nikolas %A Papadopoulos, Alessandro V. %A Ghit, Bogdan %A Epema, Dick %A Iosup, Alexandru %B Proceedings of the 8th ACM/SPEC International Conference on Performance Engineering (ICPE 2017) %C New York, NY, USA %D 2017 %I ACM %T An Experimental Performance Evaluation of Autoscaling Policies for Complex Workflows %X Simplifying the task of resource management and scheduling for customers, while still delivering complex Quality-of-Service (QoS), is key to cloud computing. Many autoscaling policies have been proposed in the past decade to decide on behalf of cloud customers when and how to provision resources to a cloud application utilizing cloud elasticity features. However, in prior work, when a new policy is proposed, it is seldom compared to the state-of-the-art, and is often compared only to static provisioning using a predefined QoS target. This reduces the ability of cloud customers and of cloud operators to choose and deploy an autoscaling policy. In our work, we conduct an experimental performance evaluation of autoscaling policies, using as application model workflows, a commonly used formalism for automating resource management for applications with well-defined yet complex structure. We present a detailed comparative study of general, state-of-the-art, generic autoscaling policies, along with two new workflow-specific policies. To understand the performance differences between the 7 policies, we conduct various forms of pairwise and group comparisons. We report both individual and aggregated metrics. Our results highlights the trade-offs between the suggested policies, and thus enable a better understanding of the current state-of-the-art.
• 
  Design and Evaluation of a Proactive, Application-Aware Auto-Scaler Bauer, André; Herbst, Nikolas; Kounev, Samuel; in Proceedings of the 8th ACM/SPEC International Conference on Performance Engineering (ICPE 2017) (2017).
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @inproceedings{BaHeKo2017-ICPE-Auto-Scaler-Tutorial, author = {Bauer, Andr{\'e} and Herbst, Nikolas and Kounev, Samuel}, booktitle = {{Proceedings of the 8th ACM/SPEC International Conference on Performance Engineering (ICPE 2017)}}, keywords = {myown}, month = {April}, title = {{Design and Evaluation of a Proactive, Application-Aware Auto-Scaler}}, year = 2017 }

  %0 Conference Paper %1 BaHeKo2017-ICPE-Auto-Scaler-Tutorial %A Bauer, André %A Herbst, Nikolas %A Kounev, Samuel %B Proceedings of the 8th ACM/SPEC International Conference on Performance Engineering (ICPE 2017) %D 2017 %T Design and Evaluation of a Proactive, Application-Aware Auto-Scaler
• 
  Self-Tuning Resource Demand Estimation Grohmann, Johannes; Herbst, Nikolas; Spinner, Simon; Kounev, Samuel; in Proceedings of the 14th IEEE International Conference on Autonomic Computing (ICAC 2017) (2017). 21–26.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  • [ Download ]
  The average time a resource needs to process incoming requests in a monitored workload mix is a key parameter of stochastic performance models. Direct measurement of these resource demands is usually infeasible due to instrumentation overheads causing measurement interferences and perturbation in production environments.Thus, a number of statistical estimation approaches (e.g., based on optimization, regression or Kalman filters) have been proposed in the literature each coming with different strengths and run-time overheads. Most approaches offer parameters in order to customize the behavior of the estimator influencing the estimation quality and the required computation time. However, their configuration usually requires exhaustive testing, as default parameters normally do not provide optimal performance.In this paper, we propose a self-tuning approach based on discrete optimization that can be used to automatically tune the parameters of resource demand estimation methods, tailoring them to the specific application scenario and thus improving their accuracy. We apply and compare different techniques on a representative data set with varying load levels and number of workload classes. We show that our selected approach for parameter tuning can automatically improve the estimation quality of certain estimators by up to 25%.

  @inproceedings{GrHeSpKo2017-ICAC-RDE, abstract = {The average time a resource needs to process incoming requests in a monitored workload mix is a key parameter of stochastic performance models. Direct measurement of these resource demands is usually infeasible due to instrumentation overheads causing measurement interferences and perturbation in production environments.Thus, a number of statistical estimation approaches (e.g., based on optimization, regression or Kalman filters) have been proposed in the literature each coming with different strengths and run-time overheads. Most approaches offer parameters in order to customize the behavior of the estimator influencing the estimation quality and the required computation time. However, their configuration usually requires exhaustive testing, as default parameters normally do not provide optimal performance.In this paper, we propose a self-tuning approach based on discrete optimization that can be used to automatically tune the parameters of resource demand estimation methods, tailoring them to the specific application scenario and thus improving their accuracy. We apply and compare different techniques on a representative data set with varying load levels and number of workload classes. We show that our selected approach for parameter tuning can automatically improve the estimation quality of certain estimators by up to 25%.}, author = {Grohmann, Johannes and Herbst, Nikolas and Spinner, Simon and Kounev, Samuel}, booktitle = {Proceedings of the 14th IEEE International Conference on Autonomic Computing (ICAC 2017)}, keywords = {myown}, month = {July}, pages = {21–26}, title = {{Self-Tuning Resource Demand Estimation}}, year = 2017 }

  %0 Conference Paper %1 GrHeSpKo2017-ICAC-RDE %A Grohmann, Johannes %A Herbst, Nikolas %A Spinner, Simon %A Kounev, Samuel %B Proceedings of the 14th IEEE International Conference on Autonomic Computing (ICAC 2017) %D 2017 %P 21--26 %R 10.1109/ICAC.2017.19 %T Self-Tuning Resource Demand Estimation %U https://doi.org/10.1109/ICAC.2017.19 %X The average time a resource needs to process incoming requests in a monitored workload mix is a key parameter of stochastic performance models. Direct measurement of these resource demands is usually infeasible due to instrumentation overheads causing measurement interferences and perturbation in production environments.Thus, a number of statistical estimation approaches (e.g., based on optimization, regression or Kalman filters) have been proposed in the literature each coming with different strengths and run-time overheads. Most approaches offer parameters in order to customize the behavior of the estimator influencing the estimation quality and the required computation time. However, their configuration usually requires exhaustive testing, as default parameters normally do not provide optimal performance.In this paper, we propose a self-tuning approach based on discrete optimization that can be used to automatically tune the parameters of resource demand estimation methods, tailoring them to the specific application scenario and thus improving their accuracy. We apply and compare different techniques on a representative data set with varying load levels and number of workload classes. We show that our selected approach for parameter tuning can automatically improve the estimation quality of certain estimators by up to 25%.
• 
  Telescope: A Hybrid Forecast Method for Univariate Time Series Züfle, Marwin; Bauer, André; Herbst, Nikolas; Curtef, Valentin; Kounev, Samuel; in Proceedings of the International work-conference on Time Series (ITISE 2017) (2017).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Forecasting is an important part of the decision-making process and used in many fields like business, economics, finance, science, and engineering. According to the No-Free-Lunch-Theorem from 1997, there is no general forecasting method, that performs best for all time series. Instead, expert knowledge is needed to decide which forecasting method to choose for a specific time series with its own characteristics. Since a trial and error approach is very inefficient and expert knowledge is useful but a time-consuming task that cannot be fully automated, we present a new hybrid multi-step-ahead forecasting approach based on time series decomposition. Initial evaluations show that this hybrid approach improves the forecast accuracy compared to six existing forecasting methods while maintaining a short runtime.

  @inproceedings{ZBBHCK2017-ITISE-Telescope, abstract = {Forecasting is an important part of the decision-making process and used in many fields like business, economics, finance, science, and engineering. According to the No-Free-Lunch-Theorem from 1997, there is no general forecasting method, that performs best for all time series. Instead, expert knowledge is needed to decide which forecasting method to choose for a specific time series with its own characteristics. Since a trial and error approach is very inefficient and expert knowledge is useful but a time-consuming task that cannot be fully automated, we present a new hybrid multi-step-ahead forecasting approach based on time series decomposition. Initial evaluations show that this hybrid approach improves the forecast accuracy compared to six existing forecasting methods while maintaining a short runtime.}, author = {Z{\"u}fle, Marwin and Bauer, Andr{\'e} and Herbst, Nikolas and Curtef, Valentin and Kounev, Samuel}, booktitle = {{Proceedings of the International work-conference on Time Series (ITISE 2017)}}, keywords = {myown}, month = {September}, title = {{Telescope: A Hybrid Forecast Method for Univariate Time Series}}, year = 2017 }

  %0 Conference Paper %1 ZBBHCK2017-ITISE-Telescope %A Züfle, Marwin %A Bauer, André %A Herbst, Nikolas %A Curtef, Valentin %A Kounev, Samuel %B Proceedings of the International work-conference on Time Series (ITISE 2017) %D 2017 %T Telescope: A Hybrid Forecast Method for Univariate Time Series %X Forecasting is an important part of the decision-making process and used in many fields like business, economics, finance, science, and engineering. According to the No-Free-Lunch-Theorem from 1997, there is no general forecasting method, that performs best for all time series. Instead, expert knowledge is needed to decide which forecasting method to choose for a specific time series with its own characteristics. Since a trial and error approach is very inefficient and expert knowledge is useful but a time-consuming task that cannot be fully automated, we present a new hybrid multi-step-ahead forecasting approach based on time series decomposition. Initial evaluations show that this hybrid approach improves the forecast accuracy compared to six existing forecasting methods while maintaining a short runtime.
• 
  Modeling and Extracting Load Intensity Profiles von Kistowski, Jóakim; Herbst, Nikolas Roman; Zoller, Daniel; Kounev, Samuel; Hotho, Andreas; in Proceedings of the 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS 2015) (2015).

  Acceptance rate: 29%
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Today's system developers and operators face the challenge of creating software systems that make efficient use of dynamically allocated resources under highly variable and dynamic load profiles, while at the same time delivering reliable performance. Benchmarking of systems under these constraints is difficult, as state-of-the-art benchmarking frameworks provide only limited support for emulating such dynamic and highly variable load profiles for the creation of realistic workload scenarios. Industrial benchmarks typically confine themselves to workloads with constant or stepwise increasing loads. Alternatively, they support replaying of recorded load traces. Statistical load intensity descriptions also do not sufficiently capture concrete pattern load profile variations over time. To address these issues, we present the Descartes Load Intensity Model (DLIM). DLIM provides a modeling formalism for describing load intensity variations over time. A DLIM instance can be used as a compact representation of a recorded load intensity trace, providing a powerful tool for benchmarking and performance analysis. As manually obtaining DLIM instances can be time consuming, we present three different automated extraction methods, which also help to enable autonomous system analysis for self-adaptive systems. Model expressiveness is validated using the presented extraction methods. Extracted DLIM instances exhibit a median modeling error of 12.4% on average over nine different real-world traces covering between two weeks and seven months. Additionally, extraction methods perform orders of magnitude faster than existing time series decomposition approaches.

  @inproceedings{KiHeZoKoHo2015_SEAMS_DLIMExtraction, abstract = {Today's system developers and operators face the challenge of creating software systems that make efficient use of dynamically allocated resources under highly variable and dynamic load profiles, while at the same time delivering reliable performance. Benchmarking of systems under these constraints is difficult, as state-of-the-art benchmarking frameworks provide only limited support for emulating such dynamic and highly variable load profiles for the creation of realistic workload scenarios. Industrial benchmarks typically confine themselves to workloads with constant or stepwise increasing loads. Alternatively, they support replaying of recorded load traces. Statistical load intensity descriptions also do not sufficiently capture concrete pattern load profile variations over time. To address these issues, we present the Descartes Load Intensity Model (DLIM). DLIM provides a modeling formalism for describing load intensity variations over time. A DLIM instance can be used as a compact representation of a recorded load intensity trace, providing a powerful tool for benchmarking and performance analysis. As manually obtaining DLIM instances can be time consuming, we present three different automated extraction methods, which also help to enable autonomous system analysis for self-adaptive systems. Model expressiveness is validated using the presented extraction methods. Extracted DLIM instances exhibit a median modeling error of 12.4% on average over nine different real-world traces covering between two weeks and seven months. Additionally, extraction methods perform orders of magnitude faster than existing time series decomposition approaches.}, author = {von Kistowski, J{\'o}akim and Herbst, Nikolas Roman and Zoller, Daniel and Kounev, Samuel and Hotho, Andreas}, booktitle = {Proceedings of the 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS 2015)}, keywords = {myown}, month = {may}, note = {Acceptance rate: 29\%}, title = {{Modeling and Extracting Load Intensity Profiles}}, year = 2015 }

  %0 Conference Paper %1 KiHeZoKoHo2015_SEAMS_DLIMExtraction %A von Kistowski, Jóakim %A Herbst, Nikolas Roman %A Zoller, Daniel %A Kounev, Samuel %A Hotho, Andreas %B Proceedings of the 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS 2015) %D 2015 %T Modeling and Extracting Load Intensity Profiles %X Today's system developers and operators face the challenge of creating software systems that make efficient use of dynamically allocated resources under highly variable and dynamic load profiles, while at the same time delivering reliable performance. Benchmarking of systems under these constraints is difficult, as state-of-the-art benchmarking frameworks provide only limited support for emulating such dynamic and highly variable load profiles for the creation of realistic workload scenarios. Industrial benchmarks typically confine themselves to workloads with constant or stepwise increasing loads. Alternatively, they support replaying of recorded load traces. Statistical load intensity descriptions also do not sufficiently capture concrete pattern load profile variations over time. To address these issues, we present the Descartes Load Intensity Model (DLIM). DLIM provides a modeling formalism for describing load intensity variations over time. A DLIM instance can be used as a compact representation of a recorded load intensity trace, providing a powerful tool for benchmarking and performance analysis. As manually obtaining DLIM instances can be time consuming, we present three different automated extraction methods, which also help to enable autonomous system analysis for self-adaptive systems. Model expressiveness is validated using the presented extraction methods. Extracted DLIM instances exhibit a median modeling error of 12.4% on average over nine different real-world traces covering between two weeks and seven months. Additionally, extraction methods perform orders of magnitude faster than existing time series decomposition approaches.
• 
  Proactive Memory Scaling of Virtualized Applications Spinner, Simon; Zhu, Xiaoyun; Lu, Lei; Uysal, Mustafa; Griffith, Rean; Herbst, Nikolas; Kounev, Samuel; in VMware R&D Innovation Offsite (RADIO) (2015).
  • [ BibTeX ]
  • [ EndNote ]
  @inproceedings{SpinnerZLUGHK-2015-RADIO-MemoryScaling, author = {Spinner, Simon and Zhu, Xiaoyun and Lu, Lei and Uysal, Mustafa and Griffith, Rean and Herbst, Nikolas and Kounev, Samuel}, booktitle = {VMware R&D Innovation Offsite (RADIO)}, keywords = {myown}, month = {July}, title = {{Proactive Memory Scaling of Virtualized Applications}}, year = 2015 }

  %0 Conference Paper %1 SpinnerZLUGHK-2015-RADIO-MemoryScaling %A Spinner, Simon %A Zhu, Xiaoyun %A Lu, Lei %A Uysal, Mustafa %A Griffith, Rean %A Herbst, Nikolas %A Kounev, Samuel %B VMware R&D Innovation Offsite (RADIO) %D 2015 %T Proactive Memory Scaling of Virtualized Applications
• 
  Proactive Memory Scaling of Virtualized Applications Spinner, Simon; Herbst, Nikolas; Kounev, Samuel; Zhu, Xiaoyun; Lu, Lei; Uysal, Mustafa; Griffith, Rean; in Proceedings of the 2015 IEEE 8th International Conference on Cloud Computing (IEEE CLOUD 2015) (2015). 277–284.

  Acceptance Rate: 15%
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Enterprise applications in virtualized environments are often subject to time-varying workloads with multiple seasonal patterns and trends. In order to ensure quality of service for such applications while avoiding over-provisioning, resources need to be dynamically adapted to accommodate the current workload demands. Many memory-intensive applications are not suitable for the traditional horizontal scaling approach often used for runtime performance management, as it relies on complex and expensive state replication. On the other hand, vertical scaling of memory often requires a restart of the application. In this paper, we propose a proactive approach to memory scaling for virtualized applications. It uses statistical forecasting to predict the future workload and reconfigure the memory size of the virtual machine of an application automatically. To this end, we propose an extended forecasting technique that leverages meta-knowledge, such as calendar information, to improve the forecast accuracy. In addition, we develop an application controller to adjust settings associated with application memory management during memory reconfiguration. Our evaluation using real-world traces shows that the forecast accuracy quantified with the MASE error metric can be improved by 11-59%. Furthermore, we demonstrate that the proactive approach can reduce the impact of reconfiguration on application availability by over 80% and significantly improve performance relative to a reactive controller.

  @inproceedings{SpHeKoZhLuUyGr2015-CLOUD-ProactiveMemoryScaling, abstract = {{Enterprise applications in virtualized environments are often subject to time-varying workloads with multiple seasonal patterns and trends. In order to ensure quality of service for such applications while avoiding over-provisioning, resources need to be dynamically adapted to accommodate the current workload demands. Many memory-intensive applications are not suitable for the traditional horizontal scaling approach often used for runtime performance management, as it relies on complex and expensive state replication. On the other hand, vertical scaling of memory often requires a restart of the application. In this paper, we propose a proactive approach to memory scaling for virtualized applications. It uses statistical forecasting to predict the future workload and reconfigure the memory size of the virtual machine of an application automatically. To this end, we propose an extended forecasting technique that leverages meta-knowledge, such as calendar information, to improve the forecast accuracy. In addition, we develop an application controller to adjust settings associated with application memory management during memory reconfiguration. Our evaluation using real-world traces shows that the forecast accuracy quantified with the MASE error metric can be improved by 11-59%. Furthermore, we demonstrate that the proactive approach can reduce the impact of reconfiguration on application availability by over 80% and significantly improve performance relative to a reactive controller.}}, author = {Spinner, Simon and Herbst, Nikolas and Kounev, Samuel and Zhu, Xiaoyun and Lu, Lei and Uysal, Mustafa and Griffith, Rean}, booktitle = {Proceedings of the 2015 IEEE 8th International Conference on Cloud Computing (IEEE CLOUD 2015)}, keywords = {myown}, month = {June}, note = {Acceptance Rate: 15%}, pages = {277–284}, publisher = {IEEE}, title = {{Proactive Memory Scaling of Virtualized Applications}}, year = 2015 }

  %0 Conference Paper %1 SpHeKoZhLuUyGr2015-CLOUD-ProactiveMemoryScaling %A Spinner, Simon %A Herbst, Nikolas %A Kounev, Samuel %A Zhu, Xiaoyun %A Lu, Lei %A Uysal, Mustafa %A Griffith, Rean %B Proceedings of the 2015 IEEE 8th International Conference on Cloud Computing (IEEE CLOUD 2015) %D 2015 %I IEEE %P 277--284 %T Proactive Memory Scaling of Virtualized Applications %X Enterprise applications in virtualized environments are often subject to time-varying workloads with multiple seasonal patterns and trends. In order to ensure quality of service for such applications while avoiding over-provisioning, resources need to be dynamically adapted to accommodate the current workload demands. Many memory-intensive applications are not suitable for the traditional horizontal scaling approach often used for runtime performance management, as it relies on complex and expensive state replication. On the other hand, vertical scaling of memory often requires a restart of the application. In this paper, we propose a proactive approach to memory scaling for virtualized applications. It uses statistical forecasting to predict the future workload and reconfigure the memory size of the virtual machine of an application automatically. To this end, we propose an extended forecasting technique that leverages meta-knowledge, such as calendar information, to improve the forecast accuracy. In addition, we develop an application controller to adjust settings associated with application memory management during memory reconfiguration. Our evaluation using real-world traces shows that the forecast accuracy quantified with the MASE error metric can be improved by 11-59%. Furthermore, we demonstrate that the proactive approach can reduce the impact of reconfiguration on application availability by over 80% and significantly improve performance relative to a reactive controller.
• 
  Using and Extending LIMBO for the Descriptive Modeling of Arrival Behaviors von Kistowski, Jóakim; Herbst, Nikolas Roman; Kounev, Samuel; in Proceedings of the Symposium on Software Performance 2014 (2014). 131–140.

  Best Poster Award
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  LIMBO is a tool for the creation of load profiles with variable intensity over time both from scratch and from existing data. Primarily, LIMBO's intended use is the description of load arrival behaviours in open workloads. Specifically, LIMBO can be employed for the creation of custom request or user arrival time-stamps or for the re-parameterization of existing traces. LIMBO bases on the Descartes Load Intensity Model (DLIM) for the formalized description of its load intensity profiles. The DLIM formalism can be understood as a structure for piece-wise defined and combined mathematical functions. We outline DLIM and its elements and demonstrate its integration within LIMBO. LIMBO is capable of generating request or user arrival time stamps from DLIM instances. In a next step, these generated time-stamps can be used for both open workload based benchmarking and simulations. The TimestampTimer plug-in for JMeter already allows the former. LIMBO also offers a range of tools for easy load intensity modeling and analysis, including, but not limited to, a visual decomposition of load intensity time-series into seasonal and trend parts, a simplified load intensity model as part of a model creation wizard, and an automated model instance extractor. As part of our LIMBO tutorial, we explain these features in detail. We demonstrate common use cases for LIMBO and show how they are supported. We also focus on LIMBOs extensible architecture and discuss how to use LIMBO as part of another tool-chain.

  @inproceedings{KiHeKo2014-SOSP-LIMBO, abstract = {{LIMBO is a tool for the creation of load profiles with variable intensity over time both from scratch and from existing data. Primarily, LIMBO's intended use is the description of load arrival behaviours in open workloads. Specifically, LIMBO can be employed for the creation of custom request or user arrival time-stamps or for the re-parameterization of existing traces. LIMBO bases on the Descartes Load Intensity Model (DLIM) for the formalized description of its load intensity profiles. The DLIM formalism can be understood as a structure for piece-wise defined and combined mathematical functions. We outline DLIM and its elements and demonstrate its integration within LIMBO. LIMBO is capable of generating request or user arrival time stamps from DLIM instances. In a next step, these generated time-stamps can be used for both open workload based benchmarking and simulations. The TimestampTimer plug-in for JMeter already allows the former. LIMBO also offers a range of tools for easy load intensity modeling and analysis, including, but not limited to, a visual decomposition of load intensity time-series into seasonal and trend parts, a simplified load intensity model as part of a model creation wizard, and an automated model instance extractor. As part of our LIMBO tutorial, we explain these features in detail. We demonstrate common use cases for LIMBO and show how they are supported. We also focus on LIMBOs extensible architecture and discuss how to use LIMBO as part of another tool-chain.}}, author = {von Kistowski, J{\'o}akim and Herbst, Nikolas Roman and Kounev, Samuel}, booktitle = {Proceedings of the Symposium on Software Performance 2014}, keywords = {descartes}, month = {November}, note = {<b>Best Poster Award</b>}, pages = {131–140}, publisher = {University of Stuttgart, Faculty of Computer Science, Electrical Engineering, and Information Technology}, title = {{Using and Extending LIMBO for the Descriptive Modeling of Arrival Behaviors}}, year = 2014 }

  %0 Conference Paper %1 KiHeKo2014-SOSP-LIMBO %A von Kistowski, Jóakim %A Herbst, Nikolas Roman %A Kounev, Samuel %B Proceedings of the Symposium on Software Performance 2014 %D 2014 %I University of Stuttgart, Faculty of Computer Science, Electrical Engineering, and Information Technology %P 131--140 %T Using and Extending LIMBO for the Descriptive Modeling of Arrival Behaviors %X LIMBO is a tool for the creation of load profiles with variable intensity over time both from scratch and from existing data. Primarily, LIMBO's intended use is the description of load arrival behaviours in open workloads. Specifically, LIMBO can be employed for the creation of custom request or user arrival time-stamps or for the re-parameterization of existing traces. LIMBO bases on the Descartes Load Intensity Model (DLIM) for the formalized description of its load intensity profiles. The DLIM formalism can be understood as a structure for piece-wise defined and combined mathematical functions. We outline DLIM and its elements and demonstrate its integration within LIMBO. LIMBO is capable of generating request or user arrival time stamps from DLIM instances. In a next step, these generated time-stamps can be used for both open workload based benchmarking and simulations. The TimestampTimer plug-in for JMeter already allows the former. LIMBO also offers a range of tools for easy load intensity modeling and analysis, including, but not limited to, a visual decomposition of load intensity time-series into seasonal and trend parts, a simplified load intensity model as part of a model creation wizard, and an automated model instance extractor. As part of our LIMBO tutorial, we explain these features in detail. We demonstrate common use cases for LIMBO and show how they are supported. We also focus on LIMBOs extensible architecture and discuss how to use LIMBO as part of another tool-chain.
• 
  Optimization Method for Request Admission Control to Guarantee Performance Isolation Krebs, Rouven; Schneider, Philipp; Herbst, Nikolas; in Proceedings of the 2nd International Workshop on Hot Topics in Cloud Service Scalability (HotTopiCS 2014), co-located with the 5th ACM/SPEC International Conference on Performance Engineering (ICPE 2014) (2014).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Software-as-a-Service (SaaS) often shares one single application instance among different tenants to reduce costs. However, sharing potentially leads to undesired influence from one tenant onto the performance observed by the others. Furthermore, providing one tenant additional resources to support its increasing demands without increasing the performance of tenants who do not pay for it is a major challenge. The application intentionally does not manage hardware resources, and the OS is not aware of application level entities like tenants. Thus, it is difficult to control the performance of different tenants to keep them isolated. These problems gain importance as performance is one of the major obstacles for cloud customers. Existing work applies request based admission control mechanisms like a weighted round robin with an individual queue for each tenant to control the share guaranteed for a tenant. However, the computation of the concrete weights for such an admission control is still challenging. In this paper, we present a fitness function and optimization approach reflecting various requirements from this field to compute proper weights with the goal to ensure an isolated performance as foundation to scale on a tenants basis.

  @inproceedings{KrScHe2014-HotTopiCS-OptimizationApproach, abstract = {{Software-as-a-Service (SaaS) often shares one single application instance among different tenants to reduce costs. However, sharing potentially leads to undesired influence from one tenant onto the performance observed by the others. Furthermore, providing one tenant additional resources to support its increasing demands without increasing the performance of tenants who do not pay for it is a major challenge. The application intentionally does not manage hardware resources, and the OS is not aware of application level entities like tenants. Thus, it is difficult to control the performance of different tenants to keep them isolated. These problems gain importance as performance is one of the major obstacles for cloud customers. Existing work applies request based admission control mechanisms like a weighted round robin with an individual queue for each tenant to control the share guaranteed for a tenant. However, the computation of the concrete weights for such an admission control is still challenging. In this paper, we present a fitness function and optimization approach reflecting various requirements from this field to compute proper weights with the goal to ensure an isolated performance as foundation to scale on a tenants basis.}}, author = {Krebs, Rouven and Schneider, Philipp and Herbst, Nikolas}, booktitle = {Proceedings of the 2nd International Workshop on Hot Topics in Cloud Service Scalability (HotTopiCS 2014), co-located with the 5th ACM/SPEC International Conference on Performance Engineering (ICPE 2014)}, keywords = {myown}, month = {March}, publisher = {ACM}, title = {{Optimization Method for Request Admission Control to Guarantee Performance Isolation}}, year = 2014 }

  %0 Conference Paper %1 KrScHe2014-HotTopiCS-OptimizationApproach %A Krebs, Rouven %A Schneider, Philipp %A Herbst, Nikolas %B Proceedings of the 2nd International Workshop on Hot Topics in Cloud Service Scalability (HotTopiCS 2014), co-located with the 5th ACM/SPEC International Conference on Performance Engineering (ICPE 2014) %D 2014 %I ACM %T Optimization Method for Request Admission Control to Guarantee Performance Isolation %X Software-as-a-Service (SaaS) often shares one single application instance among different tenants to reduce costs. However, sharing potentially leads to undesired influence from one tenant onto the performance observed by the others. Furthermore, providing one tenant additional resources to support its increasing demands without increasing the performance of tenants who do not pay for it is a major challenge. The application intentionally does not manage hardware resources, and the OS is not aware of application level entities like tenants. Thus, it is difficult to control the performance of different tenants to keep them isolated. These problems gain importance as performance is one of the major obstacles for cloud customers. Existing work applies request based admission control mechanisms like a weighted round robin with an individual queue for each tenant to control the share guaranteed for a tenant. However, the computation of the concrete weights for such an admission control is still challenging. In this paper, we present a fitness function and optimization approach reflecting various requirements from this field to compute proper weights with the goal to ensure an isolated performance as foundation to scale on a tenants basis.
• 
  LIMBO: A Tool For Modeling Variable Load Intensities von Kistowski, Jóakim; Herbst, Nikolas Roman; Kounev, Samuel; in Proceedings of the 5th ACM/SPEC International Conference on Performance Engineering (ICPE 2014) (2014). 225–226.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Modern software systems are expected to deliver reliable performance under highly variable load intensities while at the same time making efficient use of dynamically allocated resources. Conventional benchmarking frameworks provide limited support for emulating such highly variable and dynamic load profiles and workload scenarios. Industrial benchmarks typically use workloads with constant or stepwise increasing load intensity, or they simply replay recorded workload traces. In this paper, we present LIMBO - an Eclipse-based tool for modeling variable load intensity profiles based on the Descartes Load Intensity Model as an underlying modeling formalism.

  @inproceedings{KiHeKo2014-ICPEDemo-LIMBO, abstract = {{Modern software systems are expected to deliver reliable performance under highly variable load intensities while at the same time making efficient use of dynamically allocated resources. Conventional benchmarking frameworks provide limited support for emulating such highly variable and dynamic load profiles and workload scenarios. Industrial benchmarks typically use workloads with constant or stepwise increasing load intensity, or they simply replay recorded workload traces. In this paper, we present LIMBO - an Eclipse-based tool for modeling variable load intensity profiles based on the Descartes Load Intensity Model as an underlying modeling formalism.}}, address = {New York, NY, USA}, author = {von Kistowski, J{\'o}akim and Herbst, Nikolas Roman and Kounev, Samuel}, booktitle = {Proceedings of the 5th ACM/SPEC International Conference on Performance Engineering (ICPE 2014)}, keywords = {descartes}, month = {March}, pages = {225–226}, publisher = {ACM}, series = {ICPE '14}, title = {{LIMBO: A Tool For Modeling Variable Load Intensities}}, year = 2014 }

  %0 Conference Paper %1 KiHeKo2014-ICPEDemo-LIMBO %A von Kistowski, Jóakim %A Herbst, Nikolas Roman %A Kounev, Samuel %B Proceedings of the 5th ACM/SPEC International Conference on Performance Engineering (ICPE 2014) %C New York, NY, USA %D 2014 %I ACM %P 225--226 %T LIMBO: A Tool For Modeling Variable Load Intensities %U http://doi.acm.org/10.1145/2568088.2576092 %X Modern software systems are expected to deliver reliable performance under highly variable load intensities while at the same time making efficient use of dynamically allocated resources. Conventional benchmarking frameworks provide limited support for emulating such highly variable and dynamic load profiles and workload scenarios. Industrial benchmarks typically use workloads with constant or stepwise increasing load intensity, or they simply replay recorded workload traces. In this paper, we present LIMBO - an Eclipse-based tool for modeling variable load intensity profiles based on the Descartes Load Intensity Model as an underlying modeling formalism.
• 
  Modeling Variations in Load Intensity over Time von Kistowski, Jóakim; Herbst, Nikolas Roman; Kounev, Samuel; in Proceedings of the 3rd International Workshop on Large-Scale Testing (LT 2014), co-located with the 5th ACM/SPEC International Conference on Performance Engineering (ICPE 2014) (2014). 1–4.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Today's software systems are expected to deliver reliable performance under highly variable load intensities while at the same time making efficient use of dynamically allocated resources. Conventional benchmarking frameworks provide limited support for emulating such highly variable and dynamic load profiles and workload scenarios. Industrial benchmarks typically use workloads with constant or stepwise increasing load intensity, or they simply replay recorded workload traces. Based on this observation, we identify the need for means allowing flexible definition of load profiles and address this by introducing two meta-models at different abstraction levels. At the lower abstraction level, the Descartes Load Intensity Meta-Model (DLIM) offers a structured and accessible way of describing the load intensity over time by editing and combining mathematical functions. The High-Level Descartes Load Intensity Meta-Model (HLDLIM) allows the description of load variations using few defined parameters that characterize the seasonal patterns, trends, bursts and noise parts. We demonstrate that both meta-models are capable of capturing real-world load profiles with acceptable accuracy through comparison with a real life trace.

  @inproceedings{KiHeKo2014-LT-DLIM, abstract = {{Today's software systems are expected to deliver reliable performance under highly variable load intensities while at the same time making efficient use of dynamically allocated resources. Conventional benchmarking frameworks provide limited support for emulating such highly variable and dynamic load profiles and workload scenarios. Industrial benchmarks typically use workloads with constant or stepwise increasing load intensity, or they simply replay recorded workload traces. Based on this observation, we identify the need for means allowing flexible definition of load profiles and address this by introducing two meta-models at different abstraction levels. At the lower abstraction level, the Descartes Load Intensity Meta-Model (DLIM) offers a structured and accessible way of describing the load intensity over time by editing and combining mathematical functions. The High-Level Descartes Load Intensity Meta-Model (HLDLIM) allows the description of load variations using few defined parameters that characterize the seasonal patterns, trends, bursts and noise parts. We demonstrate that both meta-models are capable of capturing real-world load profiles with acceptable accuracy through comparison with a real life trace.}}, address = {New York, NY, USA}, author = {von Kistowski, J{\'o}akim and Herbst, Nikolas Roman and Kounev, Samuel}, booktitle = {Proceedings of the 3rd International Workshop on Large-Scale Testing (LT 2014), co-located with the 5th ACM/SPEC International Conference on Performance Engineering (ICPE 2014)}, keywords = {myown}, month = {March}, pages = {1–4}, publisher = {ACM}, title = {{Modeling Variations in Load Intensity over Time}}, year = 2014 }

  %0 Conference Paper %1 KiHeKo2014-LT-DLIM %A von Kistowski, Jóakim %A Herbst, Nikolas Roman %A Kounev, Samuel %B Proceedings of the 3rd International Workshop on Large-Scale Testing (LT 2014), co-located with the 5th ACM/SPEC International Conference on Performance Engineering (ICPE 2014) %C New York, NY, USA %D 2014 %I ACM %P 1--4 %T Modeling Variations in Load Intensity over Time %U http://doi.acm.org/10.1145/2577036.2577037 %X Today's software systems are expected to deliver reliable performance under highly variable load intensities while at the same time making efficient use of dynamically allocated resources. Conventional benchmarking frameworks provide limited support for emulating such highly variable and dynamic load profiles and workload scenarios. Industrial benchmarks typically use workloads with constant or stepwise increasing load intensity, or they simply replay recorded workload traces. Based on this observation, we identify the need for means allowing flexible definition of load profiles and address this by introducing two meta-models at different abstraction levels. At the lower abstraction level, the Descartes Load Intensity Meta-Model (DLIM) offers a structured and accessible way of describing the load intensity over time by editing and combining mathematical functions. The High-Level Descartes Load Intensity Meta-Model (HLDLIM) allows the description of load variations using few defined parameters that characterize the seasonal patterns, trends, bursts and noise parts. We demonstrate that both meta-models are capable of capturing real-world load profiles with acceptable accuracy through comparison with a real life trace.
• 
  Elasticity in Cloud Computing: What it is, and What it is Not Herbst, Nikolas Roman; Kounev, Samuel; Reussner, Ralf; in Proceedings of the 10th International Conference on Autonomic Computing (ICAC 2013) (2013).

  Acceptance Rate (Short Paper): 36.9%
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Originating from the field of physics and economics, the term elasticity is nowadays heavily used in the context of cloud computing. In this context, elasticity is commonly understood as the ability of a system to automatically provision and de-provision computing resources on demand as workloads change. However, elasticity still lacks a precise definition as well as representative metrics coupled with a benchmarking methodology to enable comparability of systems. Existing definitions of elasticity are largely inconsistent and unspecific leading to confusion in the use of the term and its differentiation from related terms such as scalability and efficiency; the proposed measurement methodologies do not provide means to quantify elasticity without mixing it with efficiency or scalability aspects. In this short paper, we propose a precise definition of elasticity and analyze its core properties and requirements explicitly distinguishing from related terms such as scalability, efficiency, and agility. Furthermore, we present a set of appropriate elasticity metrics and sketch a new elasticity tailored benchmarking methodology addressing the special requirements on workload design and calibration.

  @inproceedings{HeKoRe2013-ICAC-Elasticity, abstract = {{Originating from the field of physics and economics, the term elasticity is nowadays heavily used in the context of cloud computing. In this context, elasticity is commonly understood as the ability of a system to automatically provision and de-provision computing resources on demand as workloads change. However, elasticity still lacks a precise definition as well as representative metrics coupled with a benchmarking methodology to enable comparability of systems. Existing definitions of elasticity are largely inconsistent and unspecific leading to confusion in the use of the term and its differentiation from related terms such as scalability and efficiency; the proposed measurement methodologies do not provide means to quantify elasticity without mixing it with efficiency or scalability aspects. In this short paper, we propose a precise definition of elasticity and analyze its core properties and requirements explicitly distinguishing from related terms such as scalability, efficiency, and agility. Furthermore, we present a set of appropriate elasticity metrics and sketch a new elasticity tailored benchmarking methodology addressing the special requirements on workload design and calibration.}}, author = {Herbst, Nikolas Roman and Kounev, Samuel and Reussner, Ralf}, booktitle = {Proceedings of the 10th International Conference on Autonomic Computing (ICAC 2013)}, keywords = {t_short}, month = {June}, note = {Acceptance Rate (Short Paper): 36.9\%}, publisher = {USENIX}, title = {{Elasticity in Cloud Computing: What it is, and What it is Not}}, year = 2013 }

  %0 Conference Paper %1 HeKoRe2013-ICAC-Elasticity %A Herbst, Nikolas Roman %A Kounev, Samuel %A Reussner, Ralf %B Proceedings of the 10th International Conference on Autonomic Computing (ICAC 2013) %D 2013 %I USENIX %T Elasticity in Cloud Computing: What it is, and What it is Not %U https://www.usenix.org/conference/icac13/elasticity-cloud-computing-what-it-and-what-it-not %X Originating from the field of physics and economics, the term elasticity is nowadays heavily used in the context of cloud computing. In this context, elasticity is commonly understood as the ability of a system to automatically provision and de-provision computing resources on demand as workloads change. However, elasticity still lacks a precise definition as well as representative metrics coupled with a benchmarking methodology to enable comparability of systems. Existing definitions of elasticity are largely inconsistent and unspecific leading to confusion in the use of the term and its differentiation from related terms such as scalability and efficiency; the proposed measurement methodologies do not provide means to quantify elasticity without mixing it with efficiency or scalability aspects. In this short paper, we propose a precise definition of elasticity and analyze its core properties and requirements explicitly distinguishing from related terms such as scalability, efficiency, and agility. Furthermore, we present a set of appropriate elasticity metrics and sketch a new elasticity tailored benchmarking methodology addressing the special requirements on workload design and calibration.

• 
  SPEC Research Group Newsletter, vol. 3 no. 1 Kounev, Samuel; van Hoorn, Andre; Bauer, André; Herbst, Nikolas; (2020, April).

  Published by Standard Performance Evaluation Corporation (SPEC)
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  @misc{kounev2020research, author = {Kounev, Samuel and van Hoorn, Andre and Bauer, Andr{\'e} and Herbst, Nikolas}, keywords = {myown}, month = {April}, note = {Published by Standard Performance Evaluation Corporation (SPEC)}, title = {{SPEC Research Group Newsletter, vol. 3 no. 1}}, year = 2020 }

  %0 Generic %1 kounev2020research %A Kounev, Samuel %A van Hoorn, Andre %A Bauer, André %A Herbst, Nikolas %D 2020 %T SPEC Research Group Newsletter, vol. 3 no. 1 %U http://research.spec.org/en/newsletter.html
• 
  SPEC Research Group Newsletter, vol. 2 no. 4 Kounev, Samuel; van Hoorn, Andre; Bauer, André; Herbst, Nikolas; (2019, April).

  Published by Standard Performance Evaluation Corporation (SPEC)
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  @misc{SPEC-RG-NEWSLETTER-8, author = {Kounev, Samuel and van Hoorn, Andre and Bauer, Andr{\'e} and Herbst, Nikolas}, keywords = {myown}, month = {April}, note = {Published by Standard Performance Evaluation Corporation (SPEC)}, title = {{SPEC Research Group Newsletter, vol. 2 no. 4}}, year = 2019 }

  %0 Generic %1 SPEC-RG-NEWSLETTER-8 %A Kounev, Samuel %A van Hoorn, Andre %A Bauer, André %A Herbst, Nikolas %D 2019 %T SPEC Research Group Newsletter, vol. 2 no. 4 %U http://research.spec.org/en/newsletter.html
• 
  Methoden und Messverfahren für Automatisches Skalieren in Elastischen Cloud Umgebungen Herbst, Nikolas; (2019, June).
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @misc{talk-Herbst-GIAwardNomination-Dagstuhl, author = {Herbst, Nikolas}, howpublished = {GI Dissertationspreis: Seminar f{\"u}r Nominierte Doktoranden, Dagstuhl, Germany, June 28, 2019}, keywords = {myown}, month = {June}, title = {{Methoden und Messverfahren f{\"u}r Automatisches Skalieren in Elastischen Cloud Umgebungen}}, year = 2019 }

  %0 Generic %1 talk-Herbst-GIAwardNomination-Dagstuhl %A Herbst, Nikolas %D 2019 %T Methoden und Messverfahren für Automatisches Skalieren in Elastischen Cloud Umgebungen
• 
  SPEC Research Group Newsletter, vol. 2 no. 3 Kounev, Samuel; van Hoorn, Andre; Herbst, Nikolas; Bauer, André; (2018, April).

  Published by Standard Performance Evaluation Corporation (SPEC)
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  @misc{SPEC-RG-NEWSLETTER-7, author = {Kounev, Samuel and van Hoorn, Andre and Herbst, Nikolas and Bauer, Andr{\'e}}, keywords = {myown}, month = {April}, note = {Published by Standard Performance Evaluation Corporation (SPEC)}, title = {{SPEC Research Group Newsletter, vol. 2 no. 3}}, year = 2018 }

  %0 Generic %1 SPEC-RG-NEWSLETTER-7 %A Kounev, Samuel %A van Hoorn, Andre %A Herbst, Nikolas %A Bauer, André %D 2018 %T SPEC Research Group Newsletter, vol. 2 no. 3 %U http://research.spec.org/en/newsletter.html
• 
  How we built a scalable micro-service application - The Tea Store Herbst, Nikolas; (2018, July).
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  @misc{talk-Herbst-ScrumScale, author = {Herbst, Nikolas}, howpublished = {Keynote Talk at the ScrumScale workshop, Oslo, Norway, July 5, 2018}, keywords = {myown}, month = {July}, title = {{How we built a scalable micro-service application - The Tea Store}}, year = 2018 }

  %0 Generic %1 talk-Herbst-ScrumScale %A Herbst, Nikolas %D 2018 %T How we built a scalable micro-service application - The Tea Store %U https://www.dataforeningen.no/smidig-skalerbarhet.6113996-134138.html
• 
  SPEC Research Group Newsletter, vol. 2 no. 2 Kounev, Samuel; van Hoorn, Andre; Rygielski, Piotr; Herbst, Nikolas; Bauer, André; (2017, April).

  Published by Standard Performance Evaluation Corporation (SPEC)
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  @misc{SPEC-RG-NEWSLETTER-6, author = {Kounev, Samuel and van Hoorn, Andre and Rygielski, Piotr and Herbst, Nikolas and Bauer, Andr{\'e}}, keywords = {myown}, month = {April}, note = {Published by Standard Performance Evaluation Corporation (SPEC)}, title = {{SPEC Research Group Newsletter, vol. 2 no. 2}}, year = 2017 }

  %0 Generic %1 SPEC-RG-NEWSLETTER-6 %A Kounev, Samuel %A van Hoorn, Andre %A Rygielski, Piotr %A Herbst, Nikolas %A Bauer, André %D 2017 %T SPEC Research Group Newsletter, vol. 2 no. 2 %U http://research.spec.org/en/newsletter.html

• 
  Methods and Benchmarks for Auto-Scaling Mechanisms in Elastic Cloud Environments Herbst, Nikolas; Thesis; University of Würzburg, Germany. (2018, July).

  Distinction, SPEC Kaivalya Dixit Distinguished Dissertation Award 2018
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  @phdthesis{Herbst2018-phd, author = {Herbst, Nikolas}, keywords = {myown}, month = {July}, note = {<b>{Distinction, SPEC Kaivalya Dixit Distinguished Dissertation Award 2018}</b>}, school = {University of W{\"u}rzburg, Germany}, title = {Methods and Benchmarks for Auto-Scaling Mechanisms in Elastic Cloud Environments}, year = 2018 }

  %0 Thesis %1 Herbst2018-phd %A Herbst, Nikolas %D 2018 %T Methods and Benchmarks for Auto-Scaling Mechanisms in Elastic Cloud Environments %U https://opus.bibliothek.uni-wuerzburg.de/frontdoor/index/index/docId/16431

• 
  ICPE ’22: ACM/SPEC International Conference on Performance Engineering, Bejing, China, April 9 - 13, 2022 Feng, Dan; Becker, Steffen; Herbst, Nikolas; Leitner, Philipp; (D. Feng; S. Becker; N. Herbst; P. Leitner, eds.) (2022). ACM.
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  @proceedings{DBLP:conf/wosp/2022, author = {Feng, Dan and Becker, Steffen and Herbst, Nikolas and Leitner, Philipp}, editor = {Feng, Dan and Becker, Steffen and Herbst, Nikolas and Leitner, Philipp}, keywords = {myown}, publisher = {{ACM}}, title = {{ICPE} '22: {ACM/SPEC} International Conference on Performance Engineering, Bejing, China, April 9 - 13, 2022}, year = 2022 }

  %0 Conference Proceedings %1 DBLP:conf/wosp/2022 %A Feng, Dan %A Becker, Steffen %A Herbst, Nikolas %A Leitner, Philipp %D 2022 %E Feng, Dan %E Becker, Steffen %E Herbst, Nikolas %E Leitner, Philipp %I ACM %T ICPE '22: ACM/SPEC International Conference on Performance Engineering, Bejing, China, April 9 - 13, 2022 %U https://doi.org/10.1145/3489525
• 
  ICPE ’22: ACM/SPEC International Conference on Performance Engineering, Bejing, China, April 9 - 13, 2022, Companion Volume Feng, Dan; Becker, Steffen; Herbst, Nikolas; Leitner, Philipp; (D. Feng; S. Becker; N. Herbst; P. Leitner, eds.) (2022). ACM.
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  @proceedings{DBLP:conf/wosp/2022c, author = {Feng, Dan and Becker, Steffen and Herbst, Nikolas and Leitner, Philipp}, editor = {Feng, Dan and Becker, Steffen and Herbst, Nikolas and Leitner, Philipp}, keywords = {myown}, publisher = {{ACM}}, title = {{ICPE} '22: {ACM/SPEC} International Conference on Performance Engineering, Bejing, China, April 9 - 13, 2022, Companion Volume}, year = 2022 }

  %0 Conference Proceedings %1 DBLP:conf/wosp/2022c %A Feng, Dan %A Becker, Steffen %A Herbst, Nikolas %A Leitner, Philipp %D 2022 %E Feng, Dan %E Becker, Steffen %E Herbst, Nikolas %E Leitner, Philipp %I ACM %T ICPE '22: ACM/SPEC International Conference on Performance Engineering, Bejing, China, April 9 - 13, 2022, Companion Volume %U https://doi.org/10.1145/3491204

• 
  A Review of Serverless Use Cases and their Characteristics Eismann, Simon; Scheuner, Joel; van Eyk, Erwin; Schwinger, Maximilian; Grohmann, Johannes; Herbst, Nikolas; Abad, Cristina; Iosup, Alexandru; (2020).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  The serverless computing paradigm promises many desirable properties for cloud applications - low-cost, fine-grained deployment, and management-free operation. Consequently, the paradigm has underwent rapid growth: there currently exist tens of serverless platforms and all global cloud providers host serverless operations. To help tune existing platforms, guide the design of new serverless approaches, and overall contribute to understanding this paradigm, in this work we present a long-term, comprehensive effort to identify, collect, and characterize serverless use cases. We survey 89 use cases, sourced from white and grey literature, and from consultations with experts in areas such as scientific computing. We study each use case using 24 characteristics, including general aspects, but also workload, application, and requirements. When the use cases employ workflows, we further analyze their characteristics. Overall, we hope our study will be useful for both academia and industry, and encourage the community to further share and communicate their use cases.

  @techreport{eismann2020review, abstract = {The serverless computing paradigm promises many desirable properties for cloud applications - low-cost, fine-grained deployment, and management-free operation. Consequently, the paradigm has underwent rapid growth: there currently exist tens of serverless platforms and all global cloud providers host serverless operations. To help tune existing platforms, guide the design of new serverless approaches, and overall contribute to understanding this paradigm, in this work we present a long-term, comprehensive effort to identify, collect, and characterize serverless use cases. We survey 89 use cases, sourced from white and grey literature, and from consultations with experts in areas such as scientific computing. We study each use case using 24 characteristics, including general aspects, but also workload, application, and requirements. When the use cases employ workflows, we further analyze their characteristics. Overall, we hope our study will be useful for both academia and industry, and encourage the community to further share and communicate their use cases.}, author = {Eismann, Simon and Scheuner, Joel and van Eyk, Erwin and Schwinger, Maximilian and Grohmann, Johannes and Herbst, Nikolas and Abad, Cristina and Iosup, Alexandru}, keywords = {t_techreport}, month = {June}, publisher = {SPEC RG}, title = {A Review of Serverless Use Cases and their Characteristics}, year = 2020 }

  %0 Report %1 eismann2020review %A Eismann, Simon %A Scheuner, Joel %A van Eyk, Erwin %A Schwinger, Maximilian %A Grohmann, Johannes %A Herbst, Nikolas %A Abad, Cristina %A Iosup, Alexandru %D 2020 %I SPEC RG %T A Review of Serverless Use Cases and their Characteristics %U https://arxiv.org/abs/2008.11110 %X The serverless computing paradigm promises many desirable properties for cloud applications - low-cost, fine-grained deployment, and management-free operation. Consequently, the paradigm has underwent rapid growth: there currently exist tens of serverless platforms and all global cloud providers host serverless operations. To help tune existing platforms, guide the design of new serverless approaches, and overall contribute to understanding this paradigm, in this work we present a long-term, comprehensive effort to identify, collect, and characterize serverless use cases. We survey 89 use cases, sourced from white and grey literature, and from consultations with experts in areas such as scientific computing. We study each use case using 24 characteristics, including general aspects, but also workload, application, and requirements. When the use cases employ workflows, we further analyze their characteristics. Overall, we hope our study will be useful for both academia and industry, and encourage the community to further share and communicate their use cases.
• 
  Methodological Principles for Reproducible Performance Evaluation in Cloud Computing - A SPEC Research Technical Report Papadopoulos, Alessandro Vittorio; Versluis, Laurens; Bauer, André; Herbst, Nikolas; von Kistowski, Jóakim; Ali-Eldin, Ahmed; Abad, Cristina; Amaral, J. Nelson; Tuma, Petr; Iosup, Alexandru; (2019).
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  @techreport{PaVeBaHeKiEAAbAmTuIo2019-SPEC-CloudPerformance, author = {Papadopoulos, Alessandro Vittorio and Versluis, Laurens and Bauer, Andr{\'e} and Herbst, Nikolas and von Kistowski, J{\'o}akim and Ali-Eldin, Ahmed and Abad, Cristina and Amaral, J. Nelson and Tuma, Petr and Iosup, Alexandru}, institution = {SPEC Research Group — Cloud Working Group, Standard Performance Evaluation Corporation (SPEC)}, keywords = {myown}, month = {April}, number = {SPEC-RG-2019-04}, title = {{Methodological Principles for Reproducible Performance Evaluation in Cloud Computing - A SPEC Research Technical Report}}, year = 2019 }

  %0 Report %1 PaVeBaHeKiEAAbAmTuIo2019-SPEC-CloudPerformance %A Papadopoulos, Alessandro Vittorio %A Versluis, Laurens %A Bauer, André %A Herbst, Nikolas %A von Kistowski, Jóakim %A Ali-Eldin, Ahmed %A Abad, Cristina %A Amaral, J. Nelson %A Tuma, Petr %A Iosup, Alexandru %D 2019 %N SPEC-RG-2019-04 %T Methodological Principles for Reproducible Performance Evaluation in Cloud Computing - A SPEC Research Technical Report %U https://research.spec.org/fileadmin/user_upload/documents/rg_cloud/endorsed_publications/SPEC_RG_2019_Methodological_Principles_for_Reproducible_Performance_Evaluation_in_Cloud_Computing.pdf
• 
  Ready for Rain? A View from SPEC Research on the Future of Cloud Metrics Herbst, Nikolas; Krebs, Rouven; Oikonomou, Giorgos; Kousiouris, George; Evangelinou, Athanasia; Iosup, Alexandru; Kounev, Samuel; (2016).
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  @techreport{SPEC-RG-2016-01-CloudMetrics, author = {Herbst, Nikolas and Krebs, Rouven and Oikonomou, Giorgos and Kousiouris, George and Evangelinou, Athanasia and Iosup, Alexandru and Kounev, Samuel}, institution = {SPEC Research Group — Cloud Working Group, Standard Performance Evaluation Corporation (SPEC)}, keywords = {myown}, number = {SPEC-RG-2016-01}, title = {{Ready for Rain? A View from SPEC Research on the Future of Cloud Metrics}}, year = 2016 }

  %0 Report %1 SPEC-RG-2016-01-CloudMetrics %A Herbst, Nikolas %A Krebs, Rouven %A Oikonomou, Giorgos %A Kousiouris, George %A Evangelinou, Athanasia %A Iosup, Alexandru %A Kounev, Samuel %D 2016 %N SPEC-RG-2016-01 %T Ready for Rain? A View from SPEC Research on the Future of Cloud Metrics %U https://research.spec.org/fileadmin/user_upload/documents/rg_cloud/endorsed_publications/SPEC-RG-2016-01_CloudMetrics.pdf
• 
  Performance-oriented DevOps: A Research Agenda Brunnert, Andreas; van Hoorn, Andre; Willnecker, Felix; Danciu, Alexandru; Hasselbring, Wilhelm; Heger, Christoph; Herbst, Nikolas; Jamshidi, Pooyan; Jung, Reiner; von Kistowski, Joakim; Koziolek, Anne; Kross, Johannes; Spinner, Simon; Vögele, Christian; Walter, Jürgen; Wert, Alexander; (2015).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  DevOps is a trend towards a tighter integration between development (Dev) and operations (Ops) teams. The need for such an integration is driven by the requirement to continuously adapt enterprise applications (EAs) to changes in the business environment. As of today, DevOps concepts have been primarily introduced to ensure a constant flow of features and bug fixes into new releases from a functional perspective. In order to integrate a non-functional perspective into these DevOps concepts this report focuses on tools, activities, and processes to ensure one of the most important quality attributes of a software system, namely performance. Performance describes system properties concerning its timeliness and use of resources. Common metrics are response time, throughput, and resource utilization. Performance goals for EAs are typically defined by setting upper and/or lower bounds for these metrics and specific business transactions. In order to ensure that such performance goals can be met, several activities are required during development and operation of these systems as well as during the transition from Dev to Ops. Activities during development are typically summarized by the term Software Performance Engineering (SPE), whereas activities during operations are called Application Performance Management (APM). SPE and APM were historically tackled independently from each other, but the newly emerging DevOps concepts require and enable a tighter integration between both activity streams. This report presents existing solutions to support this integration as well as open research challenges in this area. The report starts by defining EAs and summarizes their characteristics that make performance evaluations for these systems particularly challenging. It continues by describing our understanding of DevOps and explaining the roots of this trend to set the context for the remaining parts of the report. Afterwards, performance management activities that are common in both life cycle phases are explained, until the particularities of SPE and APM are discussed in separate sections. Finally, the report concludes by outlining activities and challenges to support the rapid iteration between Dev and Ops.

  @techreport{SPEC-RG-2015-001-DevOpsPerformanceResearchAgenda, abstract = {DevOps is a trend towards a tighter integration between development (Dev) and operations (Ops) teams. The need for such an integration is driven by the requirement to continuously adapt enterprise applications (EAs) to changes in the business environment. As of today, DevOps concepts have been primarily introduced to ensure a constant flow of features and bug fixes into new releases from a functional perspective. In order to integrate a non-functional perspective into these DevOps concepts this report focuses on tools, activities, and processes to ensure one of the most important quality attributes of a software system, namely performance. Performance describes system properties concerning its timeliness and use of resources. Common metrics are response time, throughput, and resource utilization. Performance goals for EAs are typically defined by setting upper and/or lower bounds for these metrics and specific business transactions. In order to ensure that such performance goals can be met, several activities are required during development and operation of these systems as well as during the transition from Dev to Ops. Activities during development are typically summarized by the term Software Performance Engineering (SPE), whereas activities during operations are called Application Performance Management (APM). SPE and APM were historically tackled independently from each other, but the newly emerging DevOps concepts require and enable a tighter integration between both activity streams. This report presents existing solutions to support this integration as well as open research challenges in this area. The report starts by defining EAs and summarizes their characteristics that make performance evaluations for these systems particularly challenging. It continues by describing our understanding of DevOps and explaining the roots of this trend to set the context for the remaining parts of the report. Afterwards, performance management activities that are common in both life cycle phases are explained, until the particularities of SPE and APM are discussed in separate sections. Finally, the report concludes by outlining activities and challenges to support the rapid iteration between Dev and Ops.}, author = {Brunnert, Andreas and van Hoorn, Andre and Willnecker, Felix and Danciu, Alexandru and Hasselbring, Wilhelm and Heger, Christoph and Herbst, Nikolas and Jamshidi, Pooyan and Jung, Reiner and von Kistowski, Joakim and Koziolek, Anne and Kross, Johannes and Spinner, Simon and V{\"o}gele, Christian and Walter, J{\"u}rgen and Wert, Alexander}, institution = {SPEC Research Group — DevOps Performance Working Group, Standard Performance Evaluation Corporation (SPEC)}, keywords = {myown}, month = {August}, number = {SPEC-RG-2015-01}, title = {Performance-oriented {DevOps}: {A} Research Agenda}, year = 2015 }

  %0 Report %1 SPEC-RG-2015-001-DevOpsPerformanceResearchAgenda %A Brunnert, Andreas %A van Hoorn, Andre %A Willnecker, Felix %A Danciu, Alexandru %A Hasselbring, Wilhelm %A Heger, Christoph %A Herbst, Nikolas %A Jamshidi, Pooyan %A Jung, Reiner %A von Kistowski, Joakim %A Koziolek, Anne %A Kross, Johannes %A Spinner, Simon %A Vögele, Christian %A Walter, Jürgen %A Wert, Alexander %D 2015 %N SPEC-RG-2015-01 %T Performance-oriented DevOps: A Research Agenda %U https://research.spec.org/fileadmin/user_upload/documents/wg_devops/endorsed_publications/SPEC-RG-2015-001-DevOpsPerformanceResearchAgenda.pdf %X DevOps is a trend towards a tighter integration between development (Dev) and operations (Ops) teams. The need for such an integration is driven by the requirement to continuously adapt enterprise applications (EAs) to changes in the business environment. As of today, DevOps concepts have been primarily introduced to ensure a constant flow of features and bug fixes into new releases from a functional perspective. In order to integrate a non-functional perspective into these DevOps concepts this report focuses on tools, activities, and processes to ensure one of the most important quality attributes of a software system, namely performance. Performance describes system properties concerning its timeliness and use of resources. Common metrics are response time, throughput, and resource utilization. Performance goals for EAs are typically defined by setting upper and/or lower bounds for these metrics and specific business transactions. In order to ensure that such performance goals can be met, several activities are required during development and operation of these systems as well as during the transition from Dev to Ops. Activities during development are typically summarized by the term Software Performance Engineering (SPE), whereas activities during operations are called Application Performance Management (APM). SPE and APM were historically tackled independently from each other, but the newly emerging DevOps concepts require and enable a tighter integration between both activity streams. This report presents existing solutions to support this integration as well as open research challenges in this area. The report starts by defining EAs and summarizes their characteristics that make performance evaluations for these systems particularly challenging. It continues by describing our understanding of DevOps and explaining the roots of this trend to set the context for the remaining parts of the report. Afterwards, performance management activities that are common in both life cycle phases are explained, until the particularities of SPE and APM are discussed in separate sections. Finally, the report concludes by outlining activities and challenges to support the rapid iteration between Dev and Ops.