BUNGEE

Update: This tool is no longer maintained and supported.

BUNGEE Cloud Elasticity Benchmark

BUNGEE is a Java based framework for benchmarking elasticity of IaaS cloud platforms. The tool automates the following benchmarking activities:

A system analysis evaluates the load processing capabilities of the benchmarked platform at different scaling stages.
The benchmark calibration uses the system analysis results and adjusts a given load intensity profile in a system specific manner.
The measurement activity exposes the platform to a varying load according to the adjusted intensity profile.
The elasticity evaluation measures the quality of the observed elastic behavior using a set of elasticity metrics.

At the moment, BUNGEE supports to analyse the elasticity of CloudStack and Amazon Web Service (AWS) based clouds that scale CPU-bound virtual machines horizontally.

Links:

For more information, please contact Nikolas Herbst

The BUNGEE presentation at SEAMS2015 is available here.

Mailing List

Publications

2019[ to top ]

Methoden und Messverfahren für Automatisches Skalieren in Elastischen Cloud Umgebungen. N. Herbst; (2019, Juni).
- [ BibTeX ]
- [ EndNote ]
@misc{talk-Herbst-GIAwardNomination-Dagstuhl, author = {Herbst, Nikolas}, howpublished = {GI Dissertationspreis: Seminar f{\"u}r Nominierte Doktoranden, Dagstuhl, Germany, June 28, 2019}, keywords = {t_bookchapter}, month = {06}, title = {{Methoden und Messverfahren f{\"u}r Automatisches Skalieren in Elastischen Cloud Umgebungen}}, year = 2019 }

%0 Generic %1 talk-Herbst-GIAwardNomination-Dagstuhl %A Herbst, Nikolas %D 2019 %T Methoden und Messverfahren für Automatisches Skalieren in Elastischen Cloud Umgebungen
Chameleon: A Hybrid, Proactive Auto-Scaling Mechanism on a Level-Playing Field. A. Bauer; (2019, Juni).
- [ BibTeX ]
- [ EndNote ]
@misc{talk-Bauer-HotCloudPerf2019, author = {Bauer, Andr{\'e}}, howpublished = {Talk at HotCloudperf 2019}, keywords = {Chameleon}, month = {06}, title = {{Chameleon: A Hybrid, Proactive Auto-Scaling Mechanism on a Level-Playing Field}}, year = 2019 }

%0 Generic %1 talk-Bauer-HotCloudPerf2019 %A Bauer, André %D 2019 %T Chameleon: A Hybrid, Proactive Auto-Scaling Mechanism on a Level-Playing Field
Chameleon: A Hybrid, Proactive Auto-Scaling Mechanism on a Level-Playing Field. A. Bauer; N. Herbst; S. Spinner; A. Ali-Eldin; S. Kounev; in IEEE Transactions on Parallel and Distributed Systems (2019). 30(4) 800–813.
- [ BibTeX ]
- [ EndNote ]
@article{BaHeSpAEKo2019-TPDS-ChameleonAutoscaler, author = {Bauer, Andr{\'e} and Herbst, Nikolas and Spinner, Simon and Ali-Eldin, Ahmed and Kounev, Samuel}, journal = {IEEE Transactions on Parallel and Distributed Systems}, keywords = {Self-adaptive-systems}, month = {04}, number = 4, pages = {800–813}, publisher = {IEEE}, title = {{Chameleon: A Hybrid, Proactive Auto-Scaling Mechanism on a Level-Playing Field}}, volume = 30, year = 2019 }

%0 Journal Article %1 BaHeSpAEKo2019-TPDS-ChameleonAutoscaler %A Bauer, André %A Herbst, Nikolas %A Spinner, Simon %A Ali-Eldin, Ahmed %A Kounev, Samuel %D 2019 %I IEEE %J IEEE Transactions on Parallel and Distributed Systems %N 4 %P 800--813 %T Chameleon: A Hybrid, Proactive Auto-Scaling Mechanism on a Level-Playing Field %V 30
Systematic Search for Optimal Resource Configurations of Distributed Applications. A. Bauer; S. Eismann; J. Grohmann; N. Herbst; S. Kounev; in 2019 IEEE 4th International Workshops on Foundations and Applications of Self* Systems (FAS*W) (2019). 120–125.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
- [ DOI ]
With the advent of the micro-service paradigm, applications are divided into small, distributed parts. Knowledge of optimal resource configurations of such applications is required both for autonomic resource management as well as its assessment. Due to the high-dimensional search space of all possible configurations, the systematic measuring of the optimal configurations is challenging. To this end, we introduce a search algorithm based on hill-climbing for finding all optimal configurations in a feasible time and integrate it in an existing measuring framework. This approach enables the assessment, comparison and optimization of autonomic resource management approaches for micro-service applications. The evaluation shows that our approach is able to find all optimal configurations in the considered scenarios, while state-of-the-art multi-objective search algorithms do not.

@inproceedings{BaEiGrHeKo-ICAC-BUNGEE, abstract = {With the advent of the micro-service paradigm, applications are divided into small, distributed parts. Knowledge of optimal resource configurations of such applications is required both for autonomic resource management as well as its assessment. Due to the high-dimensional search space of all possible configurations, the systematic measuring of the optimal configurations is challenging. To this end, we introduce a search algorithm based on hill-climbing for finding all optimal configurations in a feasible time and integrate it in an existing measuring framework. This approach enables the assessment, comparison and optimization of autonomic resource management approaches for micro-service applications. The evaluation shows that our approach is able to find all optimal configurations in the considered scenarios, while state-of-the-art multi-objective search algorithms do not.}, address = {Los Alamitos, CA, USA}, author = {Bauer, Andr{\'e} and Eismann, Simon and Grohmann, Johannes and Herbst, Nikolas and Kounev, Samuel}, booktitle = {2019 IEEE 4th International Workshops on Foundations and Applications of Self* Systems (FAS*W)}, keywords = {Optimization}, month = {06}, pages = {120–125}, publisher = {IEEE Computer Society}, title = {{Systematic Search for Optimal Resource Configurations of Distributed Applications}}, year = 2019 }

%0 Conference Paper %1 BaEiGrHeKo-ICAC-BUNGEE %A Bauer, André %A Eismann, Simon %A Grohmann, Johannes %A Herbst, Nikolas %A Kounev, Samuel %B 2019 IEEE 4th International Workshops on Foundations and Applications of Self* Systems (FAS*W) %C Los Alamitos, CA, USA %D 2019 %I IEEE Computer Society %P 120--125 %R 10.1109/FAS-W.2019.00040 %T Systematic Search for Optimal Resource Configurations of Distributed Applications %U https://doi.ieeecomputersociety.org/10.1109/FAS-W.2019.00040 %X With the advent of the micro-service paradigm, applications are divided into small, distributed parts. Knowledge of optimal resource configurations of such applications is required both for autonomic resource management as well as its assessment. Due to the high-dimensional search space of all possible configurations, the systematic measuring of the optimal configurations is challenging. To this end, we introduce a search algorithm based on hill-climbing for finding all optimal configurations in a feasible time and integrate it in an existing measuring framework. This approach enables the assessment, comparison and optimization of autonomic resource management approaches for micro-service applications. The evaluation shows that our approach is able to find all optimal configurations in the considered scenarios, while state-of-the-art multi-objective search algorithms do not.

2018[ to top ]

Integrating Docker into the BUNGEE cloud elasticity benchmark. F. Roos; Thesis; University of Würzburg; Am Hubland, Informatikgebäude, 97074 Würzburg, Germany. (2018, September).
- [ BibTeX ]
- [ EndNote ]
@mastersthesis{BA-Roos-2018, address = {Am Hubland, Informatikgeb{\"a}ude, 97074 W{\"u}rzburg, Germany}, author = {Roos, Filipp}, keywords = {BUNGEE}, month = {09}, school = {University of W{\"u}rzburg}, title = {{Integrating Docker into the BUNGEE cloud elasticity benchmark}}, type = {{Bachelor Thesis}}, year = 2018 }

%0 Thesis %1 BA-Roos-2018 %A Roos, Filipp %C Am Hubland, Informatikgebäude, 97074 Würzburg, Germany %D 2018 %T Integrating Docker into the BUNGEE cloud elasticity benchmark
Quantifying Cloud Performance and Dependability: Taxonomy, Metric Design, and Emerging Challenges. N. Herbst; A. Bauer; S. Kounev; G. Oikonomou; E. van Eyk; G. Kousiouris; A. Evangelinou; R. Krebs; T. Brecht; C. L. Abad; A. Iosup; in ACM Transactions on Modeling and Performance Evaluation of Computing Systems (ToMPECS) (2018). 3(4) 19:1–19:36.
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
@article{HeBaKoOiEyKoEvKrBrAbIo2018-TOMPECS-CloudMetrics, address = {New York, NY, USA}, author = {Herbst, Nikolas and Bauer, Andr{\'e} and Kounev, Samuel and Oikonomou, Giorgos and van Eyk, Erwin and Kousiouris, George and Evangelinou, Athanasia and Krebs, Rouven and Brecht, Tim and Abad, Cristina L. and Iosup, Alexandru}, journal = {ACM Transactions on Modeling and Performance Evaluation of Computing Systems (ToMPECS)}, keywords = {SPEC}, month = {08}, number = 4, pages = {19:1–19:36}, publisher = {ACM}, title = {{Quantifying Cloud Performance and Dependability: Taxonomy, Metric Design, and Emerging Challenges}}, volume = 3, year = 2018 }

%0 Journal Article %1 HeBaKoOiEyKoEvKrBrAbIo2018-TOMPECS-CloudMetrics %A Herbst, Nikolas %A Bauer, André %A Kounev, Samuel %A Oikonomou, Giorgos %A van Eyk, Erwin %A Kousiouris, George %A Evangelinou, Athanasia %A Krebs, Rouven %A Brecht, Tim %A Abad, Cristina L. %A Iosup, Alexandru %C New York, NY, USA %D 2018 %I ACM %J ACM Transactions on Modeling and Performance Evaluation of Computing Systems (ToMPECS) %N 4 %P 19:1--19:36 %T Quantifying Cloud Performance and Dependability: Taxonomy, Metric Design, and Emerging Challenges %U http://doi.acm.org/10.1145/3236332 %V 3
Methods and Benchmarks for Auto-Scaling Mechanisms in Elastic Cloud Environments. N. Herbst; Thesis; University of Würzburg, Germany. (2018, Juli).
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
@phdthesis{Herbst2018-phd, author = {Herbst, Nikolas}, keywords = {SPEC}, month = {07}, note = {{Distinction, SPEC Kaivalya Dixit Distinguished Dissertation Award 2018}}, school = {University of W{\"u}rzburg, Germany}, title = {Methods and Benchmarks for Auto-Scaling Mechanisms in Elastic Cloud Environments}, year = 2018 }

%0 Thesis %1 Herbst2018-phd %A Herbst, Nikolas %D 2018 %T Methods and Benchmarks for Auto-Scaling Mechanisms in Elastic Cloud Environments %U https://opus.bibliothek.uni-wuerzburg.de/frontdoor/index/index/docId/16431
Elasticity Measurement in CaaS Environments - Extending the Existing BUNGEE Elasticity Benchmark to AWS’s Elastic Container Service. N. Limbourg; Thesis; Dublin Institute of Technology; Kevin Street, Dublin 2, D08 X622, Ireland. (2018, Juni).
- [ BibTeX ]
- [ EndNote ]
@mastersthesis{Limbourg2018Elasticity, address = {Kevin Street, Dublin 2, D08 X622, Ireland}, author = {Limbourg, Nora}, keywords = {Thesis_supervised_by_Veronika_Lesch}, month = {06}, school = {Dublin Institute of Technology}, title = {{Elasticity Measurement in CaaS Environments - Extending the Existing BUNGEE Elasticity Benchmark to AWS's Elastic Container Service}}, type = {{Master Thesis}}, year = 2018 }

%0 Thesis %1 Limbourg2018Elasticity %A Limbourg, Nora %C Kevin Street, Dublin 2, D08 X622, Ireland %D 2018 %T Elasticity Measurement in CaaS Environments - Extending the Existing BUNGEE Elasticity Benchmark to AWS's Elastic Container Service
Extending BUNGEE Elasticity Benchmark for Multi-Tier Cloud Applications. A. Bauer; (2018, April).
- [ BibTeX ]
- [ EndNote ]
@misc{talk-Bauer-HotCloudPerf2018, author = {Bauer, Andr{\'e}}, howpublished = {Talk at HotCloudperf 2018}, keywords = {BUNGEE}, month = {04}, title = {{Extending BUNGEE Elasticity Benchmark for Multi-Tier Cloud Applications}}, year = 2018 }

%0 Generic %1 talk-Bauer-HotCloudPerf2018 %A Bauer, André %D 2018 %T Extending BUNGEE Elasticity Benchmark for Multi-Tier Cloud Applications
FOX: Cost-Awareness for Autonomic Resource Management in Public Clouds. V. Lesch; A. Bauer; N. Herbst; S. Kounev; in Proceedings of the 9th ACM/SPEC International Conference on Performance Engineering (ICPE 2018) (2018).
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
@inproceedings{LeBaHeKo-ICPE18-FOX-AS-COST, address = {New York, NY, USA}, author = {Lesch, Veronika and Bauer, Andr{\'e} and Herbst, Nikolas and Kounev, Samuel}, booktitle = {Proceedings of the 9th ACM/SPEC International Conference on Performance Engineering (ICPE 2018)}, keywords = {Optimization}, month = {04}, note = {Full Paper Acceptance Ratio: 23,7%}, publisher = {ACM}, title = {{FOX: Cost-Awareness for Autonomic Resource Management in Public Clouds}}, year = 2018 }

%0 Conference Paper %1 LeBaHeKo-ICPE18-FOX-AS-COST %A Lesch, Veronika %A Bauer, André %A Herbst, Nikolas %A Kounev, Samuel %B Proceedings of the 9th ACM/SPEC International Conference on Performance Engineering (ICPE 2018) %C New York, NY, USA %D 2018 %I ACM %T FOX: Cost-Awareness for Autonomic Resource Management in Public Clouds %U https://doi.org/10.1145/3184407.3184415
On the Value of Service Demand Estimation for Auto-Scaling. A. Bauer; J. Grohmann; N. Herbst; S. Kounev; in Proceedings of 19th International GI/ITG Conference on Measurement, Modelling and Evaluation of Computing Systems (MMB 2018) (2018). (Bd. 10740) 142–156.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
- [ DOI ]
In the context of performance models, service demands are key model parameters capturing the average time individual requests of different workload classes are actively processed. In a system under load, due to measurement interference, service demands normally cannot be measured directly, however, a number of estimation approaches exist based on high-level performance metrics. In this paper, we show that service demands provide significant benefits for implementing modern auto-scalers. Auto-scaling describes the process of dynamically adjusting the number of allocated virtual resources (e.g., virtual machines) in a data center according to the incoming workload. We demonstrate that even a simple auto-scaler that leverages information about service demands significantly outperforms auto-scalers solely based on CPU utilization measurements. This is shown by testing two approaches in three different scenarios. Our results show that the service demand-based auto-scaler outperforms the CPU utilization-based one in all scenarios. Our results encourage further research on the application of service demand estimates for resource management in data centers.

@inproceedings{BaGrHeKo2018-MMB-ServiceDemand, abstract = {In the context of performance models, service demands are key model parameters capturing the average time individual requests of different workload classes are actively processed. In a system under load, due to measurement interference, service demands normally cannot be measured directly, however, a number of estimation approaches exist based on high-level performance metrics. In this paper, we show that service demands provide significant benefits for implementing modern auto-scalers. Auto-scaling describes the process of dynamically adjusting the number of allocated virtual resources (e.g., virtual machines) in a data center according to the incoming workload. We demonstrate that even a simple auto-scaler that leverages information about service demands significantly outperforms auto-scalers solely based on CPU utilization measurements. This is shown by testing two approaches in three different scenarios. Our results show that the service demand-based auto-scaler outperforms the CPU utilization-based one in all scenarios. Our results encourage further research on the application of service demand estimates for resource management in data centers.}, address = {Cham}, author = {Bauer, Andr{\'e} and Grohmann, Johannes and Herbst, Nikolas and Kounev, Samuel}, booktitle = {Proceedings of 19th International GI/ITG Conference on Measurement, Modelling and Evaluation of Computing Systems (MMB 2018)}, keywords = {Virtualization}, month = {02}, pages = {142–156}, publisher = {Springer}, series = {Lecture Notes in Computer Science}, title = {{On the Value of Service Demand Estimation for Auto-Scaling}}, volume = 10740, year = 2018 }

%0 Conference Paper %1 BaGrHeKo2018-MMB-ServiceDemand %A Bauer, André %A Grohmann, Johannes %A Herbst, Nikolas %A Kounev, Samuel %B Proceedings of 19th International GI/ITG Conference on Measurement, Modelling and Evaluation of Computing Systems (MMB 2018) %C Cham %D 2018 %I Springer %P 142--156 %R 10.1007/978-3-319-74947-1_10 %T On the Value of Service Demand Estimation for Auto-Scaling %U https://doi.org/10.1007/978-3-319-74947-1_10 %V 10740 %X In the context of performance models, service demands are key model parameters capturing the average time individual requests of different workload classes are actively processed. In a system under load, due to measurement interference, service demands normally cannot be measured directly, however, a number of estimation approaches exist based on high-level performance metrics. In this paper, we show that service demands provide significant benefits for implementing modern auto-scalers. Auto-scaling describes the process of dynamically adjusting the number of allocated virtual resources (e.g., virtual machines) in a data center according to the incoming workload. We demonstrate that even a simple auto-scaler that leverages information about service demands significantly outperforms auto-scalers solely based on CPU utilization measurements. This is shown by testing two approaches in three different scenarios. Our results show that the service demand-based auto-scaler outperforms the CPU utilization-based one in all scenarios. Our results encourage further research on the application of service demand estimates for resource management in data centers. %@ 978-3-319-74947-1

2017[ to top ]

An Experimental Performance Evaluation of Autoscaling Policies for Complex Workflows. A. Ilyushkin; A. Ali-Eldin; N. Herbst; A. V. Papadopoulos; B. Ghit; D. Epema; A. Iosup; in Proceedings of the 8th ACM/SPEC International Conference on Performance Engineering (ICPE 2017) (2017).
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
Simplifying the task of resource management and scheduling for customers, while still delivering complex Quality-of-Service (QoS), is key to cloud computing. Many autoscaling policies have been proposed in the past decade to decide on behalf of cloud customers when and how to provision resources to a cloud application utilizing cloud elasticity features. However, in prior work, when a new policy is proposed, it is seldom compared to the state-of-the-art, and is often compared only to static provisioning using a predefined QoS target. This reduces the ability of cloud customers and of cloud operators to choose and deploy an autoscaling policy. In our work, we conduct an experimental performance evaluation of autoscaling policies, using as application model workflows, a commonly used formalism for automating resource management for applications with well-defined yet complex structure. We present a detailed comparative study of general, state-of-the-art, generic autoscaling policies, along with two new workflow-specific policies. To understand the performance differences between the 7 policies, we conduct various forms of pairwise and group comparisons. We report both individual and aggregated metrics. Our results highlights the trade-offs between the suggested policies, and thus enable a better understanding of the current state-of-the-art.

@inproceedings{IlAlHePa-ICPE17-AutoScalerWorkflowEval, abstract = {Simplifying the task of resource management and scheduling for customers, while still delivering complex Quality-of-Service (QoS), is key to cloud computing. Many autoscaling policies have been proposed in the past decade to decide on behalf of cloud customers when and how to provision resources to a cloud application utilizing cloud elasticity features. However, in prior work, when a new policy is proposed, it is seldom compared to the state-of-the-art, and is often compared only to static provisioning using a predefined QoS target. This reduces the ability of cloud customers and of cloud operators to choose and deploy an autoscaling policy. In our work, we conduct an experimental performance evaluation of autoscaling policies, using as application model workflows, a commonly used formalism for automating resource management for applications with well-defined yet complex structure. We present a detailed comparative study of general, state-of-the-art, generic autoscaling policies, along with two new workflow-specific policies. To understand the performance differences between the 7 policies, we conduct various forms of pairwise and group comparisons. We report both individual and aggregated metrics. Our results highlights the trade-offs between the suggested policies, and thus enable a better understanding of the current state-of-the-art.}, address = {New York, NY, USA}, author = {Ilyushkin, Alexey and Ali-Eldin, Ahmed and Herbst, Nikolas and Papadopoulos, Alessandro V. and Ghit, Bogdan and Epema, Dick and Iosup, Alexandru}, booktitle = {Proceedings of the 8th ACM/SPEC International Conference on Performance Engineering (ICPE 2017)}, keywords = {Self-adaptive-systems}, month = {04}, note = {Best Paper Candidate (1/4)}, publisher = {ACM}, title = {{An Experimental Performance Evaluation of Autoscaling Policies for Complex Workflows}}, year = 2017 }

%0 Conference Paper %1 IlAlHePa-ICPE17-AutoScalerWorkflowEval %A Ilyushkin, Alexey %A Ali-Eldin, Ahmed %A Herbst, Nikolas %A Papadopoulos, Alessandro V. %A Ghit, Bogdan %A Epema, Dick %A Iosup, Alexandru %B Proceedings of the 8th ACM/SPEC International Conference on Performance Engineering (ICPE 2017) %C New York, NY, USA %D 2017 %I ACM %T An Experimental Performance Evaluation of Autoscaling Policies for Complex Workflows %X Simplifying the task of resource management and scheduling for customers, while still delivering complex Quality-of-Service (QoS), is key to cloud computing. Many autoscaling policies have been proposed in the past decade to decide on behalf of cloud customers when and how to provision resources to a cloud application utilizing cloud elasticity features. However, in prior work, when a new policy is proposed, it is seldom compared to the state-of-the-art, and is often compared only to static provisioning using a predefined QoS target. This reduces the ability of cloud customers and of cloud operators to choose and deploy an autoscaling policy. In our work, we conduct an experimental performance evaluation of autoscaling policies, using as application model workflows, a commonly used formalism for automating resource management for applications with well-defined yet complex structure. We present a detailed comparative study of general, state-of-the-art, generic autoscaling policies, along with two new workflow-specific policies. To understand the performance differences between the 7 policies, we conduct various forms of pairwise and group comparisons. We report both individual and aggregated metrics. Our results highlights the trade-offs between the suggested policies, and thus enable a better understanding of the current state-of-the-art.
Metrics and Benchmarks for Self-Aware Computing Systems. N. Herbst; S. Becker; S. Kounev; H. Koziolek; M. Maggio; A. Milenkoski; E. Smirni; in Self-Aware Computing Systems, S. Kounev, J. O. Kephart, A. Milenkoski, X. Zhu (Hrsg.) (2017).
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
In this chapter, we propose a list of metrics grouped by the MAPE-K paradigm for quantifying properties of self-aware computing systems. This set of metrics can be seen as a starting point toward benchmarking and comparing self-aware computing systems on a level-playing field. We discuss state-of-the art approaches in the related fields of self-adaptation and self-protection to identify commonalities in metrics for self-aware computing. We illustrate the need for benchmarking self-aware computing systems with the help of an approach that uncovers real-time characteristics of operating systems. Gained insights of this approach can be seen as a way of enhancing self-awareness by a measurement methodology on an ongoing basis. At the end of this chapter, we address new challenges in reference workload definition for benchmarking self-aware computing systems, namely load intensity patterns and burstiness modeling.

@incollection{HeBeKoKoMaMiSm2017-self, abstract = {{In this chapter, we propose a list of metrics grouped by the MAPE-K paradigm for quantifying properties of self-aware computing systems. This set of metrics can be seen as a starting point toward benchmarking and comparing self-aware computing systems on a level-playing field. We discuss state-of-the art approaches in the related fields of self-adaptation and self-protection to identify commonalities in metrics for self-aware computing. We illustrate the need for benchmarking self-aware computing systems with the help of an approach that uncovers real-time characteristics of operating systems. Gained insights of this approach can be seen as a way of enhancing self-awareness by a measurement methodology on an ongoing basis. At the end of this chapter, we address new challenges in reference workload definition for benchmarking self-aware computing systems, namely load intensity patterns and burstiness modeling.}}, address = {{Berlin Heidelberg, Germany}}, author = {Herbst, Nikolas and Becker, Steffen and Kounev, Samuel and Koziolek, Heiko and Maggio, Martina and Milenkoski, Aleksandar and Smirni, Evgenia}, booktitle = {{Self-Aware Computing Systems}}, editor = {Kounev, Samuel and Kephart, Jeffrey O. and Milenkoski, Aleksandar and Zhu, Xiaoyun}, keywords = {Self-adaptive-systems}, publisher = {{Springer Verlag}}, title = {{Metrics and Benchmarks for Self-Aware Computing Systems}}, year = 2017 }

%0 Book Section %1 HeBeKoKoMaMiSm2017-self %A Herbst, Nikolas %A Becker, Steffen %A Kounev, Samuel %A Koziolek, Heiko %A Maggio, Martina %A Milenkoski, Aleksandar %A Smirni, Evgenia %B Self-Aware Computing Systems %C Berlin Heidelberg, Germany %D 2017 %E Kounev, Samuel %E Kephart, Jeffrey O. %E Milenkoski, Aleksandar %E Zhu, Xiaoyun %I Springer Verlag %T Metrics and Benchmarks for Self-Aware Computing Systems %U https://link.springer.com/chapter/10.1007%2F978-3-319-47474-8_14 %X In this chapter, we propose a list of metrics grouped by the MAPE-K paradigm for quantifying properties of self-aware computing systems. This set of metrics can be seen as a starting point toward benchmarking and comparing self-aware computing systems on a level-playing field. We discuss state-of-the art approaches in the related fields of self-adaptation and self-protection to identify commonalities in metrics for self-aware computing. We illustrate the need for benchmarking self-aware computing systems with the help of an approach that uncovers real-time characteristics of operating systems. Gained insights of this approach can be seen as a way of enhancing self-awareness by a measurement methodology on an ongoing basis. At the end of this chapter, we address new challenges in reference workload definition for benchmarking self-aware computing systems, namely load intensity patterns and burstiness modeling.
Modeling and Extracting Load Intensity Profiles. J. von Kistowski; N. Herbst; S. Kounev; H. Groenda; C. Stier; S. Lehrig; in ACM Transactions on Autonomous and Adaptive Systems (TAAS) (2017). 11(4) 23:1–23:28.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Today's system developers and operators face the challenge of creating software systems that make efficient use of dynamically allocated resources under highly variable and dynamic load profiles, while at the same time delivering reliable performance. Autonomic controllers, e.g., an advanced auto-scaling mechanism in a cloud computing context, can benefit from an abstracted load model as knowledge to reconfigure on time and precisely. Existing workload characterization approaches have limited support to capture variations the inter-arrival times of incoming work units over time (i.e., a variable load profile). For example, industrial and scientific benchmarks support constant or stepwise increasing load, or inter-arrival times defined by statistical distributions or recorded traces. These options show shortcomings either in representative character of load variation patterns or in abstraction and flexibility of their format. In this article, we present the Descartes Load Intensity Model (DLIM) approach addressing these issues. DLIM provides a modeling formalism for describing load intensity variations over time. A DLIM instance is a compact formal description of a load intensity trace. DLIM-based tools provide features for benchmarking, performance and recorded load intensity trace analysis. As manually obtaining and maintaining DLIM instances becomes time consuming, we contribute three automated extraction methods and devised metrics for comparison and method selection. We discuss how these features are used to enhance system management approaches for adaptations during run-time, and how they are integrated into simulation contexts and enable benchmarking of elastic or adaptive behavior. We show that automatically extracted DLIM instances exhibit an average modeling error of 15.2% over ten different real-world traces that cover between two weeks and seven months. These results underline DLIM model expressiveness. In terms of accuracy and processing speed, our proposed extraction methods for the descriptive models are comparable to existing time series decomposition methods. Additionally, we illustrate DLIM applicability by outlining approaches of workload modeling in systems engineering that employ or rely on our proposed load intensity modeling formalism.

@article{KiHeKo-TAAS17-ModelExtractLoadProfiles, abstract = {Today's system developers and operators face the challenge of creating software systems that make efficient use of dynamically allocated resources under highly variable and dynamic load profiles, while at the same time delivering reliable performance. Autonomic controllers, e.g., an advanced auto-scaling mechanism in a cloud computing context, can benefit from an abstracted load model as knowledge to reconfigure on time and precisely. Existing workload characterization approaches have limited support to capture variations the inter-arrival times of incoming work units over time (i.e., a variable load profile). For example, industrial and scientific benchmarks support constant or stepwise increasing load, or inter-arrival times defined by statistical distributions or recorded traces. These options show shortcomings either in representative character of load variation patterns or in abstraction and flexibility of their format. In this article, we present the Descartes Load Intensity Model (DLIM) approach addressing these issues. DLIM provides a modeling formalism for describing load intensity variations over time. A DLIM instance is a compact formal description of a load intensity trace. DLIM-based tools provide features for benchmarking, performance and recorded load intensity trace analysis. As manually obtaining and maintaining DLIM instances becomes time consuming, we contribute three automated extraction methods and devised metrics for comparison and method selection. We discuss how these features are used to enhance system management approaches for adaptations during run-time, and how they are integrated into simulation contexts and enable benchmarking of elastic or adaptive behavior. We show that automatically extracted DLIM instances exhibit an average modeling error of 15.2\% over ten different real-world traces that cover between two weeks and seven months. These results underline DLIM model expressiveness. In terms of accuracy and processing speed, our proposed extraction methods for the descriptive models are comparable to existing time series decomposition methods. Additionally, we illustrate DLIM applicability by outlining approaches of workload modeling in systems engineering that employ or rely on our proposed load intensity modeling formalism.}, address = {New York, NY, USA}, author = {von Kistowski, J{\'o}akim and Herbst, Nikolas and Kounev, Samuel and Groenda, Henning and Stier, Christian and Lehrig, Sebastian}, journal = {ACM Transactions on Autonomous and Adaptive Systems (TAAS)}, keywords = {Self-adaptive-systems}, month = {01}, number = 4, pages = {23:1–23:28}, publisher = {ACM}, title = {{Modeling and Extracting Load Intensity Profiles}}, volume = 11, year = 2017 }

%0 Journal Article %1 KiHeKo-TAAS17-ModelExtractLoadProfiles %A von Kistowski, Jóakim %A Herbst, Nikolas %A Kounev, Samuel %A Groenda, Henning %A Stier, Christian %A Lehrig, Sebastian %C New York, NY, USA %D 2017 %I ACM %J ACM Transactions on Autonomous and Adaptive Systems (TAAS) %N 4 %P 23:1--23:28 %T Modeling and Extracting Load Intensity Profiles %U http://doi.acm.org/10.1145/3019596 %V 11 %X Today's system developers and operators face the challenge of creating software systems that make efficient use of dynamically allocated resources under highly variable and dynamic load profiles, while at the same time delivering reliable performance. Autonomic controllers, e.g., an advanced auto-scaling mechanism in a cloud computing context, can benefit from an abstracted load model as knowledge to reconfigure on time and precisely. Existing workload characterization approaches have limited support to capture variations the inter-arrival times of incoming work units over time (i.e., a variable load profile). For example, industrial and scientific benchmarks support constant or stepwise increasing load, or inter-arrival times defined by statistical distributions or recorded traces. These options show shortcomings either in representative character of load variation patterns or in abstraction and flexibility of their format. In this article, we present the Descartes Load Intensity Model (DLIM) approach addressing these issues. DLIM provides a modeling formalism for describing load intensity variations over time. A DLIM instance is a compact formal description of a load intensity trace. DLIM-based tools provide features for benchmarking, performance and recorded load intensity trace analysis. As manually obtaining and maintaining DLIM instances becomes time consuming, we contribute three automated extraction methods and devised metrics for comparison and method selection. We discuss how these features are used to enhance system management approaches for adaptations during run-time, and how they are integrated into simulation contexts and enable benchmarking of elastic or adaptive behavior. We show that automatically extracted DLIM instances exhibit an average modeling error of 15.2% over ten different real-world traces that cover between two weeks and seven months. These results underline DLIM model expressiveness. In terms of accuracy and processing speed, our proposed extraction methods for the descriptive models are comparable to existing time series decomposition methods. Additionally, we illustrate DLIM applicability by outlining approaches of workload modeling in systems engineering that employ or rely on our proposed load intensity modeling formalism.
Elasticity Benchmarking for Multi-Tier Cloud Applications. M. Wilhelm; Thesis; University of Würzburg; Am Hubland, Informatikgebäude, 97074 Würzburg, Germany. (2017, Juni).
- [ BibTeX ]
- [ EndNote ]
@mastersthesis{BA-Wilhelm-2016, address = {Am Hubland, Informatikgeb{\"a}ude, 97074 W{\"u}rzburg, Germany}, author = {Wilhelm, Marcus}, keywords = {Optimization}, month = {06}, note = {W{\"u}rzburg Software Engineering Award sponsored by Bosch Rexroth}, school = {University of W{\"u}rzburg}, title = {{Elasticity Benchmarking for Multi-Tier Cloud Applications}}, type = {{Bachelor Thesis}}, year = 2017 }

%0 Thesis %1 BA-Wilhelm-2016 %A Wilhelm, Marcus %C Am Hubland, Informatikgebäude, 97074 Würzburg, Germany %D 2017 %T Elasticity Benchmarking for Multi-Tier Cloud Applications
Design and Evaluation of a Proactive, Application-Aware Auto-Scaler. A. Bauer; N. Herbst; S. Kounev; in Proceedings of the 8th ACM/SPEC International Conference on Performance Engineering (ICPE 2017) (2017).
- [ BibTeX ]
- [ EndNote ]
@inproceedings{BaHeKo2017-ICPE-Auto-Scaler-Tutorial, author = {Bauer, Andr{\'e} and Herbst, Nikolas and Kounev, Samuel}, booktitle = {{Proceedings of the 8th ACM/SPEC International Conference on Performance Engineering (ICPE 2017)}}, keywords = {LIMBO}, month = {04}, title = {{Design and Evaluation of a Proactive, Application-Aware Auto-Scaler}}, year = 2017 }

%0 Conference Paper %1 BaHeKo2017-ICPE-Auto-Scaler-Tutorial %A Bauer, André %A Herbst, Nikolas %A Kounev, Samuel %B Proceedings of the 8th ACM/SPEC International Conference on Performance Engineering (ICPE 2017) %D 2017 %T Design and Evaluation of a Proactive, Application-Aware Auto-Scaler
Scalability Analysis of Cloud Software Services. G. Brataas; N. Herbst; S. Ivansek; J. Polutnik; in Companion Proceedings of the 14th IEEE International Conference on Autonomic Computing (ICAC 2017), Self Organizing Self Managing Clouds Workshop (SOSeMC 2017) (2017).
- [ BibTeX ]
- [ EndNote ]
@inproceedings{BrHe2017CloudScalability, author = {Brataas, Gunnar and Herbst, Nikolas and Ivansek, Simon and Polutnik, Jure}, booktitle = {Companion Proceedings of the 14th IEEE International Conference on Autonomic Computing (ICAC 2017), Self Organizing Self Managing Clouds Workshop (SOSeMC 2017)}, keywords = {t_workshop}, month = {07}, publisher = {IEEE}, title = {{Scalability Analysis of Cloud Software Services}}, year = 2017 }

%0 Conference Paper %1 BrHe2017CloudScalability %A Brataas, Gunnar %A Herbst, Nikolas %A Ivansek, Simon %A Polutnik, Jure %B Companion Proceedings of the 14th IEEE International Conference on Autonomic Computing (ICAC 2017), Self Organizing Self Managing Clouds Workshop (SOSeMC 2017) %D 2017 %I IEEE %T Scalability Analysis of Cloud Software Services
Self-Aware Multidimensional Auto-Scaling. V. Lesch; Thesis; University of Würzburg; Am Hubland, Informatikgebäude, 97074 Würzburg, Germany. (2017, September).
- [ BibTeX ]
- [ EndNote ]
@mastersthesis{Lesch2017Self-Aware, address = {Am Hubland, Informatikgeb{\"a}ude, 97074 W{\"u}rzburg, Germany}, author = {Lesch, Veronika}, keywords = {Virtualization}, month = {09}, note = {W{\"u}rzburg Software Engineering Award sponsored by Bosch Rexroth}, school = {University of W{\"u}rzburg}, title = {{Self-Aware Multidimensional Auto-Scaling}}, type = {{Master Thesis}}, year = 2017 }

%0 Thesis %1 Lesch2017Self-Aware %A Lesch, Veronika %C Am Hubland, Informatikgebäude, 97074 Würzburg, Germany %D 2017 %T Self-Aware Multidimensional Auto-Scaling

2016[ to top ]

Design and Evaluation of a Proactive, Application-Aware Elasticity Mechanism. A. Bauer; (2016, November).
- [ BibTeX ]
- [ EndNote ]
@misc{talk-Bauer-SSP2016, author = {Bauer, Andr{\'e}}, howpublished = {Experience Talk at SSP 2016}, keywords = {BUNGEE}, month = 11, title = {{Design and Evaluation of a Proactive, Application-Aware Elasticity Mechanism}}, year = 2016 }

%0 Generic %1 talk-Bauer-SSP2016 %A Bauer, André %D 2016 %T Design and Evaluation of a Proactive, Application-Aware Elasticity Mechanism
Design and Evaluation of a Proactive, Application-Aware Elasticity Mechanism. A. Bauer; Thesis; University of Würzburg; Am Hubland, Informatikgebäude, 97074 Würzburg, Germany. (2016, September).
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
In order to guarantee a reliable service quality to their customers and due to the increasing complexity of cloud applications, SaaS cloud providers usually run their business critical applications with a fixed amount of resources. This approach has drawbacks such as a higher costs (e.g., bad energy efficiency, if the system is not fully utilized), or bad performance regarding unexpected work peaks. Amazon with its EC2, for instance, provides a configurable scheduled-based and rule-based auto-scaler. However, this reactive scaling has a latency depending on resource type in the order of minutes. Therefore, proactive and application aware auto-scalers are required - intelligent controllers, which can reconfigure the system on time, to ensure high availability and constant performance under changing conditions. The existing auto-scalers can be classified into five groups and prominent examples of each class were investigated. The result of this observation is that the existing controllers are either application-specific or generic. I.e. the auto-scalers can either perform well only on the chosen system or they perform far beyond their means. In this thesis, a novel proactive, application-aware elasticity mechanism called Chameleon is introduced. The proposed controller employs established forecast methods for short-, mid- and long-term predictions of the arriving load intensity, application knowledge, and resource demand estimation to calculate the required resources per work unit. While taking these information into account, the mechanism reconfigures the deployment of an application in a way that the supply of resources matches the current and estimated future demand for resources. I.e., Chameleon consists of two mechanisms: (I) a reactive rule-based controller as fall-back insurance and (II) a proactive controller which has three major building blocks: (a) continuous workload forecasting (using a modified version of WCF) with dynamic forecast method selection, (b) a descriptive performance model/ application knowledge (DML Model@RunTime), and (c) optimized resource demand estimation approaches (using the tool LibReDE). In the evaluation, Chameleon is compared against four different state-of-the-art controllers in a private CloudStack-based environment. The BUNGEE elasticity benchmark framework and metrics are used to conduct and analyse the row of experiments. As workload scenarios, a CPU intensive http application that solves nxn matrices (whereas n is the request parameter) and SPECjEnterprise2010 is used. The applications are driven with variable load profiles (extracted with LIMBO from the FIFA World Cup 1998 trace) generated with JMeter (instead of Faban - as classically used by SPECjEnterprise2010). During the CPU intensive scenario Chameleon achieves the best scaling behaviour (based on user and elasticity metrics) and in the SPECjEnterprise2010 the second best user metrics.

@mastersthesis{Bauer2016, abstract = {In order to guarantee a reliable service quality to their customers and due to the increasing complexity of cloud applications, SaaS cloud providers usually run their business critical applications with a fixed amount of resources. This approach has drawbacks such as a higher costs (e.g., bad energy efficiency, if the system is not fully utilized), or bad performance regarding unexpected work peaks. Amazon with its EC2, for instance, provides a configurable scheduled-based and rule-based auto-scaler. However, this reactive scaling has a latency depending on resource type in the order of minutes. Therefore, proactive and application aware auto-scalers are required - intelligent controllers, which can reconfigure the system on time, to ensure high availability and constant performance under changing conditions. The existing auto-scalers can be classified into five groups and prominent examples of each class were investigated. The result of this observation is that the existing controllers are either application-specific or generic. I.e. the auto-scalers can either perform well only on the chosen system or they perform far beyond their means. In this thesis, a novel proactive, application-aware elasticity mechanism called Chameleon is introduced. The proposed controller employs established forecast methods for short-, mid- and long-term predictions of the arriving load intensity, application knowledge, and resource demand estimation to calculate the required resources per work unit. While taking these information into account, the mechanism reconfigures the deployment of an application in a way that the supply of resources matches the current and estimated future demand for resources. I.e., Chameleon consists of two mechanisms: (I) a reactive rule-based controller as fall-back insurance and (II) a proactive controller which has three major building blocks: (a) continuous workload forecasting (using a modified version of WCF) with dynamic forecast method selection, (b) a descriptive performance model/ application knowledge (DML Model@RunTime), and (c) optimized resource demand estimation approaches (using the tool LibReDE). In the evaluation, Chameleon is compared against four different state-of-the-art controllers in a private CloudStack-based environment. The BUNGEE elasticity benchmark framework and metrics are used to conduct and analyse the row of experiments. As workload scenarios, a CPU intensive http application that solves nxn matrices (whereas n is the request parameter) and SPECjEnterprise2010 is used. The applications are driven with variable load profiles (extracted with LIMBO from the FIFA World Cup 1998 trace) generated with JMeter (instead of Faban - as classically used by SPECjEnterprise2010). During the CPU intensive scenario Chameleon achieves the best scaling behaviour (based on user and elasticity metrics) and in the SPECjEnterprise2010 the second best user metrics.}, address = {Am Hubland, Informatikgeb{\"a}ude, 97074 W{\"u}rzburg, Germany}, author = {Bauer, Andr{\'e}}, keywords = {Self-adaptive-systems}, month = {09}, school = {University of W{\"u}rzburg}, title = {{Design and Evaluation of a Proactive, Application-Aware Elasticity Mechanism}}, type = {{Master Thesis}}, year = 2016 }

%0 Thesis %1 Bauer2016 %A Bauer, André %C Am Hubland, Informatikgebäude, 97074 Würzburg, Germany %D 2016 %T Design and Evaluation of a Proactive, Application-Aware Elasticity Mechanism %X In order to guarantee a reliable service quality to their customers and due to the increasing complexity of cloud applications, SaaS cloud providers usually run their business critical applications with a fixed amount of resources. This approach has drawbacks such as a higher costs (e.g., bad energy efficiency, if the system is not fully utilized), or bad performance regarding unexpected work peaks. Amazon with its EC2, for instance, provides a configurable scheduled-based and rule-based auto-scaler. However, this reactive scaling has a latency depending on resource type in the order of minutes. Therefore, proactive and application aware auto-scalers are required - intelligent controllers, which can reconfigure the system on time, to ensure high availability and constant performance under changing conditions. The existing auto-scalers can be classified into five groups and prominent examples of each class were investigated. The result of this observation is that the existing controllers are either application-specific or generic. I.e. the auto-scalers can either perform well only on the chosen system or they perform far beyond their means. In this thesis, a novel proactive, application-aware elasticity mechanism called Chameleon is introduced. The proposed controller employs established forecast methods for short-, mid- and long-term predictions of the arriving load intensity, application knowledge, and resource demand estimation to calculate the required resources per work unit. While taking these information into account, the mechanism reconfigures the deployment of an application in a way that the supply of resources matches the current and estimated future demand for resources. I.e., Chameleon consists of two mechanisms: (I) a reactive rule-based controller as fall-back insurance and (II) a proactive controller which has three major building blocks: (a) continuous workload forecasting (using a modified version of WCF) with dynamic forecast method selection, (b) a descriptive performance model/ application knowledge (DML Model@RunTime), and (c) optimized resource demand estimation approaches (using the tool LibReDE). In the evaluation, Chameleon is compared against four different state-of-the-art controllers in a private CloudStack-based environment. The BUNGEE elasticity benchmark framework and metrics are used to conduct and analyse the row of experiments. As workload scenarios, a CPU intensive http application that solves nxn matrices (whereas n is the request parameter) and SPECjEnterprise2010 is used. The applications are driven with variable load profiles (extracted with LIMBO from the FIFA World Cup 1998 trace) generated with JMeter (instead of Faban - as classically used by SPECjEnterprise2010). During the CPU intensive scenario Chameleon achieves the best scaling behaviour (based on user and elasticity metrics) and in the SPECjEnterprise2010 the second best user metrics.
Ready for Rain? A View from SPEC Research on the Future of Cloud Metrics. (SPEC-RG-2016-01), N. Herbst; R. Krebs; G. Oikonomou; G. Kousiouris; A. Evangelinou; A. Iosup; S. Kounev; (2016).
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
@techreport{SPEC-RG-2016-01-CloudMetrics, author = {Herbst, Nikolas and Krebs, Rouven and Oikonomou, Giorgos and Kousiouris, George and Evangelinou, Athanasia and Iosup, Alexandru and Kounev, Samuel}, institution = {SPEC Research Group — Cloud Working Group, Standard Performance Evaluation Corporation (SPEC)}, keywords = {Reliability}, number = {SPEC-RG-2016-01}, title = {{Ready for Rain? A View from SPEC Research on the Future of Cloud Metrics}}, year = 2016 }

%0 Report %1 SPEC-RG-2016-01-CloudMetrics %A Herbst, Nikolas %A Krebs, Rouven %A Oikonomou, Giorgos %A Kousiouris, George %A Evangelinou, Athanasia %A Iosup, Alexandru %A Kounev, Samuel %D 2016 %N SPEC-RG-2016-01 %T Ready for Rain? A View from SPEC Research on the Future of Cloud Metrics %U https://research.spec.org/fileadmin/user_upload/documents/rg_cloud/endorsed_publications/SPEC-RG-2016-01_CloudMetrics.pdf

2015[ to top ]

BUNGEE: An Elasticity Benchmark for Self-Adaptive IaaS Cloud Environments. N. R. Herbst; S. Kounev; A. Weber; H. Groenda; in Proceedings of the 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS 2015) (2015).
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
Today's infrastructure clouds provide resource elasticity (i.e. auto-scaling) mechanisms enabling self-adaptive resource provisioning to reflect variations in the load intensity over time. These mechanisms impact on the application performance, however, their effect in specific situations is hard to quantify and compare. To evaluate the quality of elasticity mechanisms provided by different platforms and configurations, respective metrics and benchmarks are required. Existing metrics for elasticity only consider the time required to provision and deprovision resources or the costs impact of adaptations. Existing benchmarks lack the capability to handle open workloads with realistic load intensity profiles and do not explicitly distinguish between the performance exhibited by the provisioned underlying resources, on the one hand, and the quality of the elasticity mechanisms themselves, on the other hand. In this paper, we propose reliable metrics for quantifying the timing aspects and accuracy of elasticity. Based on these metrics, we propose a novel approach for benchmarking the elasticity of Infrastructure-as-a-Service (IaaS) cloud platforms independent of the performance exhibited by the provisioned underlying resources. We show that the proposed metrics provide consistent ranking of elastic platforms on an ordinal scale. Finally, we present an extensive case study of real-world complexity demonstrating that the proposed approach is applicable in realistic scenarios and can cope with different levels of resource efficiency.

@inproceedings{HeKoWeGr_2015_SEAMS_BUNGEE, abstract = {Today's infrastructure clouds provide resource elasticity (i.e. auto-scaling) mechanisms enabling self-adaptive resource provisioning to reflect variations in the load intensity over time. These mechanisms impact on the application performance, however, their effect in specific situations is hard to quantify and compare. To evaluate the quality of elasticity mechanisms provided by different platforms and configurations, respective metrics and benchmarks are required. Existing metrics for elasticity only consider the time required to provision and deprovision resources or the costs impact of adaptations. Existing benchmarks lack the capability to handle open workloads with realistic load intensity profiles and do not explicitly distinguish between the performance exhibited by the provisioned underlying resources, on the one hand, and the quality of the elasticity mechanisms themselves, on the other hand. In this paper, we propose reliable metrics for quantifying the timing aspects and accuracy of elasticity. Based on these metrics, we propose a novel approach for benchmarking the elasticity of Infrastructure-as-a-Service (IaaS) cloud platforms independent of the performance exhibited by the provisioned underlying resources. We show that the proposed metrics provide consistent ranking of elastic platforms on an ordinal scale. Finally, we present an extensive case study of real-world complexity demonstrating that the proposed approach is applicable in realistic scenarios and can cope with different levels of resource efficiency.}, author = {Herbst, Nikolas Roman and Kounev, Samuel and Weber, Andreas and Groenda, Henning}, booktitle = {Proceedings of the 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS 2015)}, keywords = {LIMBO}, month = {05}, note = {Acceptance rate: 29\%}, title = {{BUNGEE: An Elasticity Benchmark for Self-Adaptive IaaS Cloud Environments}}, year = 2015 }

%0 Conference Paper %1 HeKoWeGr_2015_SEAMS_BUNGEE %A Herbst, Nikolas Roman %A Kounev, Samuel %A Weber, Andreas %A Groenda, Henning %B Proceedings of the 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS 2015) %D 2015 %T BUNGEE: An Elasticity Benchmark for Self-Adaptive IaaS Cloud Environments %X Today's infrastructure clouds provide resource elasticity (i.e. auto-scaling) mechanisms enabling self-adaptive resource provisioning to reflect variations in the load intensity over time. These mechanisms impact on the application performance, however, their effect in specific situations is hard to quantify and compare. To evaluate the quality of elasticity mechanisms provided by different platforms and configurations, respective metrics and benchmarks are required. Existing metrics for elasticity only consider the time required to provision and deprovision resources or the costs impact of adaptations. Existing benchmarks lack the capability to handle open workloads with realistic load intensity profiles and do not explicitly distinguish between the performance exhibited by the provisioned underlying resources, on the one hand, and the quality of the elasticity mechanisms themselves, on the other hand. In this paper, we propose reliable metrics for quantifying the timing aspects and accuracy of elasticity. Based on these metrics, we propose a novel approach for benchmarking the elasticity of Infrastructure-as-a-Service (IaaS) cloud platforms independent of the performance exhibited by the provisioned underlying resources. We show that the proposed metrics provide consistent ranking of elastic platforms on an ordinal scale. Finally, we present an extensive case study of real-world complexity demonstrating that the proposed approach is applicable in realistic scenarios and can cope with different levels of resource efficiency.
Load Testing Elasticity and Performance Isolation in Shared Execution Environments. S. Kounev; in Proceedings of the 4th International Workshop on Large-Scale Testing (LT 2015), co-located with the 6th ACM/SPEC International Conference on Performance Engineering (ICPE 2015) (2015).
- [ BibTeX ]
- [ EndNote ]
@inproceedings{Ko2015-ICPE_LT_Workshop-Keynote, address = {New York, NY, USA}, author = {Kounev, Samuel}, booktitle = {Proceedings of the 4th International Workshop on Large-Scale Testing (LT 2015), co-located with the 6th ACM/SPEC International Conference on Performance Engineering (ICPE 2015)}, keywords = {Isolation}, month = {02}, publisher = {ACM}, title = {{Load Testing Elasticity and Performance Isolation in Shared Execution Environments}}, year = 2015 }

%0 Conference Paper %1 Ko2015-ICPE_LT_Workshop-Keynote %A Kounev, Samuel %B Proceedings of the 4th International Workshop on Large-Scale Testing (LT 2015), co-located with the 6th ACM/SPEC International Conference on Performance Engineering (ICPE 2015) %C New York, NY, USA %D 2015 %I ACM %T Load Testing Elasticity and Performance Isolation in Shared Execution Environments

2014[ to top ]

Quantitative Evaluation of Service Dependability in Shared Execution Environments (Extended Abstract of Keynote Talk). S. Kounev; in Proceedings of the 11th International Conference on Quantitative Evaluation of SysTems (QEST 2014), Florence, Italy (2014).
- [ BibTeX ]
- [ EndNote ]
@inproceedings{Ko2014-QEST-Keynote, author = {Kounev, Samuel}, booktitle = {Proceedings of the 11th International Conference on Quantitative Evaluation of SysTems (QEST 2014), Florence, Italy}, keywords = {Virtualization}, month = {{September}}, title = {{Quantitative Evaluation of Service Dependability in Shared Execution Environments (Extended Abstract of Keynote Talk)}}, year = 2014 }

%0 Conference Paper %1 Ko2014-QEST-Keynote %A Kounev, Samuel %B Proceedings of the 11th International Conference on Quantitative Evaluation of SysTems (QEST 2014), Florence, Italy %D 2014 %T Quantitative Evaluation of Service Dependability in Shared Execution Environments (Extended Abstract of Keynote Talk)
Resource Elasticity Benchmarking in Cloud Environments. A. Weber; Thesis; Karlsruhe Institute of Technology (KIT); Am Fasanengarten 5, 76131 Karlsruhe, Germany. (2014, August).
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
Auto-scaling features offered by today's cloud infrastructures provide increased flexibility, especially for customers that experience high variations in the load intensity over time. However, auto-scaling features introduce new system quality attributes when considering their accuracy and timing. Therefore, distinguishing between different offerings has become a complex task, as it is not yet supported by reliable metrics and measurement approaches. This thesis discusses the shortcomings of existing approaches for measuring and evaluating elastic behavior and proposes a novel benchmark methodology specifically designed for evaluating the elasticity aspects of modern cloud platforms. The benchmarking concept uses open workloads with realistic load intensity profiles in order to induce resource demand variations on the benchmarked system and compares them with the actual variation of the allocated resources. To ensure a fair elasticity comparison between systems with di erent underlying hardware performance, the load intensity profiles are adjusted to induce identical resource demand variations on all compared platforms. Furthermore, this thesis proposes new metrics that capture the accuracy of resource allocations and deallocations, as well as the timing aspects of an auto-scaling mechanism, explicitly. The benchmark concept comprises four activities: The System Analysis evaluates the load processing capabilities of the benchmarked platform for different scaling stages. The Benchmark Calibration then uses the analysis results and adjusts a given load intensity profile in a system specific manner. Within the Measurement activity, the evaluated platformis exposed to a load varying ccording to the adjusted intensity profile. The final Elasticity Evaluation measures the quality of the observed elastic behavior using the proposed elasticity metrics. A java based framework for benchmarking the elasticity of IaaS cloud platforms called BUNGEE implements this concept and automates benchmarking activities. At the moment, BUNGEE allows to analyze the elasticity of CloudStack and Amazon Web Service (AWS) based clouds that scale CPU-bound virtual machines horizontally. Within an extensive evaluation, this thesis demonstrates the ability of the proposed elasticity metrics to consistently rank elastic systems on an ordinal scale. A case study that uses a realistic load profile, consisting of several millions of request submissions, exhibits the applicability of the benchmarking methodology for realistic scenarios. The case study is conducted on a private as well as on a public cloud and uses eleven different elasticity rule configurations and four instance types assigned to resources with different levels of effciency.

@mastersthesis{Weber2014, abstract = {{Auto-scaling features offered by today's cloud infrastructures provide increased flexibility, especially for customers that experience high variations in the load intensity over time. However, auto-scaling features introduce new system quality attributes when considering their accuracy and timing. Therefore, distinguishing between different offerings has become a complex task, as it is not yet supported by reliable metrics and measurement approaches. This thesis discusses the shortcomings of existing approaches for measuring and evaluating elastic behavior and proposes a novel benchmark methodology specifically designed for evaluating the elasticity aspects of modern cloud platforms. The benchmarking concept uses open workloads with realistic load intensity profiles in order to induce resource demand variations on the benchmarked system and compares them with the actual variation of the allocated resources. To ensure a fair elasticity comparison between systems with di erent underlying hardware performance, the load intensity profiles are adjusted to induce identical resource demand variations on all compared platforms. Furthermore, this thesis proposes new metrics that capture the accuracy of resource allocations and deallocations, as well as the timing aspects of an auto-scaling mechanism, explicitly. The benchmark concept comprises four activities: The System Analysis evaluates the load processing capabilities of the benchmarked platform for different scaling stages. The Benchmark Calibration then uses the analysis results and adjusts a given load intensity profile in a system specific manner. Within the Measurement activity, the evaluated platformis exposed to a load varying ccording to the adjusted intensity profile. The final Elasticity Evaluation measures the quality of the observed elastic behavior using the proposed elasticity metrics. A java based framework for benchmarking the elasticity of IaaS cloud platforms called BUNGEE implements this concept and automates benchmarking activities. At the moment, BUNGEE allows to analyze the elasticity of CloudStack and Amazon Web Service (AWS) based clouds that scale CPU-bound virtual machines horizontally. Within an extensive evaluation, this thesis demonstrates the ability of the proposed elasticity metrics to consistently rank elastic systems on an ordinal scale. A case study that uses a realistic load profile, consisting of several millions of request submissions, exhibits the applicability of the benchmarking methodology for realistic scenarios. The case study is conducted on a private as well as on a public cloud and uses eleven different elasticity rule configurations and four instance types assigned to resources with different levels of effciency.}}, address = {Am Fasanengarten 5, 76131 Karlsruhe, Germany}, author = {Weber, Andreas}, keywords = {Self-adaptive-systems}, month = {08}, school = {{Karlsruhe Institute of Technology (KIT)}}, title = {{Resource Elasticity Benchmarking in Cloud Environments}}, type = {{Master Thesis}}, year = 2014 }

%0 Thesis %1 Weber2014 %A Weber, Andreas %C Am Fasanengarten 5, 76131 Karlsruhe, Germany %D 2014 %T Resource Elasticity Benchmarking in Cloud Environments %X Auto-scaling features offered by today's cloud infrastructures provide increased flexibility, especially for customers that experience high variations in the load intensity over time. However, auto-scaling features introduce new system quality attributes when considering their accuracy and timing. Therefore, distinguishing between different offerings has become a complex task, as it is not yet supported by reliable metrics and measurement approaches. This thesis discusses the shortcomings of existing approaches for measuring and evaluating elastic behavior and proposes a novel benchmark methodology specifically designed for evaluating the elasticity aspects of modern cloud platforms. The benchmarking concept uses open workloads with realistic load intensity profiles in order to induce resource demand variations on the benchmarked system and compares them with the actual variation of the allocated resources. To ensure a fair elasticity comparison between systems with di erent underlying hardware performance, the load intensity profiles are adjusted to induce identical resource demand variations on all compared platforms. Furthermore, this thesis proposes new metrics that capture the accuracy of resource allocations and deallocations, as well as the timing aspects of an auto-scaling mechanism, explicitly. The benchmark concept comprises four activities: The System Analysis evaluates the load processing capabilities of the benchmarked platform for different scaling stages. The Benchmark Calibration then uses the analysis results and adjusts a given load intensity profile in a system specific manner. Within the Measurement activity, the evaluated platformis exposed to a load varying ccording to the adjusted intensity profile. The final Elasticity Evaluation measures the quality of the observed elastic behavior using the proposed elasticity metrics. A java based framework for benchmarking the elasticity of IaaS cloud platforms called BUNGEE implements this concept and automates benchmarking activities. At the moment, BUNGEE allows to analyze the elasticity of CloudStack and Amazon Web Service (AWS) based clouds that scale CPU-bound virtual machines horizontally. Within an extensive evaluation, this thesis demonstrates the ability of the proposed elasticity metrics to consistently rank elastic systems on an ordinal scale. A case study that uses a realistic load profile, consisting of several millions of request submissions, exhibits the applicability of the benchmarking methodology for realistic scenarios. The case study is conducted on a private as well as on a public cloud and uses eleven different elasticity rule configurations and four instance types assigned to resources with different levels of effciency.
Towards a Resource Elasticity Benchmark for Cloud Environments. A. Weber; N. R. Herbst; H. Groenda; S. Kounev; in Proceedings of the 2nd International Workshop on Hot Topics in Cloud Service Scalability (HotTopiCS 2014), co-located with the 5th ACM/SPEC International Conference on Performance Engineering (ICPE 2014) (2014). 5:1–5:8.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Auto-scaling features offered by today's cloud infrastructures provide increased flexibility especially for customers that experience high variations in the load intensity over time. However, auto-scaling features introduce new system quality attributes when considering their accuracy, timing, and boundaries. Therefore, distinguishing between different offerings has become a complex task, as it is not yet supported by reliable metrics and measurement approaches. In this paper, we discuss shortcomings of existing approaches for measuring and evaluating elastic behavior and propose a novel benchmark methodology specifically designed for evaluating the elasticity aspects of modern cloud platforms. The benchmark is based on open workloads with realistic load variation profiles that are calibrated to induce identical resource demand variations independent of the underlying hardware performance. Furthermore, we propose new metrics that capture the accuracy of resource allocations and de-allocations, as well as the timing aspects of an auto-scaling mechanism explicitly.

@inproceedings{WeHeGrKo2014-HotTopicsWS-ElaBench, abstract = {{Auto-scaling features offered by today's cloud infrastructures provide increased flexibility especially for customers that experience high variations in the load intensity over time. However, auto-scaling features introduce new system quality attributes when considering their accuracy, timing, and boundaries. Therefore, distinguishing between different offerings has become a complex task, as it is not yet supported by reliable metrics and measurement approaches. In this paper, we discuss shortcomings of existing approaches for measuring and evaluating elastic behavior and propose a novel benchmark methodology specifically designed for evaluating the elasticity aspects of modern cloud platforms. The benchmark is based on open workloads with realistic load variation profiles that are calibrated to induce identical resource demand variations independent of the underlying hardware performance. Furthermore, we propose new metrics that capture the accuracy of resource allocations and de-allocations, as well as the timing aspects of an auto-scaling mechanism explicitly.}}, address = {New York, NY, USA}, author = {Weber, Andreas and Herbst, Nikolas Roman and Groenda, Henning and Kounev, Samuel}, booktitle = {Proceedings of the 2nd International Workshop on Hot Topics in Cloud Service Scalability (HotTopiCS 2014), co-located with the 5th ACM/SPEC International Conference on Performance Engineering (ICPE 2014)}, keywords = {t_workshop}, month = {03}, pages = {5:1–5:8}, publisher = {ACM}, series = {HotTopiCS '14}, title = {{Towards a Resource Elasticity Benchmark for Cloud Environments}}, year = 2014 }

%0 Conference Paper %1 WeHeGrKo2014-HotTopicsWS-ElaBench %A Weber, Andreas %A Herbst, Nikolas Roman %A Groenda, Henning %A Kounev, Samuel %B Proceedings of the 2nd International Workshop on Hot Topics in Cloud Service Scalability (HotTopiCS 2014), co-located with the 5th ACM/SPEC International Conference on Performance Engineering (ICPE 2014) %C New York, NY, USA %D 2014 %I ACM %P 5:1--5:8 %T Towards a Resource Elasticity Benchmark for Cloud Environments %U http://doi.acm.org/10.1145/2649563.2649571 %X Auto-scaling features offered by today's cloud infrastructures provide increased flexibility especially for customers that experience high variations in the load intensity over time. However, auto-scaling features introduce new system quality attributes when considering their accuracy, timing, and boundaries. Therefore, distinguishing between different offerings has become a complex task, as it is not yet supported by reliable metrics and measurement approaches. In this paper, we discuss shortcomings of existing approaches for measuring and evaluating elastic behavior and propose a novel benchmark methodology specifically designed for evaluating the elasticity aspects of modern cloud platforms. The benchmark is based on open workloads with realistic load variation profiles that are calibrated to induce identical resource demand variations independent of the underlying hardware performance. Furthermore, we propose new metrics that capture the accuracy of resource allocations and de-allocations, as well as the timing aspects of an auto-scaling mechanism explicitly.

2013[ to top ]

Elasticity in Cloud Computing: What it is, and What it is Not. N. R. Herbst; S. Kounev; R. Reussner; in Proceedings of the 10th International Conference on Autonomic Computing (ICAC 2013) (2013).
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Originating from the field of physics and economics, the term elasticity is nowadays heavily used in the context of cloud computing. In this context, elasticity is commonly understood as the ability of a system to automatically provision and de-provision computing resources on demand as workloads change. However, elasticity still lacks a precise definition as well as representative metrics coupled with a benchmarking methodology to enable comparability of systems. Existing definitions of elasticity are largely inconsistent and unspecific leading to confusion in the use of the term and its differentiation from related terms such as scalability and efficiency; the proposed measurement methodologies do not provide means to quantify elasticity without mixing it with efficiency or scalability aspects. In this short paper, we propose a precise definition of elasticity and analyze its core properties and requirements explicitly distinguishing from related terms such as scalability, efficiency, and agility. Furthermore, we present a set of appropriate elasticity metrics and sketch a new elasticity tailored benchmarking methodology addressing the special requirements on workload design and calibration.

@inproceedings{HeKoRe2013-ICAC-Elasticity, abstract = {{Originating from the field of physics and economics, the term elasticity is nowadays heavily used in the context of cloud computing. In this context, elasticity is commonly understood as the ability of a system to automatically provision and de-provision computing resources on demand as workloads change. However, elasticity still lacks a precise definition as well as representative metrics coupled with a benchmarking methodology to enable comparability of systems. Existing definitions of elasticity are largely inconsistent and unspecific leading to confusion in the use of the term and its differentiation from related terms such as scalability and efficiency; the proposed measurement methodologies do not provide means to quantify elasticity without mixing it with efficiency or scalability aspects. In this short paper, we propose a precise definition of elasticity and analyze its core properties and requirements explicitly distinguishing from related terms such as scalability, efficiency, and agility. Furthermore, we present a set of appropriate elasticity metrics and sketch a new elasticity tailored benchmarking methodology addressing the special requirements on workload design and calibration.}}, author = {Herbst, Nikolas Roman and Kounev, Samuel and Reussner, Ralf}, booktitle = {Proceedings of the 10th International Conference on Autonomic Computing (ICAC 2013)}, keywords = {Self-adaptive-systems}, month = {06}, note = {Acceptance Rate (Short Paper): 36.9\%}, publisher = {USENIX}, title = {{Elasticity in Cloud Computing: What it is, and What it is Not}}, year = 2013 }

%0 Conference Paper %1 HeKoRe2013-ICAC-Elasticity %A Herbst, Nikolas Roman %A Kounev, Samuel %A Reussner, Ralf %B Proceedings of the 10th International Conference on Autonomic Computing (ICAC 2013) %D 2013 %I USENIX %T Elasticity in Cloud Computing: What it is, and What it is Not %U https://www.usenix.org/conference/icac13/elasticity-cloud-computing-what-it-and-what-it-not %X Originating from the field of physics and economics, the term elasticity is nowadays heavily used in the context of cloud computing. In this context, elasticity is commonly understood as the ability of a system to automatically provision and de-provision computing resources on demand as workloads change. However, elasticity still lacks a precise definition as well as representative metrics coupled with a benchmarking methodology to enable comparability of systems. Existing definitions of elasticity are largely inconsistent and unspecific leading to confusion in the use of the term and its differentiation from related terms such as scalability and efficiency; the proposed measurement methodologies do not provide means to quantify elasticity without mixing it with efficiency or scalability aspects. In this short paper, we propose a precise definition of elasticity and analyze its core properties and requirements explicitly distinguishing from related terms such as scalability, efficiency, and agility. Furthermore, we present a set of appropriate elasticity metrics and sketch a new elasticity tailored benchmarking methodology addressing the special requirements on workload design and calibration.

Your E-mail address:
Your Name (optional):

Update: This tool is no longer maintained and supported.

BUNGEE Cloud Elasticity Benchmark

Mailing List

Bildnachweise