LibReDE

Update: This tool is no longer maintained and supported.

Library for Resource Demand Estimation

When creating a performance model, it is necessary to quantify the amount of resources consumed by an application serving individual requests. In distributed enterprise systems, these resource demands usually cannot be observed directly, their estimation is a major challenge. Different statistical approaches to resource demand estimation based on monitoring data have been proposed, e.g., using linear regression or Kalman filtering techniques.

LibReDE is a library of ready-to-use implementations of state-of-the-art approaches to resource demand estimation that can be used for online and offline analysis. It is the first publicly available tool for this task and aims at supporting performance engineers during performance model construction. The library enables the quick comparison of the estimation accuracy of different approaches in a given context and thus helps to select an optimal one.

More information can be found on the following pages:

If you have any questions, please contact Simon Spinner or Johannes Grohmann.

Mailing List

To stay updated on our tools, please subscribe to our descartes-tools mailing list (low traffic, only announcements related to our tools)

Publications

2019[ to top ]

Utilizing Clustering to Optimize Resource Demand Estimation Approaches. J. Grohmann; S. Eismann; A. Bauer; M. Zuefle; N. Herbst; S. Kounev; in 2019 IEEE 4th International Workshops on Foundations and Applications of Self* Systems (FAS*W) (2019). 134–139.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
- [ DOI ]
Resource demands are crucial parameters for modeling and predicting the performance of software systems. Direct measurement of these resource demands is usually infeasible due to instrumentation overheads causing measurement interferences and perturbation in production environments. Thus, a number of statistical estimation approaches (e.g., based on optimization, regression or Kalman filters) have been proposed in the literature. Most of these approaches are parameterized. These parameters influence the estimation quality and the required computation time. Existing work uses historical data as training sets to optimize those parameters and to minimize the estimation error of those approaches. However, if the data traces are fundamentally different, the optimal parameter settings are different as well. In this paper, we propose to use automated clustering in order to group training sets into groups of similar optimization behavior. This way, optimization can be specifically tailored to certain groups of traces in a self-aware manner. During run-time, every trace is first sorted into a cluster, where the respective cluster-wide parameter optimum can be applied. A preliminary case study shows that clustering can provide promising improvements.

@inproceedings{GrEiBaZuHeKo2019-ICAC-RDEClustering, abstract = {Resource demands are crucial parameters for modeling and predicting the performance of software systems. Direct measurement of these resource demands is usually infeasible due to instrumentation overheads causing measurement interferences and perturbation in production environments. Thus, a number of statistical estimation approaches (e.g., based on optimization, regression or Kalman filters) have been proposed in the literature. Most of these approaches are parameterized. These parameters influence the estimation quality and the required computation time. Existing work uses historical data as training sets to optimize those parameters and to minimize the estimation error of those approaches. However, if the data traces are fundamentally different, the optimal parameter settings are different as well. In this paper, we propose to use automated clustering in order to group training sets into groups of similar optimization behavior. This way, optimization can be specifically tailored to certain groups of traces in a self-aware manner. During run-time, every trace is first sorted into a cluster, where the respective cluster-wide parameter optimum can be applied. A preliminary case study shows that clustering can provide promising improvements.}, author = {Grohmann, Johannes and Eismann, Simon and Bauer, Andre and Zuefle, Marwin and Herbst, Nikolas and Kounev, Samuel}, booktitle = {2019 IEEE 4th International Workshops on Foundations and Applications of Self* Systems (FAS*W)}, keywords = {Optimization}, month = {06}, pages = {134–139}, publisher = {IEEE}, title = {{Utilizing Clustering to Optimize Resource Demand Estimation Approaches}}, year = 2019 }

%0 Conference Paper %1 GrEiBaZuHeKo2019-ICAC-RDEClustering %A Grohmann, Johannes %A Eismann, Simon %A Bauer, Andre %A Zuefle, Marwin %A Herbst, Nikolas %A Kounev, Samuel %B 2019 IEEE 4th International Workshops on Foundations and Applications of Self* Systems (FAS*W) %D 2019 %I IEEE %P 134--139 %R 10.1109/FAS-W.2019.00043 %T Utilizing Clustering to Optimize Resource Demand Estimation Approaches %U https://doi.org/10.1109/FAS-W.2019.00043 %X Resource demands are crucial parameters for modeling and predicting the performance of software systems. Direct measurement of these resource demands is usually infeasible due to instrumentation overheads causing measurement interferences and perturbation in production environments. Thus, a number of statistical estimation approaches (e.g., based on optimization, regression or Kalman filters) have been proposed in the literature. Most of these approaches are parameterized. These parameters influence the estimation quality and the required computation time. Existing work uses historical data as training sets to optimize those parameters and to minimize the estimation error of those approaches. However, if the data traces are fundamentally different, the optimal parameter settings are different as well. In this paper, we propose to use automated clustering in order to group training sets into groups of similar optimization behavior. This way, optimization can be specifically tailored to certain groups of traces in a self-aware manner. During run-time, every trace is first sorted into a cluster, where the respective cluster-wide parameter optimum can be applied. A preliminary case study shows that clustering can provide promising improvements.
Chameleon: A Hybrid, Proactive Auto-Scaling Mechanism on a Level-Playing Field. A. Bauer; (2019, Juni).
- [ BibTeX ]
- [ EndNote ]
@misc{talk-Bauer-HotCloudPerf2019, author = {Bauer, Andr{\'e}}, howpublished = {Talk at HotCloudperf 2019}, keywords = {Chameleon}, month = {06}, title = {{Chameleon: A Hybrid, Proactive Auto-Scaling Mechanism on a Level-Playing Field}}, year = 2019 }

%0 Generic %1 talk-Bauer-HotCloudPerf2019 %A Bauer, André %D 2019 %T Chameleon: A Hybrid, Proactive Auto-Scaling Mechanism on a Level-Playing Field
Online model learning for self-aware computing infrastructures. S. Spinner; J. Grohmann; S. Eismann; S. Kounev; in Journal of Systems and Software (2019). 147 1–16.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
- [ DOI ]
Performance models are valuable and powerful tools for performance prediction. However, the creation of performance models usually requires significant manual effort. Furthermore, as the modeled structures are subject to frequent change in modern infrastructures, such performance models need to be adapted as well. We therefore propose a reference architecture for online model learning in virtualized environments, which enables the automatic extraction of the aforementioned performance models. We follow an agent-based approach, which enables us to incorporate the extraction of information about the application structure as well as the virtualization structures present in modern computing centers. Our evaluation shows that our collaborating agents are able to reduce the manual effort of performance model extraction by 85.4%. The resulting performance model is able to predict the system utilization with an absolute error of less than 4% and the end-to-end response time with a relative error of less than 21%.

@article{SpGrEiKo2019-JSS-ModelLearning, abstract = {Performance models are valuable and powerful tools for performance prediction. However, the creation of performance models usually requires significant manual effort. Furthermore, as the modeled structures are subject to frequent change in modern infrastructures, such performance models need to be adapted as well. We therefore propose a reference architecture for online model learning in virtualized environments, which enables the automatic extraction of the aforementioned performance models. We follow an agent-based approach, which enables us to incorporate the extraction of information about the application structure as well as the virtualization structures present in modern computing centers. Our evaluation shows that our collaborating agents are able to reduce the manual effort of performance model extraction by 85.4%. The resulting performance model is able to predict the system utilization with an absolute error of less than 4% and the end-to-end response time with a relative error of less than 21%.}, author = {Spinner, Simon and Grohmann, Johannes and Eismann, Simon and Kounev, Samuel}, journal = {Journal of Systems and Software}, keywords = {Virtualization}, pages = {1–16}, title = {Online model learning for self-aware computing infrastructures}, volume = 147, year = 2019 }

%0 Journal Article %1 SpGrEiKo2019-JSS-ModelLearning %A Spinner, Simon %A Grohmann, Johannes %A Eismann, Simon %A Kounev, Samuel %D 2019 %J Journal of Systems and Software %P 1--16 %R 10.1016/j.jss.2018.09.089 %T Online model learning for self-aware computing infrastructures %U https://doi.org/10.1016/j.jss.2018.09.089 %V 147 %X Performance models are valuable and powerful tools for performance prediction. However, the creation of performance models usually requires significant manual effort. Furthermore, as the modeled structures are subject to frequent change in modern infrastructures, such performance models need to be adapted as well. We therefore propose a reference architecture for online model learning in virtualized environments, which enables the automatic extraction of the aforementioned performance models. We follow an agent-based approach, which enables us to incorporate the extraction of information about the application structure as well as the virtualization structures present in modern computing centers. Our evaluation shows that our collaborating agents are able to reduce the manual effort of performance model extraction by 85.4%. The resulting performance model is able to predict the system utilization with an absolute error of less than 4% and the end-to-end response time with a relative error of less than 21%.

2018[ to top ]

Using Machine Learning for Recommending Service Demand Estimation Approaches. J. Grohmann; N. Herbst; S. Spinner; S. Kounev; in Proceedings of the 8th International Conference on Cloud Computing and Services Science (CLOSER 2018) (2018). 473–480.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
- [ DOI ]
Service demands are key parameters in service and performance modeling. Hence, a variety of different approaches to service demand estimation exist in the literature. However, given a specific scenario, it is not trivial to select the currently best approach, since deep expertise in statistical estimation techniques is required and the requirements and characteristics of the application scenario might change over time (e.g., by varying load patterns). To tackle this problem, we propose the use of machine learning techniques to automatically recommend the best suitable approach for the target scenario. The approach works in an online fashion and can incorporate new measurement data and changing characteristics on-the-fly. Preliminary results show that executing only the recommended estimation approach achieves 99.6% accuracy compared to executing all approaches available, while speeding up the estimation time by 57%.

@inproceedings{GrHeSpKo-Closer18-RecServDemEst, abstract = {Service demands are key parameters in service and performance modeling. Hence, a variety of different approaches to service demand estimation exist in the literature. However, given a specific scenario, it is not trivial to select the currently best approach, since deep expertise in statistical estimation techniques is required and the requirements and characteristics of the application scenario might change over time (e.g., by varying load patterns). To tackle this problem, we propose the use of machine learning techniques to automatically recommend the best suitable approach for the target scenario. The approach works in an online fashion and can incorporate new measurement data and changing characteristics on-the-fly. Preliminary results show that executing only the recommended estimation approach achieves 99.6% accuracy compared to executing all approaches available, while speeding up the estimation time by 57%.}, author = {Grohmann, Johannes and Herbst, Nikolas and Spinner, Simon and Kounev, Samuel}, booktitle = {Proceedings of the 8th International Conference on Cloud Computing and Services Science (CLOSER 2018)}, keywords = {Optimization}, month = {03}, organization = {INSTICC}, pages = {473–480}, publisher = {SciTePress}, title = {Using Machine Learning for Recommending Service Demand Estimation Approaches}, year = 2018 }

%0 Conference Paper %1 GrHeSpKo-Closer18-RecServDemEst %A Grohmann, Johannes %A Herbst, Nikolas %A Spinner, Simon %A Kounev, Samuel %B Proceedings of the 8th International Conference on Cloud Computing and Services Science (CLOSER 2018) %D 2018 %I SciTePress %P 473--480 %R 10.5220/0006761104730480 %T Using Machine Learning for Recommending Service Demand Estimation Approaches %U https://doi.org/10.5220/0006761104730480 %X Service demands are key parameters in service and performance modeling. Hence, a variety of different approaches to service demand estimation exist in the literature. However, given a specific scenario, it is not trivial to select the currently best approach, since deep expertise in statistical estimation techniques is required and the requirements and characteristics of the application scenario might change over time (e.g., by varying load patterns). To tackle this problem, we propose the use of machine learning techniques to automatically recommend the best suitable approach for the target scenario. The approach works in an online fashion and can incorporate new measurement data and changing characteristics on-the-fly. Preliminary results show that executing only the recommended estimation approach achieves 99.6% accuracy compared to executing all approaches available, while speeding up the estimation time by 57%. %@ 978-989-758-295-0
On the Value of Service Demand Estimation for Auto-Scaling. A. Bauer; J. Grohmann; N. Herbst; S. Kounev; in Proceedings of 19th International GI/ITG Conference on Measurement, Modelling and Evaluation of Computing Systems (MMB 2018) (2018). (Bd. 10740) 142–156.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
- [ DOI ]
In the context of performance models, service demands are key model parameters capturing the average time individual requests of different workload classes are actively processed. In a system under load, due to measurement interference, service demands normally cannot be measured directly, however, a number of estimation approaches exist based on high-level performance metrics. In this paper, we show that service demands provide significant benefits for implementing modern auto-scalers. Auto-scaling describes the process of dynamically adjusting the number of allocated virtual resources (e.g., virtual machines) in a data center according to the incoming workload. We demonstrate that even a simple auto-scaler that leverages information about service demands significantly outperforms auto-scalers solely based on CPU utilization measurements. This is shown by testing two approaches in three different scenarios. Our results show that the service demand-based auto-scaler outperforms the CPU utilization-based one in all scenarios. Our results encourage further research on the application of service demand estimates for resource management in data centers.

@inproceedings{BaGrHeKo2018-MMB-ServiceDemand, abstract = {In the context of performance models, service demands are key model parameters capturing the average time individual requests of different workload classes are actively processed. In a system under load, due to measurement interference, service demands normally cannot be measured directly, however, a number of estimation approaches exist based on high-level performance metrics. In this paper, we show that service demands provide significant benefits for implementing modern auto-scalers. Auto-scaling describes the process of dynamically adjusting the number of allocated virtual resources (e.g., virtual machines) in a data center according to the incoming workload. We demonstrate that even a simple auto-scaler that leverages information about service demands significantly outperforms auto-scalers solely based on CPU utilization measurements. This is shown by testing two approaches in three different scenarios. Our results show that the service demand-based auto-scaler outperforms the CPU utilization-based one in all scenarios. Our results encourage further research on the application of service demand estimates for resource management in data centers.}, address = {Cham}, author = {Bauer, Andr{\'e} and Grohmann, Johannes and Herbst, Nikolas and Kounev, Samuel}, booktitle = {Proceedings of 19th International GI/ITG Conference on Measurement, Modelling and Evaluation of Computing Systems (MMB 2018)}, keywords = {Virtualization}, month = {02}, pages = {142–156}, publisher = {Springer}, series = {Lecture Notes in Computer Science}, title = {{On the Value of Service Demand Estimation for Auto-Scaling}}, volume = 10740, year = 2018 }

%0 Conference Paper %1 BaGrHeKo2018-MMB-ServiceDemand %A Bauer, André %A Grohmann, Johannes %A Herbst, Nikolas %A Kounev, Samuel %B Proceedings of 19th International GI/ITG Conference on Measurement, Modelling and Evaluation of Computing Systems (MMB 2018) %C Cham %D 2018 %I Springer %P 142--156 %R 10.1007/978-3-319-74947-1_10 %T On the Value of Service Demand Estimation for Auto-Scaling %U https://doi.org/10.1007/978-3-319-74947-1_10 %V 10740 %X In the context of performance models, service demands are key model parameters capturing the average time individual requests of different workload classes are actively processed. In a system under load, due to measurement interference, service demands normally cannot be measured directly, however, a number of estimation approaches exist based on high-level performance metrics. In this paper, we show that service demands provide significant benefits for implementing modern auto-scalers. Auto-scaling describes the process of dynamically adjusting the number of allocated virtual resources (e.g., virtual machines) in a data center according to the incoming workload. We demonstrate that even a simple auto-scaler that leverages information about service demands significantly outperforms auto-scalers solely based on CPU utilization measurements. This is shown by testing two approaches in three different scenarios. Our results show that the service demand-based auto-scaler outperforms the CPU utilization-based one in all scenarios. Our results encourage further research on the application of service demand estimates for resource management in data centers. %@ 978-3-319-74947-1

2017[ to top ]

Self-Tuning Resource Demand Estimation. J. Grohmann; N. Herbst; S. Spinner; S. Kounev; in Proceedings of the 14th IEEE International Conference on Autonomic Computing (ICAC 2017) (2017). 21–26.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
- [ DOI ]
The average time a resource needs to process incoming requests in a monitored workload mix is a key parameter of stochastic performance models. Direct measurement of these resource demands is usually infeasible due to instrumentation overheads causing measurement interferences and perturbation in production environments.Thus, a number of statistical estimation approaches (e.g., based on optimization, regression or Kalman filters) have been proposed in the literature each coming with different strengths and run-time overheads. Most approaches offer parameters in order to customize the behavior of the estimator influencing the estimation quality and the required computation time. However, their configuration usually requires exhaustive testing, as default parameters normally do not provide optimal performance.In this paper, we propose a self-tuning approach based on discrete optimization that can be used to automatically tune the parameters of resource demand estimation methods, tailoring them to the specific application scenario and thus improving their accuracy. We apply and compare different techniques on a representative data set with varying load levels and number of workload classes. We show that our selected approach for parameter tuning can automatically improve the estimation quality of certain estimators by up to 25%.

@inproceedings{GrHeSpKo2017-ICAC-RDE, abstract = {The average time a resource needs to process incoming requests in a monitored workload mix is a key parameter of stochastic performance models. Direct measurement of these resource demands is usually infeasible due to instrumentation overheads causing measurement interferences and perturbation in production environments.Thus, a number of statistical estimation approaches (e.g., based on optimization, regression or Kalman filters) have been proposed in the literature each coming with different strengths and run-time overheads. Most approaches offer parameters in order to customize the behavior of the estimator influencing the estimation quality and the required computation time. However, their configuration usually requires exhaustive testing, as default parameters normally do not provide optimal performance.In this paper, we propose a self-tuning approach based on discrete optimization that can be used to automatically tune the parameters of resource demand estimation methods, tailoring them to the specific application scenario and thus improving their accuracy. We apply and compare different techniques on a representative data set with varying load levels and number of workload classes. We show that our selected approach for parameter tuning can automatically improve the estimation quality of certain estimators by up to 25%.}, author = {Grohmann, Johannes and Herbst, Nikolas and Spinner, Simon and Kounev, Samuel}, booktitle = {Proceedings of the 14th IEEE International Conference on Autonomic Computing (ICAC 2017)}, keywords = {Self-adaptive-systems}, month = {07}, pages = {21–26}, title = {{Self-Tuning Resource Demand Estimation}}, year = 2017 }

%0 Conference Paper %1 GrHeSpKo2017-ICAC-RDE %A Grohmann, Johannes %A Herbst, Nikolas %A Spinner, Simon %A Kounev, Samuel %B Proceedings of the 14th IEEE International Conference on Autonomic Computing (ICAC 2017) %D 2017 %P 21--26 %R 10.1109/ICAC.2017.19 %T Self-Tuning Resource Demand Estimation %U https://doi.org/10.1109/ICAC.2017.19 %X The average time a resource needs to process incoming requests in a monitored workload mix is a key parameter of stochastic performance models. Direct measurement of these resource demands is usually infeasible due to instrumentation overheads causing measurement interferences and perturbation in production environments.Thus, a number of statistical estimation approaches (e.g., based on optimization, regression or Kalman filters) have been proposed in the literature each coming with different strengths and run-time overheads. Most approaches offer parameters in order to customize the behavior of the estimator influencing the estimation quality and the required computation time. However, their configuration usually requires exhaustive testing, as default parameters normally do not provide optimal performance.In this paper, we propose a self-tuning approach based on discrete optimization that can be used to automatically tune the parameters of resource demand estimation methods, tailoring them to the specific application scenario and thus improving their accuracy. We apply and compare different techniques on a representative data set with varying load levels and number of workload classes. We show that our selected approach for parameter tuning can automatically improve the estimation quality of certain estimators by up to 25%.
Design and Evaluation of a Proactive, Application-Aware Auto-Scaler. A. Bauer; N. Herbst; S. Kounev; in Proceedings of the 8th ACM/SPEC International Conference on Performance Engineering (ICPE 2017) (2017).
- [ BibTeX ]
- [ EndNote ]
@inproceedings{BaHeKo2017-ICPE-Auto-Scaler-Tutorial, author = {Bauer, Andr{\'e} and Herbst, Nikolas and Kounev, Samuel}, booktitle = {{Proceedings of the 8th ACM/SPEC International Conference on Performance Engineering (ICPE 2017)}}, keywords = {LIMBO}, month = {04}, title = {{Design and Evaluation of a Proactive, Application-Aware Auto-Scaler}}, year = 2017 }

%0 Conference Paper %1 BaHeKo2017-ICPE-Auto-Scaler-Tutorial %A Bauer, André %A Herbst, Nikolas %A Kounev, Samuel %B Proceedings of the 8th ACM/SPEC International Conference on Performance Engineering (ICPE 2017) %D 2017 %T Design and Evaluation of a Proactive, Application-Aware Auto-Scaler

2016[ to top ]

Design and Evaluation of a Proactive, Application-Aware Elasticity Mechanism. A. Bauer; (2016, November).
- [ BibTeX ]
- [ EndNote ]
@misc{talk-Bauer-SSP2016, author = {Bauer, Andr{\'e}}, howpublished = {Experience Talk at SSP 2016}, keywords = {BUNGEE}, month = 11, title = {{Design and Evaluation of a Proactive, Application-Aware Elasticity Mechanism}}, year = 2016 }

%0 Generic %1 talk-Bauer-SSP2016 %A Bauer, André %D 2016 %T Design and Evaluation of a Proactive, Application-Aware Elasticity Mechanism
Chameleon: Design and Evaluation of a Proactive, Application-Aware Elasticity Mechanism. A. Bauer; (2016, Oktober).
- [ BibTeX ]
- [ EndNote ]
@misc{talk-Bauer-InvSpec16, author = {Bauer, Andr{\'e}}, howpublished = {Invited Talk, Spec Research - Cloud Working Group}, keywords = {Application-aware}, month = 10, title = {{Chameleon: Design and Evaluation of a Proactive, Application-Aware Elasticity Mechanism}}, year = 2016 }

%0 Generic %1 talk-Bauer-InvSpec16 %A Bauer, André %D 2016 %T Chameleon: Design and Evaluation of a Proactive, Application-Aware Elasticity Mechanism
Automated Parameterization of Performance Models from Measurements. G. Casale; S. Spinner; W. Wang; in Proceedings of the 7th ACM/SPEC International Conference on Performance Engineering (ICPE 2016) (2016).
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
Estimating parameters of performance models from empirical measurements is a critical task, which often has a major influence on the predictive accuracy of a model. This tutorial presents the problem of parameter estimation in queueing systems and queueing networks. The focus is on reliable estimation of the em arrival rates of the requests and of the em service demands they place at the servers. The tutorial covers common estimation techniques such as regression methods, maximum-likelihood estimation, and moment-matching, discussing their sensitivity with respect to data and model characteristics. The tutorial also demonstrates the automated estimation of model parameters using new open source tools.

@inproceedings{CaSpWa2016-ICPE-AutomatedParameterizationTutorial, abstract = {Estimating parameters of performance models from empirical measurements is a critical task, which often has a major influence on the predictive accuracy of a model. This tutorial presents the problem of parameter estimation in queueing systems and queueing networks. The focus is on reliable estimation of the {\em arrival rates} of the requests and of the {\em service demands} they place at the servers. The tutorial covers common estimation techniques such as regression methods, maximum-likelihood estimation, and moment-matching, discussing their sensitivity with respect to data and model characteristics. The tutorial also demonstrates the automated estimation of model parameters using new open source tools.}, author = {Casale, Giuliano and Spinner, Simon and Wang, Weikun}, booktitle = {{Proceedings of the 7th ACM/SPEC International Conference on Performance Engineering (ICPE 2016)}}, keywords = {LibReDE}, month = {03}, title = {{Automated Parameterization of Performance Models from Measurements}}, year = 2016 }

%0 Conference Paper %1 CaSpWa2016-ICPE-AutomatedParameterizationTutorial %A Casale, Giuliano %A Spinner, Simon %A Wang, Weikun %B Proceedings of the 7th ACM/SPEC International Conference on Performance Engineering (ICPE 2016) %D 2016 %T Automated Parameterization of Performance Models from Measurements %X Estimating parameters of performance models from empirical measurements is a critical task, which often has a major influence on the predictive accuracy of a model. This tutorial presents the problem of parameter estimation in queueing systems and queueing networks. The focus is on reliable estimation of the em arrival rates of the requests and of the em service demands they place at the servers. The tutorial covers common estimation techniques such as regression methods, maximum-likelihood estimation, and moment-matching, discussing their sensitivity with respect to data and model characteristics. The tutorial also demonstrates the automated estimation of model parameters using new open source tools.
A Reference Architecture for Online Performance Model Extraction in Virtualized Environments. S. Spinner; J. Walter; S. Kounev; in Proceedings of the 2016 Workshop on Challenges in Performance Methods for Software Development (WOSP-C’16) co-located with 7th ACM/SPEC International Conference on Performance Engineering (ICPE 2016) (2016).
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
Performance models can support decisions throughout the life-cycle of a software system. However, the manual construction of such performance models is a complex and time-consuming task requiring deep system knowledge. Therefore, automatic approaches for creating and updating performance models of a running system are necessary. Existing work focuses on single aspects of model extraction or proposes approaches specifically designed for a certain technology stack. In virtualized environments, we often see different applications based on diverse technology stacks sharing the same infrastructure. In order to enable online performance model extraction in such environments, we describe a new reference architecture for integrating different specialized model extraction solutions.

@inproceedings{SpWaKo2015-WOSPC-ReferenceArchitecture, abstract = {Performance models can support decisions throughout the life-cycle of a software system. However, the manual construction of such performance models is a complex and time-consuming task requiring deep system knowledge. Therefore, automatic approaches for creating and updating performance models of a running system are necessary. Existing work focuses on single aspects of model extraction or proposes approaches specifically designed for a certain technology stack. In virtualized environments, we often see different applications based on diverse technology stacks sharing the same infrastructure. In order to enable online performance model extraction in such environments, we describe a new reference architecture for integrating different specialized model extraction solutions.}, author = {Spinner, Simon and Walter, J{\"u}rgen and Kounev, Samuel}, booktitle = {{Proceedings of the 2016 Workshop on Challenges in Performance Methods for Software Development (WOSP-C'16) co-located with 7th ACM/SPEC International Conference on Performance Engineering (ICPE 2016)}}, keywords = {t_workshop}, month = {03}, title = {{A Reference Architecture for Online Performance Model Extraction in Virtualized Environments}}, year = 2016 }

%0 Conference Paper %1 SpWaKo2015-WOSPC-ReferenceArchitecture %A Spinner, Simon %A Walter, Jürgen %A Kounev, Samuel %B Proceedings of the 2016 Workshop on Challenges in Performance Methods for Software Development (WOSP-C'16) co-located with 7th ACM/SPEC International Conference on Performance Engineering (ICPE 2016) %D 2016 %T A Reference Architecture for Online Performance Model Extraction in Virtualized Environments %X Performance models can support decisions throughout the life-cycle of a software system. However, the manual construction of such performance models is a complex and time-consuming task requiring deep system knowledge. Therefore, automatic approaches for creating and updating performance models of a running system are necessary. Existing work focuses on single aspects of model extraction or proposes approaches specifically designed for a certain technology stack. In virtualized environments, we often see different applications based on diverse technology stacks sharing the same infrastructure. In order to enable online performance model extraction in such environments, we describe a new reference architecture for integrating different specialized model extraction solutions.

2015[ to top ]

Comparing the Accuracy of Resource Demand Measurement and Estimation Techniques. F. Willnecker; M. Dlugi; A. Brunnert; S. Spinner; S. Kounev; H. Krcmar; in Computer Performance Engineering - Proceedings of the 12th European Workshop (EPEW 2015), M. Beltrán, W. Knottenbelt, J. Bradley (Hrsg.) (2015). (Bd. 9272) 115–129.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
Resource demands are a core aspect of performance models. They describe how an operation utilizes a resource and therefore influence the systems performance metrics: response time, resource utilization and throughput. Such demands can be determined by two extraction classes: direct measurement or demand estimation. Selecting the best suited technique depends on available tools, acceptable measurement overhead and the level of granularity necessary for the performance model. This work compares two direct measurement techniques and an adaptive estimation technique based on multiple statistical approaches to evaluate strengths and weaknesses of each technique.We conduct a series of experiments using the SPECjEnterprise2010 industry benchmark and an automatic performance model generator for architecture-level performance models based on the Palladio Component Model. To compare the techniques we conduct two experiments with different levels of granularity on a standalone system, followed by one experiment using a distributed SPECjEnterprise2010 deployment combining both extraction classes for generating a full-stack performance model.

@inproceedings{WiDlBrSpKoKr2015-EPEW-LibredePMWT, abstract = {Resource demands are a core aspect of performance models. They describe how an operation utilizes a resource and therefore influence the systems performance metrics: response time, resource utilization and throughput. Such demands can be determined by two extraction classes: direct measurement or demand estimation. Selecting the best suited technique depends on available tools, acceptable measurement overhead and the level of granularity necessary for the performance model. This work compares two direct measurement techniques and an adaptive estimation technique based on multiple statistical approaches to evaluate strengths and weaknesses of each technique.We conduct a series of experiments using the SPECjEnterprise2010 industry benchmark and an automatic performance model generator for architecture-level performance models based on the Palladio Component Model. To compare the techniques we conduct two experiments with different levels of granularity on a standalone system, followed by one experiment using a distributed SPECjEnterprise2010 deployment combining both extraction classes for generating a full-stack performance model.}, author = {Willnecker, Felix and Dlugi, Markus and Brunnert, Andreas and Spinner, Simon and Kounev, Samuel and Krcmar, Helmut}, booktitle = {Computer Performance Engineering - Proceedings of the 12th European Workshop (EPEW 2015)}, editor = {Beltr{\'a}n, Marta and Knottenbelt, William and Bradley, Jeremy}, keywords = {t_workshop}, month = {{August}}, pages = {115-129}, publisher = {Springer}, series = {Lecture Notes in Computer Science}, title = {{Comparing the Accuracy of Resource Demand Measurement and Estimation Techniques}}, volume = 9272, year = 2015 }

%0 Conference Paper %1 WiDlBrSpKoKr2015-EPEW-LibredePMWT %A Willnecker, Felix %A Dlugi, Markus %A Brunnert, Andreas %A Spinner, Simon %A Kounev, Samuel %A Krcmar, Helmut %B Computer Performance Engineering - Proceedings of the 12th European Workshop (EPEW 2015) %D 2015 %E Beltrán, Marta %E Knottenbelt, William %E Bradley, Jeremy %I Springer %P 115-129 %T Comparing the Accuracy of Resource Demand Measurement and Estimation Techniques %V 9272 %X Resource demands are a core aspect of performance models. They describe how an operation utilizes a resource and therefore influence the systems performance metrics: response time, resource utilization and throughput. Such demands can be determined by two extraction classes: direct measurement or demand estimation. Selecting the best suited technique depends on available tools, acceptable measurement overhead and the level of granularity necessary for the performance model. This work compares two direct measurement techniques and an adaptive estimation technique based on multiple statistical approaches to evaluate strengths and weaknesses of each technique.We conduct a series of experiments using the SPECjEnterprise2010 industry benchmark and an automatic performance model generator for architecture-level performance models based on the Palladio Component Model. To compare the techniques we conduct two experiments with different levels of granularity on a standalone system, followed by one experiment using a distributed SPECjEnterprise2010 deployment combining both extraction classes for generating a full-stack performance model.
Evaluating Approaches to Resource Demand Estimation. S. Spinner; G. Casale; F. Brosig; S. Kounev; in Performance Evaluation (2015). 92 51–71.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
Resource demands are a key parameter of stochastic performance models that needs to be determined when performing a quantitative performance analysis of a system. However, the direct measurement of resource demands is not feasible in most realistic systems. Therefore, statistical approaches that estimate resource demands based on coarse-grained monitoring data (e.g., CPU utilization, and response times) have been proposed in the literature. These approaches have different assumptions and characteristics that need to be considered when estimating resource demands. This paper surveys the state-of-the-art in resource demand estimation and proposes a classification scheme for estimation approaches. Furthermore, it contains an experimental evaluation comparing the impact of different factors (monitoring window size, number of workload classes, load level, collinearity, and model mismatch) on the estimation accuracy of seven different approaches. The classification scheme and the experimental comparison helps performance engineers to select an approach to resource demand estimation that fulfills the requirements of a given analysis scenario.

@article{SpCaBrKo2015-PEVA-ResDemEstSurvey, abstract = {Resource demands are a key parameter of stochastic performance models that needs to be determined when performing a quantitative performance analysis of a system. However, the direct measurement of resource demands is not feasible in most realistic systems. Therefore, statistical approaches that estimate resource demands based on coarse-grained monitoring data (e.g., CPU utilization, and response times) have been proposed in the literature. These approaches have different assumptions and characteristics that need to be considered when estimating resource demands. This paper surveys the state-of-the-art in resource demand estimation and proposes a classification scheme for estimation approaches. Furthermore, it contains an experimental evaluation comparing the impact of different factors (monitoring window size, number of workload classes, load level, collinearity, and model mismatch) on the estimation accuracy of seven different approaches. The classification scheme and the experimental comparison helps performance engineers to select an approach to resource demand estimation that fulfills the requirements of a given analysis scenario.}, author = {Spinner, Simon and Casale, Giuliano and Brosig, Fabian and Kounev, Samuel}, journal = {Performance Evaluation}, keywords = {Instrumentation_profiling_and_workload_characterization}, month = 10, pages = {51–71}, publisher = {Elsevier B.V.}, title = {{Evaluating Approaches to Resource Demand Estimation}}, volume = 92, year = 2015 }

%0 Journal Article %1 SpCaBrKo2015-PEVA-ResDemEstSurvey %A Spinner, Simon %A Casale, Giuliano %A Brosig, Fabian %A Kounev, Samuel %D 2015 %I Elsevier B.V. %J Performance Evaluation %P 51--71 %T Evaluating Approaches to Resource Demand Estimation %U http://www.sciencedirect.com/science/article/pii/S0166531615000711 %V 92 %X Resource demands are a key parameter of stochastic performance models that needs to be determined when performing a quantitative performance analysis of a system. However, the direct measurement of resource demands is not feasible in most realistic systems. Therefore, statistical approaches that estimate resource demands based on coarse-grained monitoring data (e.g., CPU utilization, and response times) have been proposed in the literature. These approaches have different assumptions and characteristics that need to be considered when estimating resource demands. This paper surveys the state-of-the-art in resource demand estimation and proposes a classification scheme for estimation approaches. Furthermore, it contains an experimental evaluation comparing the impact of different factors (monitoring window size, number of workload classes, load level, collinearity, and model mismatch) on the estimation accuracy of seven different approaches. The classification scheme and the experimental comparison helps performance engineers to select an approach to resource demand estimation that fulfills the requirements of a given analysis scenario.

2014[ to top ]

LibReDE: A Library for Resource Demand Estimation. S. Spinner; J. Walter; (2014, November).
- [ BibTeX ]
- [ EndNote ]
@misc{talk-Spinner-SOSP14, author = {Spinner, Simon and Walter, J{\"u}rgen}, howpublished = {Technical Talk at Symposion on Software Performance (SOSP) 2014}, keywords = {Talks}, month = 11, title = {{LibReDE: A Library for Resource Demand Estimation}}, year = 2014 }

%0 Generic %1 talk-Spinner-SOSP14 %A Spinner, Simon %A Walter, Jürgen %D 2014 %T LibReDE: A Library for Resource Demand Estimation
LibReDE: A Library for Resource Demand Estimation. S. Spinner; G. Casale; X. Zhu; S. Kounev; in Proceedings of the 5th ACM/SPEC International Conference on Performance Engineering (ICPE 2014) (2014). 227–228.
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
- [ URL ]
When creating a performance model, it is necessary to quantify the amount of resources consumed by an application serving individual requests. In distributed enterprise systems, these resource demands usually cannot be observed directly, their estimation is a major challenge. Different statistical approaches to resource demand estimation based on monitoring data have been proposed, e.g., using linear regression or Kalman filtering techniques. In this paper, we present LibReDE, a library of ready-to-use implementations of approaches to resource demand estimation that can be used for online and offline analysis. It is the first publicly available tool for this task and aims at supporting performance engineers during performance model construction. The library enables the quick comparison of the estimation accuracy of different approaches in a given context and thus helps to select an optimal one.

@inproceedings{SpCaZhKo2014-ICPEDemo-LibReDE, abstract = {When creating a performance model, it is necessary to quantify the amount of resources consumed by an application serving individual requests. In distributed enterprise systems, these resource demands usually cannot be observed directly, their estimation is a major challenge. Different statistical approaches to resource demand estimation based on monitoring data have been proposed, e.g., using linear regression or Kalman filtering techniques. In this paper, we present LibReDE, a library of ready-to-use implementations of approaches to resource demand estimation that can be used for online and offline analysis. It is the first publicly available tool for this task and aims at supporting performance engineers during performance model construction. The library enables the quick comparison of the estimation accuracy of different approaches in a given context and thus helps to select an optimal one.}, address = {New York, NY, USA}, author = {Spinner, Simon and Casale, Giuliano and Zhu, Xiaoyun and Kounev, Samuel}, booktitle = {Proceedings of the 5th ACM/SPEC International Conference on Performance Engineering (ICPE 2014)}, keywords = {Instrumentation_profiling_and_workload_characterization}, month = {03}, pages = {227–228}, publisher = {ACM Press}, title = {{LibReDE: A Library for Resource Demand Estimation}}, year = 2014 }

%0 Conference Paper %1 SpCaZhKo2014-ICPEDemo-LibReDE %A Spinner, Simon %A Casale, Giuliano %A Zhu, Xiaoyun %A Kounev, Samuel %B Proceedings of the 5th ACM/SPEC International Conference on Performance Engineering (ICPE 2014) %C New York, NY, USA %D 2014 %I ACM Press %P 227--228 %T LibReDE: A Library for Resource Demand Estimation %U http://doi.acm.org/10.1145/2568088.2576093 %X When creating a performance model, it is necessary to quantify the amount of resources consumed by an application serving individual requests. In distributed enterprise systems, these resource demands usually cannot be observed directly, their estimation is a major challenge. Different statistical approaches to resource demand estimation based on monitoring data have been proposed, e.g., using linear regression or Kalman filtering techniques. In this paper, we present LibReDE, a library of ready-to-use implementations of approaches to resource demand estimation that can be used for online and offline analysis. It is the first publicly available tool for this task and aims at supporting performance engineers during performance model construction. The library enables the quick comparison of the estimation accuracy of different approaches in a given context and thus helps to select an optimal one.

2011[ to top ]

Evaluating Approaches to Resource Demand Estimation. S. Spinner; Thesis; Karlsruhe Institute of Technology (KIT); Am Fasanengarten 5, 76131 Karlsruhe, Germany. (2011, Juli).
- [ Abstract ]
- [ BibTeX ]
- [ EndNote ]
Proactively managing the performance and resource efficiency of running software systems requires techniques to predict system performance and resource consumption. Typically, performance predictions are based on performance models that capture the performance-relevant aspects of the considered software system. Building performance models involves the estimation of resource demands, i.e., estimating the time a unit of work spends obtaining service from a resource. A number of approaches to estimating the resource demands of a system already exist, e.g., based on regression analysis or stochastic filtering. These approaches differ in their accuracy, their robustness and their applicability. For instance, there are notable differences in the amount and type of measurement data that is required as input. However, to the best of our knowledge, a comprehensive evaluation and comparison of these approaches in a representative context does not exist. In this thesis, we give an overview of the state-of-the-art in resource demand estimation and develop a classification scheme for approaches to resource demand estimation. We implement a sub-set of these estimation approaches and evaluate them in a representative environment. We analyze the influence of various factors of the environment on the estimation accuracy, considering the impact of current technologies, such as multi-core processors and virtualization. Our work improves the comparability of existing estimation approaches and facilitates the selection of an approach in a given application scenario. Additionally, it shows possible directions for future research in the field of resource demand estimation.

@mastersthesis{Spinner2011a, abstract = {Proactively managing the performance and resource efficiency of running software systems requires techniques to predict system performance and resource consumption. Typically, performance predictions are based on performance models that capture the performance-relevant aspects of the considered software system. Building performance models involves the estimation of resource demands, i.e., estimating the time a unit of work spends obtaining service from a resource. A number of approaches to estimating the resource demands of a system already exist, e.g., based on regression analysis or stochastic filtering. These approaches differ in their accuracy, their robustness and their applicability. For instance, there are notable differences in the amount and type of measurement data that is required as input. However, to the best of our knowledge, a comprehensive evaluation and comparison of these approaches in a representative context does not exist. In this thesis, we give an overview of the state-of-the-art in resource demand estimation and develop a classification scheme for approaches to resource demand estimation. We implement a sub-set of these estimation approaches and evaluate them in a representative environment. We analyze the influence of various factors of the environment on the estimation accuracy, considering the impact of current technologies, such as multi-core processors and virtualization. Our work improves the comparability of existing estimation approaches and facilitates the selection of an approach in a given application scenario. Additionally, it shows possible directions for future research in the field of resource demand estimation.}, address = {Am Fasanengarten 5, 76131 Karlsruhe, Germany}, author = {Spinner, Simon}, keywords = {QPME_Bibliography}, month = {07}, note = {Best Graduate Award from the Faculty of Informatics}, school = {Karlsruhe Institute of Technology (KIT)}, title = {{Evaluating Approaches to Resource Demand Estimation}}, year = 2011 }

%0 Thesis %1 Spinner2011a %A Spinner, Simon %C Am Fasanengarten 5, 76131 Karlsruhe, Germany %D 2011 %T Evaluating Approaches to Resource Demand Estimation %X Proactively managing the performance and resource efficiency of running software systems requires techniques to predict system performance and resource consumption. Typically, performance predictions are based on performance models that capture the performance-relevant aspects of the considered software system. Building performance models involves the estimation of resource demands, i.e., estimating the time a unit of work spends obtaining service from a resource. A number of approaches to estimating the resource demands of a system already exist, e.g., based on regression analysis or stochastic filtering. These approaches differ in their accuracy, their robustness and their applicability. For instance, there are notable differences in the amount and type of measurement data that is required as input. However, to the best of our knowledge, a comprehensive evaluation and comparison of these approaches in a representative context does not exist. In this thesis, we give an overview of the state-of-the-art in resource demand estimation and develop a classification scheme for approaches to resource demand estimation. We implement a sub-set of these estimation approaches and evaluate them in a representative environment. We analyze the influence of various factors of the environment on the estimation accuracy, considering the impact of current technologies, such as multi-core processors and virtualization. Our work improves the comparability of existing estimation approaches and facilitates the selection of an approach in a given application scenario. Additionally, it shows possible directions for future research in the field of resource demand estimation.

Update: This tool is no longer maintained and supported.

Library for Resource Demand Estimation

Mailing List

Publications

Hinweis zum Datenschutz

Hinweis zum Datenschutz

Bildnachweise

Your E-mail address:
Your Name (optional):