Publications

Beyond-local neural information processing in neuronal networks. Balkenhol, Johannes; Händel, Barbara; Biswas, Sounak; Grohmann, Johannes; v. Kistowski, Jóakim; Prada, Juan; Bosman, Conrado A.; Ehrenreich, Hannelore; Wojcik, Sonja M.; Kounev, Samuel; Blum, Robert; Dandekar, Thomas; in Computational and Structural Biotechnology Journal (2024).

[ Abstract ]
[ BibTeX ]
[ EndNote ]
[ URL ]
[ DOI ]

While there is much knowledge about local neuronal circuitry, considerably less is known about how neuronal input is integrated and combined across neuronal networks to encode higher order brain functions. One challenge lies in the large number of complex neural interactions. Neural networks use oscillating activity for information exchange between distributed nodes. To better understand building principles underlying the observation of synchronized oscillatory activity in a large-scale network, we developed a reductionistic neuronal network model. Fundamental building principles are laterally and temporally interconnected virtual nodes (microcircuits), wherein each node was modeled as a local oscillator. By this building principle, the neuronal network model can integrate information in time and space. The simulation gives rise to a wave interference pattern that spreads over all simulated columns in form of a travelling wave. The model design stabilizes states of efficient information processing across all participating neuronal equivalents. Model-specific oscillatory patterns, generated by complex input stimuli, were similar to electrophysiological high-frequency signals that we could confirm in the primate visual cortex during a visual perception task. Important oscillatory model pre-runners, limitations and strength of our reductionistic model are discussed. Our simple scalable model shows unique integration properties and successfully reproduces a variety of biological phenomena such as harmonics, coherence patterns, frequency-speed relationships, and oscillatory activities. We suggest that our scalable model simulates aspects of a basic building principle underlying oscillatory, large-scale integration of information in small and large brains.

@article{BALKENHOL2024,
  abstract = {While there is much knowledge about local neuronal circuitry, considerably less is known about how neuronal input is integrated and combined across neuronal networks to encode higher order brain functions. One challenge lies in the large number of complex neural interactions. Neural networks use oscillating activity for information exchange between distributed nodes. To better understand building principles underlying the observation of synchronized oscillatory activity in a large-scale network, we developed a reductionistic neuronal network model. Fundamental building principles are laterally and temporally interconnected virtual nodes (microcircuits), wherein each node was modeled as a local oscillator. By this building principle, the neuronal network model can integrate information in time and space. The simulation gives rise to a wave interference pattern that spreads over all simulated columns in form of a travelling wave. The model design stabilizes states of efficient information processing across all participating neuronal equivalents. Model-specific oscillatory patterns, generated by complex input stimuli, were similar to electrophysiological high-frequency signals that we could confirm in the primate visual cortex during a visual perception task. Important oscillatory model pre-runners, limitations and strength of our reductionistic model are discussed. Our simple scalable model shows unique integration properties and successfully reproduces a variety of biological phenomena such as harmonics, coherence patterns, frequency-speed relationships, and oscillatory activities. We suggest that our scalable model simulates aspects of a basic building principle underlying oscillatory, large-scale integration of information in small and large brains.},
  author = {Balkenhol, Johannes and Händel, Barbara and Biswas, Sounak and Grohmann, Johannes and v. Kistowski, Jóakim and Prada, Juan and Bosman, Conrado A. and Ehrenreich, Hannelore and Wojcik, Sonja M. and Kounev, Samuel and Blum, Robert and Dandekar, Thomas},
  journal = {Computational and Structural Biotechnology Journal},
  keywords = {t_journalmagazine},
  title = {Beyond-local neural information processing in neuronal networks},
  year = 2024
}

%0 Journal Article
%1 BALKENHOL2024
%A Balkenhol, Johannes
%A Händel, Barbara
%A Biswas, Sounak
%A Grohmann, Johannes
%A v. Kistowski, Jóakim
%A Prada, Juan
%A Bosman, Conrado A.
%A Ehrenreich, Hannelore
%A Wojcik, Sonja M.
%A Kounev, Samuel
%A Blum, Robert
%A Dandekar, Thomas
%D 2024
%J Computational and Structural Biotechnology Journal
%R https://doi.org/10.1016/j.csbj.2024.10.040
%T Beyond-local neural information processing in neuronal networks
%U https://www.sciencedirect.com/science/article/pii/S200103702400360X
%X While there is much knowledge about local neuronal circuitry, considerably less is known about how neuronal input is integrated and combined across neuronal networks to encode higher order brain functions. One challenge lies in the large number of complex neural interactions. Neural networks use oscillating activity for information exchange between distributed nodes. To better understand building principles underlying the observation of synchronized oscillatory activity in a large-scale network, we developed a reductionistic neuronal network model. Fundamental building principles are laterally and temporally interconnected virtual nodes (microcircuits), wherein each node was modeled as a local oscillator. By this building principle, the neuronal network model can integrate information in time and space. The simulation gives rise to a wave interference pattern that spreads over all simulated columns in form of a travelling wave. The model design stabilizes states of efficient information processing across all participating neuronal equivalents. Model-specific oscillatory patterns, generated by complex input stimuli, were similar to electrophysiological high-frequency signals that we could confirm in the primate visual cortex during a visual perception task. Important oscillatory model pre-runners, limitations and strength of our reductionistic model are discussed. Our simple scalable model shows unique integration properties and successfully reproduces a variety of biological phenomena such as harmonics, coherence patterns, frequency-speed relationships, and oscillatory activities. We suggest that our scalable model simulates aspects of a basic building principle underlying oscillatory, large-scale integration of information in small and large brains.

Autoscaler Evaluation and Configuration: A Practitioner’s Guideline. Straesser, Martin; Eismann, Simon; von Kistowski, J{ó}akim; Bauer, Andr{é}; Kounev, Samuel; in Proceedings of the 2023 ACM/SPEC International Conference on Performance Engineering (2023). 31–41. Association for Computing Machinery, New York, NY, USA.

[ Abstract ]
[ BibTeX ]
[ EndNote ]
[ URL ]
[ DOI ]
[ Download ]

Autoscalers are indispensable parts of modern cloud deployments and determine the service quality and cost of a cloud application in dynamic workloads. The configuration of an autoscaler strongly influences its performance and is also one of the biggest challenges and showstoppers for the practical applicability of many research autoscalers. Many proposed cloud experiment methodologies can only be partially applied in practice, and many autoscaling papers use custom evaluation methods and metrics. This paper presents a practical guideline for obtaining meaningful and interpretable results on autoscaler performance with reasonable overhead. We provide step-by-step instructions for defining realistic usage behaviors and traffic patterns. We divide the analysis of autoscaler performance into a qualitative antipattern-based analysis and a quantitative analysis. To demonstrate the applicability of our guideline, we conduct several experiments with a microservice of our industry partner in a realistic test environment.

@inproceedings{straesser2023guideline,
  abstract = {Autoscalers are indispensable parts of modern cloud deployments and determine the service quality and cost of a cloud application in dynamic workloads. The configuration of an autoscaler strongly influences its performance and is also one of the biggest challenges and showstoppers for the practical applicability of many research autoscalers. Many proposed cloud experiment methodologies can only be partially applied in practice, and many autoscaling papers use custom evaluation methods and metrics. This paper presents a practical guideline for obtaining meaningful and interpretable results on autoscaler performance with reasonable overhead. We provide step-by-step instructions for defining realistic usage behaviors and traffic patterns. We divide the analysis of autoscaler performance into a qualitative antipattern-based analysis and a quantitative analysis. To demonstrate the applicability of our guideline, we conduct several experiments with a microservice of our industry partner in a realistic test environment.},
  address = {New York, NY, USA},
  author = {Straesser, Martin and Eismann, Simon and von Kistowski, J\'{o}akim and Bauer, Andr\'{e} and Kounev, Samuel},
  booktitle = {Proceedings of the 2023 ACM/SPEC International Conference on Performance Engineering},
  keywords = {methodology},
  pages = {31-41},
  publisher = {Association for Computing Machinery},
  series = {ICPE '23},
  title = {Autoscaler Evaluation and Configuration: A Practitioner's Guideline},
  year = 2023
}

%0 Conference Paper
%1 straesser2023guideline
%A Straesser, Martin
%A Eismann, Simon
%A von Kistowski, J{ó}akim
%A Bauer, Andr{é}
%A Kounev, Samuel
%B Proceedings of the 2023 ACM/SPEC International Conference on Performance Engineering
%C New York, NY, USA
%D 2023
%I Association for Computing Machinery
%P 31-41
%R 10.1145/3578244.3583721
%T Autoscaler Evaluation and Configuration: A Practitioner's Guideline
%U https://doi.org/10.1145/3578244.3583721
%X Autoscalers are indispensable parts of modern cloud deployments and determine the service quality and cost of a cloud application in dynamic workloads. The configuration of an autoscaler strongly influences its performance and is also one of the biggest challenges and showstoppers for the practical applicability of many research autoscalers. Many proposed cloud experiment methodologies can only be partially applied in practice, and many autoscaling papers use custom evaluation methods and metrics. This paper presents a practical guideline for obtaining meaningful and interpretable results on autoscaler performance with reasonable overhead. We provide step-by-step instructions for defining realistic usage behaviors and traffic patterns. We divide the analysis of autoscaler performance into a qualitative antipattern-based analysis and a quantitative analysis. To demonstrate the applicability of our guideline, we conduct several experiments with a microservice of our industry partner in a realistic test environment.
%@ 9798400700682

Why Is It Not Solved Yet? Challenges for Production-Ready Autoscaling. Straesser, Martin; Grohmann, Johannes; von Kistowski, J{ó}akim; Eismann, Simon; Bauer, Andr{é}; Kounev, Samuel; in Proceedings of the 2022 ACM/SPEC on International Conference on Performance Engineering (2022). 105–115. Association for Computing Machinery, New York, NY, USA.

[ Abstract ]
[ BibTeX ]
[ EndNote ]
[ URL ]
[ Download ]

Autoscaling is a task of major importance in the cloud computing domain as it directly affects both operating costs and customer experience. Although there has been active research in this area for over ten years now, there is still a significant gap between the proposed methods in the literature and the deployed autoscalers in practice. Hence, many research autoscalers do not find their way into production deployments. This paper describes six core challenges that arise in production systems that are still not solved by most research autoscalers. We illustrate these problems through experiments in a realistic cloud environment with a real-world multi-service business application and show that commonly used autoscalers have various shortcomings. In addition, we analyze the behavior of overloaded services and show that these can be problematic for existing autoscalers. Generally, we analyze that these challenges are only insufficiently addressed in the literature and conclude that future scaling approaches should focus on the needs of production systems.

@inproceedings{10.1145/3489525.3511680,
  abstract = {Autoscaling is a task of major importance in the cloud computing domain as it directly affects both operating costs and customer experience. Although there has been active research in this area for over ten years now, there is still a significant gap between the proposed methods in the literature and the deployed autoscalers in practice. Hence, many research autoscalers do not find their way into production deployments. This paper describes six core challenges that arise in production systems that are still not solved by most research autoscalers. We illustrate these problems through experiments in a realistic cloud environment with a real-world multi-service business application and show that commonly used autoscalers have various shortcomings. In addition, we analyze the behavior of overloaded services and show that these can be problematic for existing autoscalers. Generally, we analyze that these challenges are only insufficiently addressed in the literature and conclude that future scaling approaches should focus on the needs of production systems.},
  address = {New York, NY, USA},
  author = {Straesser, Martin and Grohmann, Johannes and von Kistowski, J\'{o}akim and Eismann, Simon and Bauer, Andr\'{e} and Kounev, Samuel},
  booktitle = {Proceedings of the 2022 ACM/SPEC on International Conference on Performance Engineering},
  keywords = {microservices},
  pages = {105–115},
  publisher = {Association for Computing Machinery},
  series = {ICPE '22},
  title = {Why Is It Not Solved Yet? Challenges for Production-Ready Autoscaling},
  year = 2022
}

%0 Conference Paper
%1 10.1145/3489525.3511680
%A Straesser, Martin
%A Grohmann, Johannes
%A von Kistowski, J{ó}akim
%A Eismann, Simon
%A Bauer, Andr{é}
%A Kounev, Samuel
%B Proceedings of the 2022 ACM/SPEC on International Conference on Performance Engineering
%C New York, NY, USA
%D 2022
%I Association for Computing Machinery
%P 105–115
%T Why Is It Not Solved Yet? Challenges for Production-Ready Autoscaling
%U https://doi.org/10.1145/3489525.3511680
%X Autoscaling is a task of major importance in the cloud computing domain as it directly affects both operating costs and customer experience. Although there has been active research in this area for over ten years now, there is still a significant gap between the proposed methods in the literature and the deployed autoscalers in practice. Hence, many research autoscalers do not find their way into production deployments. This paper describes six core challenges that arise in production systems that are still not solved by most research autoscalers. We illustrate these problems through experiments in a realistic cloud environment with a real-world multi-service business application and show that commonly used autoscalers have various shortcomings. In addition, we analyze the behavior of overloaded services and show that these can be problematic for existing autoscalers. Generally, we analyze that these challenges are only insufficiently addressed in the literature and conclude that future scaling approaches should focus on the needs of production systems.

{Systems Benchmarking} Kounev, Samuel; Lange, Klaus-Dieter; von Kistowski, Jóakim; (2020). (1st ed.) Springer International Publishing.

[ BibTeX ]
[ EndNote ]

@book{KoLaKi-2020-SystemsBenchmarking,
  author = {Kounev, Samuel and Lange, Klaus-Dieter and von Kistowski, Jóakim},
  edition = 1,
  keywords = {t_book},
  publisher = {Springer International Publishing},
  title = {{Systems Benchmarking}},
  year = 2020
}

%0 Book
%1 KoLaKi-2020-SystemsBenchmarking
%A Kounev, Samuel
%A Lange, Klaus-Dieter
%A von Kistowski, Jóakim
%D 2020
%I Springer International Publishing
%T {Systems Benchmarking}
%7 1

{Methodological Principles for Reproducible Performance Evaluation in Cloud Computing}. Papadopoulos, Alessandro Vittorio; Versluis, Laurens; Bauer, Andr{{é}}; Herbst, Nikolas; von Kistowski, J{{ó}}akim; Ali{-}Eldin, Ahmed; Abad, Cristina L.; Amaral, Jos{{é}} Nelson; Tuma, Petr; Iosup, Alexandru; in Software Engineering 2020, Fachtagung des GI-Fachbereichs Softwaretechnik, 24.-28. Februar 2020, Innsbruck, Austria, M. Felderer, W. Hasselbring, R. Rabiser, R. Jung (eds.) (2020). (Vol. {P-300}) 93–94. Gesellschaft f{{ü}}r Informatik e.V.

[ BibTeX ]
[ EndNote ]
[ URL ]
[ Download ]

@inproceedings{DBLP:conf/se/PapadopoulosVBH20,
  author = {Papadopoulos, Alessandro Vittorio and Versluis, Laurens and Bauer, Andr{{\'e}} and Herbst, Nikolas and von Kistowski, J{{\'o}}akim and Ali{-}Eldin, Ahmed and Abad, Cristina L. and Amaral, Jos{{\'e}} Nelson and Tuma, Petr and Iosup, Alexandru},
  booktitle = {Software Engineering 2020, Fachtagung des GI-Fachbereichs Softwaretechnik, 24.-28. Februar 2020, Innsbruck, Austria},
  editor = {Felderer, Michael and Hasselbring, Wilhelm and Rabiser, Rick and Jung, Reiner},
  keywords = {se2},
  pages = {93–94},
  publisher = {Gesellschaft f{{\"u}}r Informatik e.V.},
  series = {{LNI}},
  title = {{Methodological Principles for Reproducible Performance Evaluation in Cloud Computing}},
  volume = {{P-300}},
  year = 2020
}

%0 Conference Paper
%1 DBLP:conf/se/PapadopoulosVBH20
%A Papadopoulos, Alessandro Vittorio
%A Versluis, Laurens
%A Bauer, Andr{{é}}
%A Herbst, Nikolas
%A von Kistowski, J{{ó}}akim
%A Ali{-}Eldin, Ahmed
%A Abad, Cristina L.
%A Amaral, Jos{{é}} Nelson
%A Tuma, Petr
%A Iosup, Alexandru
%B Software Engineering 2020, Fachtagung des GI-Fachbereichs Softwaretechnik, 24.-28. Februar 2020, Innsbruck, Austria
%D 2020
%E Felderer, Michael
%E Hasselbring, Wilhelm
%E Rabiser, Rick
%E Jung, Reiner
%I Gesellschaft f{{ü}}r Informatik e.V.
%P 93--94
%T {Methodological Principles for Reproducible Performance Evaluation in Cloud Computing}
%U https://doi.org/10.18420/SE2020\_27
%V {P-300}

{Detecting Parametric Dependencies for Performance Models Using Feature Selection Techniques}. Grohmann, Johannes; Eismann, Simon; Elflein, Sven; Mazkatli, Manar; von Kistowski, J{ó}akim; Kounev, Samuel; in 2019 IEEE 27th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS) (2019). 309–322. IEEE Computer Society.

{Acceptance Rate: 23.8% (29/122)}

[ Abstract ]
[ BibTeX ]
[ EndNote ]
[ URL ]
[ DOI ]
[ Download ]

Architectural performance models are a common approach to predict the performance properties of a software system. Parametric dependencies, which describe the relation between the input parameters of a component and its performance properties, significantly increase the prediction accuracy of architectural performance models. However, manually modeling parametric dependencies is time-intensive and requires expert knowledge. Existing automated extraction approaches require dedicated performance tests, which are often infeasible. In this paper, we introduce an approach to automatically identify parametric dependencies from monitoring data using feature selection techniques from the area of machine learning. We evaluate the applicability of three techniques selected from each of the three groups of feature selection methods: a filter method, an embedded method, and a wrapper method. Our evaluation shows that the filter technique outperforms the other approaches. Based on these results, we apply this technique to a distributed micro-service web-shop, where it correctly identifies 11 performance-relevant dependencies, achieving a precision of 91.7% based on a manually labeled gold-standard.

@inproceedings{GrEiElMaKiKo2019-MASCOTS-DependencyIdentification,
  abstract = {Architectural performance models are a common approach to predict the performance properties of a software system. Parametric dependencies, which describe the relation between the input parameters of a component and its performance properties, significantly increase the prediction accuracy of architectural performance models. However, manually modeling parametric dependencies is time-intensive and requires expert knowledge. Existing automated extraction approaches require dedicated performance tests, which are often infeasible. In this paper, we introduce an approach to automatically identify parametric dependencies from monitoring data using feature selection techniques from the area of machine learning. We evaluate the applicability of three techniques selected from each of the three groups of feature selection methods: a filter method, an embedded method, and a wrapper method. Our evaluation shows that the filter technique outperforms the other approaches. Based on these results, we apply this technique to a distributed micro-service web-shop, where it correctly identifies 11 performance-relevant dependencies, achieving a precision of 91.7% based on a manually labeled gold-standard.},
  author = {Grohmann, Johannes and Eismann, Simon and Elflein, Sven and Mazkatli, Manar and von Kistowski, J{\'o}akim and Kounev, Samuel},
  booktitle = {2019 IEEE 27th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS)},
  keywords = {Automated_model_learning},
  month = 10,
  note = {{Acceptance Rate: 23.8% (29/122)}},
  pages = {309–322},
  publisher = {IEEE Computer Society},
  series = {MASCOTS '19},
  title = {{Detecting Parametric Dependencies for Performance Models Using Feature Selection Techniques}},
  year = 2019
}

%0 Conference Paper
%1 GrEiElMaKiKo2019-MASCOTS-DependencyIdentification
%A Grohmann, Johannes
%A Eismann, Simon
%A Elflein, Sven
%A Mazkatli, Manar
%A von Kistowski, J{ó}akim
%A Kounev, Samuel
%B 2019 IEEE 27th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS)
%D 2019
%I IEEE Computer Society
%P 309--322
%R 10.1109/MASCOTS.2019.00042
%T {Detecting Parametric Dependencies for Performance Models Using Feature Selection Techniques}
%U https://doi.org/10.1109/MASCOTS.2019.00042
%X Architectural performance models are a common approach to predict the performance properties of a software system. Parametric dependencies, which describe the relation between the input parameters of a component and its performance properties, significantly increase the prediction accuracy of architectural performance models. However, manually modeling parametric dependencies is time-intensive and requires expert knowledge. Existing automated extraction approaches require dedicated performance tests, which are often infeasible. In this paper, we introduce an approach to automatically identify parametric dependencies from monitoring data using feature selection techniques from the area of machine learning. We evaluate the applicability of three techniques selected from each of the three groups of feature selection methods: a filter method, an embedded method, and a wrapper method. Our evaluation shows that the filter technique outperforms the other approaches. Based on these results, we apply this technique to a distributed micro-service web-shop, where it correctly identifies 11 performance-relevant dependencies, achieving a precision of 91.7% based on a manually labeled gold-standard.
%@ 978-1-7281-4950-9

Methodological Principles for Reproducible Performance Evaluation in Cloud Computing. Papadopoulos, Alessandro Vittorio; Versluis, Laurens; Bauer, Andr{é}; Herbst, Nikolas; von Kistowski, J{ó}akim; Ali-Eldin, Ahmed; Abad, Cristina; Amaral, J. Nelson; Tuma, Petr; Iosup, Alexandru; in IEEE Transactions on Software Engineering (2019). 47(8) 1528–1543.

[ BibTeX ]
[ EndNote ]
[ DOI ]
[ Download ]

@article{PaVeBaHeKiAEAbAmTuIo2019-TSE-MethodologicalPrinciples,
  author = {Papadopoulos, Alessandro Vittorio and Versluis, Laurens and Bauer, Andr{\'e} and Herbst, Nikolas and von Kistowski, J{\'o}akim and Ali-Eldin, Ahmed and Abad, Cristina and Amaral, J. Nelson and Tuma, Petr and Iosup, Alexandru},
  journal = {IEEE Transactions on Software Engineering},
  keywords = {SPEC},
  month = {07},
  number = 8,
  pages = {1528-1543},
  title = {Methodological Principles for Reproducible Performance Evaluation in Cloud Computing},
  volume = 47,
  year = 2019
}

%0 Journal Article
%1 PaVeBaHeKiAEAbAmTuIo2019-TSE-MethodologicalPrinciples
%A Papadopoulos, Alessandro Vittorio
%A Versluis, Laurens
%A Bauer, Andr{é}
%A Herbst, Nikolas
%A von Kistowski, J{ó}akim
%A Ali-Eldin, Ahmed
%A Abad, Cristina
%A Amaral, J. Nelson
%A Tuma, Petr
%A Iosup, Alexandru
%D 2019
%J IEEE Transactions on Software Engineering
%N 8
%P 1528-1543
%R 10.1109/TSE.2019.2927908
%T Methodological Principles for Reproducible Performance Evaluation in Cloud Computing
%V 47

{Methodological Principles for Reproducible Performance Evaluation in Cloud Computing - A SPEC Research Technical Report}. (SPEC-RG-2019-04), Papadopoulos, Alessandro Vittorio; Versluis, Laurens; Bauer, Andr{é}; Herbst, Nikolas; von Kistowski, J{ó}akim; Ali-Eldin, Ahmed; Abad, Cristina; Amaral, J. Nelson; Tuma, Petr; Iosup, Alexandru; (2019). SPEC Research Group --- Cloud Working Group, Standard Performance Evaluation Corporation (SPEC).

[ BibTeX ]
[ EndNote ]
[ URL ]

@techreport{PaVeBaHeKiEAAbAmTuIo2019-SPEC-CloudPerformance,
  author = {Papadopoulos, Alessandro Vittorio and Versluis, Laurens and Bauer, Andr{\'e} and Herbst, Nikolas and von Kistowski, J{\'o}akim and Ali-Eldin, Ahmed and Abad, Cristina and Amaral, J. Nelson and Tuma, Petr and Iosup, Alexandru},
  institution = {SPEC Research Group — Cloud Working Group, Standard Performance Evaluation Corporation (SPEC)},
  keywords = {SPEC},
  month = {04},
  number = {SPEC-RG-2019-04},
  title = {{Methodological Principles for Reproducible Performance Evaluation in Cloud Computing - A SPEC Research Technical Report}},
  year = 2019
}

%0 Report
%1 PaVeBaHeKiEAAbAmTuIo2019-SPEC-CloudPerformance
%A Papadopoulos, Alessandro Vittorio
%A Versluis, Laurens
%A Bauer, Andr{é}
%A Herbst, Nikolas
%A von Kistowski, J{ó}akim
%A Ali-Eldin, Ahmed
%A Abad, Cristina
%A Amaral, J. Nelson
%A Tuma, Petr
%A Iosup, Alexandru
%D 2019
%N SPEC-RG-2019-04
%T {Methodological Principles for Reproducible Performance Evaluation in Cloud Computing - A SPEC Research Technical Report}
%U https://research.spec.org/fileadmin/user_upload/documents/rg_cloud/endorsed_publications/SPEC_RG_2019_Methodological_Principles_for_Reproducible_Performance_Evaluation_in_Cloud_Computing.pdf

{Integrating Statistical Response Time Models in Architectural Performance Models}. Eismann, Simon; Grohmann, Johannes; Walter, J{ü}rgen; von Kistowski, J{ó}akim; Kounev, Samuel; in Proceedings of the 2019 IEEE International Conference on Software Architecture (ICSA) (2019). 71–80. IEEE.

Acceptance Rate: 21,9\% (21/96)

[ Abstract ]
[ BibTeX ]
[ EndNote ]
[ URL ]
[ DOI ]
[ Download ]

Performance predictions enable software architects to optimize the performance of a software system early in the development cycle. Architectural performance models and statistical response time models are commonly used to derive these performance predictions. However, both methods have significant downsides: Statistical response time models can only predict scenarios for which training data is available, making the prediction of previously unseen system configurations infeasible. In contrast, the time required to simulate an architectural performance model increases exponentially with both system size and level of modeling detail, making the analysis of large, detailed models challenging. Existing approaches use statistical response time models in architectural performance models to avoid modeling subsystems that are difficult or time-consuming to model, yet they do not consider simulation time. In this paper, we propose to model software systems using classical queuing theory and statistical response time models in parallel. This approach allows users to tailor the model for each analysis run, based on the performed adaptations and the requested performance metrics. Our approach enables faster model solution compared to traditional performance models while retaining their ability to predict previously unseen scenarios. In our experiments we observed speedups of up to 94.8%, making the analysis of much larger and more detailed systems feasible.

@inproceedings{EiGrWaKiKo2019-ICSA-Integrating,
  abstract = {Performance predictions enable software architects to optimize the performance of a software system early in the development cycle. Architectural performance models and statistical response time models are commonly used to derive these performance predictions. However, both methods have significant downsides: Statistical response time models can only predict scenarios for which training data is available, making the prediction of previously unseen system configurations infeasible. In contrast, the time required to simulate an architectural performance model increases exponentially with both system size and level of modeling detail, making the analysis of large, detailed models challenging. Existing approaches use statistical response time models in architectural performance models to avoid modeling subsystems that are difficult or time-consuming to model, yet they do not consider simulation time. In this paper, we propose to model software systems using classical queuing theory and statistical response time models in parallel. This approach allows users to tailor the model for each analysis run, based on the performed adaptations and the requested performance metrics. Our approach enables faster model solution compared to traditional performance models while retaining their ability to predict previously unseen scenarios. In our experiments we observed speedups of up to 94.8%, making the analysis of much larger and more detailed systems feasible.},
  author = {Eismann, Simon and Grohmann, Johannes and Walter, J{\"u}rgen and von Kistowski, J{\'o}akim and Kounev, Samuel},
  booktitle = {Proceedings of the 2019 IEEE International Conference on Software Architecture (ICSA)},
  keywords = {PRISMA},
  month = {03},
  note = {Acceptance Rate: 21,9\% (21/96)},
  pages = {71–80},
  publisher = {IEEE},
  title = {{Integrating Statistical Response Time Models in Architectural Performance Models}},
  year = 2019
}

%0 Conference Paper
%1 EiGrWaKiKo2019-ICSA-Integrating
%A Eismann, Simon
%A Grohmann, Johannes
%A Walter, J{ü}rgen
%A von Kistowski, J{ó}akim
%A Kounev, Samuel
%B Proceedings of the 2019 IEEE International Conference on Software Architecture (ICSA)
%D 2019
%I IEEE
%P 71--80
%R 10.1109/ICSA.2019.00016
%T {Integrating Statistical Response Time Models in Architectural Performance Models}
%U https://doi.org/10.1109/ICSA.2019.00016
%X Performance predictions enable software architects to optimize the performance of a software system early in the development cycle. Architectural performance models and statistical response time models are commonly used to derive these performance predictions. However, both methods have significant downsides: Statistical response time models can only predict scenarios for which training data is available, making the prediction of previously unseen system configurations infeasible. In contrast, the time required to simulate an architectural performance model increases exponentially with both system size and level of modeling detail, making the analysis of large, detailed models challenging. Existing approaches use statistical response time models in architectural performance models to avoid modeling subsystems that are difficult or time-consuming to model, yet they do not consider simulation time. In this paper, we propose to model software systems using classical queuing theory and statistical response time models in parallel. This approach allows users to tailor the model for each analysis run, based on the performed adaptations and the requested performance metrics. Our approach enables faster model solution compared to traditional performance models while retaining their ability to predict previously unseen scenarios. In our experiments we observed speedups of up to 94.8%, making the analysis of much larger and more detailed systems feasible.

Measuring, Rating, and Predicting the Energy Efficiency of Servers. von Kistowski, J{ó}akim; (2019, March). University of W{ü}rzburg, Germany.

[ BibTeX ]
[ EndNote ]
[ URL ]

@phdthesis{Kistowski2019-phd,
  author = {von Kistowski, J{\'o}akim},
  keywords = {Power},
  month = {03},
  school = {University of W{\"u}rzburg, Germany},
  title = {Measuring, Rating, and Predicting the Energy Efficiency of Servers},
  year = 2019
}

%0 Thesis
%1 Kistowski2019-phd
%A von Kistowski, J{ó}akim
%D 2019
%T Measuring, Rating, and Predicting the Energy Efficiency of Servers
%U https://opus.bibliothek.uni-wuerzburg.de/frontdoor/index/index/docId/17847

{Measuring the Energy Efficiency of Transactional Loads on GPGPU}. von Kistowski, J{ó}akim; Pais, Johann; Wahl, Tobias; Lange, Klaus-Dieter; Block, Hansfried; Beckett, John; Kounev, Samuel; in Proceedings of the 19th ACM/SPEC International Conference on Performance Engineering (2019). ACM, New York, NY, USA.

{Best Industry Paper Award}

[ Abstract ]
[ BibTeX ]
[ EndNote ]
[ Download ]

{General Purpose Graphics Processing Units (GPGPUs) are becoming more and more common in current servers and data centers, which in turn consume a significant amount of electrical power. Measuring and benchmarking this power consumption is important as it helps with optimization and selection of these servers. However, benchmarking and comparing the energy efficiency of GPGPU workloads is challenging as standardized workloads are rare and standardized power and efficiency measurement methods and metrics do not exist. In addition, not all GPGPU systems run at maximum load all the time. Systems that are utilized in transactional, request driven workloads, for example, can run at lower utilization levels. Existing benchmarks for GPGPU systems primarily consider performance and are intended only to run at maximum load. They do not measure performance or energy efficiency at other loads. In turn, server energy-efficiency benchmarks that consider multiple load levels do not address GPGPUs. This paper introduces a measurement methodology for servers with GPGPU accelerators that considers multiple load levels for transactional workloads. The methodology also addresses verifiability of results in order to achieve comparability of different device solutions. We analyze our methodology on three different systems with solutions from two different accelerator vendors. We investigate the efficacy of different methods of load levels scaling and our methodology's reproducibility. We show that the methodology is able to produce consistent and reproducible results with a maximum coefficient of variation of 1.4\% regarding power consumption.}

@inproceedings{KiPaWaLaBlBeKo2019-ICPE-GPGPUPower,
  abstract = {{General Purpose Graphics Processing Units (GPGPUs) are becoming more and more common in current servers and data centers, which in turn consume a significant amount of electrical power. Measuring and benchmarking this power consumption is important as it helps with optimization and selection of these servers. However, benchmarking and comparing the energy efficiency of GPGPU workloads is challenging as standardized workloads are rare and standardized power and efficiency measurement methods and metrics do not exist. In addition, not all GPGPU systems run at maximum load all the time. Systems that are utilized in transactional, request driven workloads, for example, can run at lower utilization levels. Existing benchmarks for GPGPU systems primarily consider performance and are intended only to run at maximum load. They do not measure performance or energy efficiency at other loads. In turn, server energy-efficiency benchmarks that consider multiple load levels do not address GPGPUs. This paper introduces a measurement methodology for servers with GPGPU accelerators that considers multiple load levels for transactional workloads. The methodology also addresses verifiability of results in order to achieve comparability of different device solutions. We analyze our methodology on three different systems with solutions from two different accelerator vendors. We investigate the efficacy of different methods of load levels scaling and our methodology's reproducibility. We show that the methodology is able to produce consistent and reproducible results with a maximum coefficient of variation of 1.4\% regarding power consumption.}},
  address = {New York, NY, USA},
  author = {von Kistowski, J{\'o}akim and Pais, Johann and Wahl, Tobias and Lange, Klaus-Dieter and Block, Hansfried and Beckett, John and Kounev, Samuel},
  booktitle = {Proceedings of the 19th ACM/SPEC International Conference on Performance Engineering},
  keywords = {Power},
  note = {{<b>Best Industry Paper Award</b>}},
  publisher = {ACM},
  series = {ICPE '19},
  title = {{Measuring the Energy Efficiency of Transactional Loads on GPGPU}},
  year = 2019
}

%0 Conference Paper
%1 KiPaWaLaBlBeKo2019-ICPE-GPGPUPower
%A von Kistowski, J{ó}akim
%A Pais, Johann
%A Wahl, Tobias
%A Lange, Klaus-Dieter
%A Block, Hansfried
%A Beckett, John
%A Kounev, Samuel
%B Proceedings of the 19th ACM/SPEC International Conference on Performance Engineering
%C New York, NY, USA
%D 2019
%I ACM
%T {Measuring the Energy Efficiency of Transactional Loads on GPGPU}
%X {General Purpose Graphics Processing Units (GPGPUs) are becoming more and more common in current servers and data centers, which in turn consume a significant amount of electrical power. Measuring and benchmarking this power consumption is important as it helps with optimization and selection of these servers. However, benchmarking and comparing the energy efficiency of GPGPU workloads is challenging as standardized workloads are rare and standardized power and efficiency measurement methods and metrics do not exist. In addition, not all GPGPU systems run at maximum load all the time. Systems that are utilized in transactional, request driven workloads, for example, can run at lower utilization levels. Existing benchmarks for GPGPU systems primarily consider performance and are intended only to run at maximum load. They do not measure performance or energy efficiency at other loads. In turn, server energy-efficiency benchmarks that consider multiple load levels do not address GPGPUs. This paper introduces a measurement methodology for servers with GPGPU accelerators that considers multiple load levels for transactional workloads. The methodology also addresses verifiability of results in order to achieve comparability of different device solutions. We analyze our methodology on three different systems with solutions from two different accelerator vendors. We investigate the efficacy of different methods of load levels scaling and our methodology's reproducibility. We show that the methodology is able to produce consistent and reproducible results with a maximum coefficient of variation of 1.4\% regarding power consumption.}

{Measuring and Rating the Energy-Efficiency of Servers}. von Kistowski, J{ó}akim; Lange, Klaus-Dieter; Arnold, Jeremy A.; Beckett, John; Block, Hansfried; Tricker, Mike; Sharma, Sanjay; Pais, Johann; Kounev, Samuel; in Future Generation Computer Systems (2019). 100 579–589.

[ Abstract ]
[ BibTeX ]
[ EndNote ]
[ URL ]

Data centers and servers consume a significant amount of electrical power. As future data centers increase in size, it becomes increasingly important to be able to select servers based on their energy efficiency. Rating the efficiency of servers is challenging, as it depends on the software executed on that server and on its load profile. To account for this, a measurement and rating methodology for servers must make use of both realistic and varying workloads. Existing energy-efficiency benchmarks either run standardized application loads at multiple load levels or multiple micro-benchmarks at full load. This does not enable a full analysis of system behavior as the energy efficiency for different kinds of software at low load levels remains unknown. This article introduces a measurement methodology and metrics for energy-efficiency rating of servers. The methodology defines system setup and instrumentation for reproducible results. It is designed to use multiple, specifically chosen workloads at different load levels for a full system characterization. All partial results are aggregated in a final metric that can be used by regulators to define server-labeling criteria. The methodology and workloads have been implemented in the standardized Server Efficiency Rating Tool. We show the applicability and use of the measurement methodology specifically considering its reproducibility, fairness, and relevance. A reproducibility analysis shows that efficiency scores vary with a maximum coefficient of 1.04\% for repeated experiments. In addition, we evaluate the proposed metrics by investigating their energy-efficiency rating on a set of 385 different servers and show the relevance of the selected workloads by analyzing relationships to real-world applications.

@article{KiLaArBeBlTrShPaKo-FGCS2019-ServerEfficiency,
  abstract = {Data centers and servers consume a significant amount of electrical power. As future data centers increase in size, it becomes increasingly important to be able to select servers based on their energy efficiency. Rating the efficiency of servers is challenging, as it depends on the software executed on that server and on its load profile. To account for this, a measurement and rating methodology for servers must make use of both realistic and varying workloads. Existing energy-efficiency benchmarks either run standardized application loads at multiple load levels or multiple micro-benchmarks at full load. This does not enable a full analysis of system behavior as the energy efficiency for different kinds of software at low load levels remains unknown. This article introduces a measurement methodology and metrics for energy-efficiency rating of servers. The methodology defines system setup and instrumentation for reproducible results. It is designed to use multiple, specifically chosen workloads at different load levels for a full system characterization. All partial results are aggregated in a final metric that can be used by regulators to define server-labeling criteria. The methodology and workloads have been implemented in the standardized Server Efficiency Rating Tool. We show the applicability and use of the measurement methodology specifically considering its reproducibility, fairness, and relevance. A reproducibility analysis shows that efficiency scores vary with a maximum coefficient of 1.04\% for repeated experiments. In addition, we evaluate the proposed metrics by investigating their energy-efficiency rating on a set of 385 different servers and show the relevance of the selected workloads by analyzing relationships to real-world applications.},
  author = {von Kistowski, J{\'o}akim and Lange, Klaus-Dieter and Arnold, Jeremy A. and Beckett, John and Block, Hansfried and Tricker, Mike and Sharma, Sanjay and Pais, Johann and Kounev, Samuel},
  journal = {Future Generation Computer Systems},
  keywords = {Power},
  pages = {579 - 589},
  title = {{Measuring and Rating the Energy-Efficiency of Servers}},
  volume = 100,
  year = 2019
}

%0 Journal Article
%1 KiLaArBeBlTrShPaKo-FGCS2019-ServerEfficiency
%A von Kistowski, J{ó}akim
%A Lange, Klaus-Dieter
%A Arnold, Jeremy A.
%A Beckett, John
%A Block, Hansfried
%A Tricker, Mike
%A Sharma, Sanjay
%A Pais, Johann
%A Kounev, Samuel
%D 2019
%J Future Generation Computer Systems
%P 579 - 589
%T {Measuring and Rating the Energy-Efficiency of Servers}
%U http://www.sciencedirect.com/science/article/pii/S0167739X18312123
%V 100
%X Data centers and servers consume a significant amount of electrical power. As future data centers increase in size, it becomes increasingly important to be able to select servers based on their energy efficiency. Rating the efficiency of servers is challenging, as it depends on the software executed on that server and on its load profile. To account for this, a measurement and rating methodology for servers must make use of both realistic and varying workloads. Existing energy-efficiency benchmarks either run standardized application loads at multiple load levels or multiple micro-benchmarks at full load. This does not enable a full analysis of system behavior as the energy efficiency for different kinds of software at low load levels remains unknown. This article introduces a measurement methodology and metrics for energy-efficiency rating of servers. The methodology defines system setup and instrumentation for reproducible results. It is designed to use multiple, specifically chosen workloads at different load levels for a full system characterization. All partial results are aggregated in a final metric that can be used by regulators to define server-labeling criteria. The methodology and workloads have been implemented in the standardized Server Efficiency Rating Tool. We show the applicability and use of the measurement methodology specifically considering its reproducibility, fairness, and relevance. A reproducibility analysis shows that efficiency scores vary with a maximum coefficient of 1.04\% for repeated experiments. In addition, we evaluate the proposed metrics by investigating their energy-efficiency rating on a set of 385 different servers and show the relevance of the selected workloads by analyzing relationships to real-world applications.

{Predicting Server Power Consumption from Standard Rating Results}. von Kistowski, J{ó}akim; Grohmann, Johannes; Schmitt, Norbert; Kounev, Samuel; in Proceedings of the 19th ACM/SPEC International Conference on Performance Engineering (2019). 301–312. Association for Computing Machinery (ACM), New York, NY, USA.

{Full Paper Acceptance Rate: 18.6\% (13/70)}

[ Abstract ]
[ BibTeX ]
[ EndNote ]
[ URL ]
[ DOI ]
[ Download ]

Data center providers and server operators try to reduce the power consumption of their servers. Finding an energy efficient server for a specific target application is a first step in this regard. Estimating the power consumption of an application on an unavailable server is difficult, as nameplate power values are generally overestimations. Offline power models are able to predict the consumption accurately, but are usually intended for system design, requiring very specific and detailed knowledge about the system under consideration. In this paper, we introduce an offline power prediction method that uses the results of standard power rating tools. The method predicts the power consumption of a specific application for multiple load levels on a target server that is otherwise unavailable for testing. We evaluate our approach by predicting the power consumption of three applications on different physical servers. Our method is able to achieve an average prediction error of 9.49% for three workloads running on real-world, physical servers.

@inproceedings{KiGrScKo2019-ICPE-PowerPrediction,
  abstract = {Data center providers and server operators try to reduce the power consumption of their servers. Finding an energy efficient server for a specific target application is a first step in this regard. Estimating the power consumption of an application on an unavailable server is difficult, as nameplate power values are generally overestimations. Offline power models are able to predict the consumption accurately, but are usually intended for system design, requiring very specific and detailed knowledge about the system under consideration. In this paper, we introduce an offline power prediction method that uses the results of standard power rating tools. The method predicts the power consumption of a specific application for multiple load levels on a target server that is otherwise unavailable for testing. We evaluate our approach by predicting the power consumption of three applications on different physical servers. Our method is able to achieve an average prediction error of 9.49% for three workloads running on real-world, physical servers.},
  address = {New York, NY, USA},
  author = {von Kistowski, J{\'o}akim and Grohmann, Johannes and Schmitt, Norbert and Kounev, Samuel},
  booktitle = {Proceedings of the 19th ACM/SPEC International Conference on Performance Engineering},
  keywords = {Power},
  note = {{Full Paper Acceptance Rate: 18.6\% (13/70)}},
  pages = {301–312},
  publisher = {Association for Computing Machinery (ACM)},
  series = {ICPE '19},
  title = {{Predicting Server Power Consumption from Standard Rating Results}},
  year = 2019
}

%0 Conference Paper
%1 KiGrScKo2019-ICPE-PowerPrediction
%A von Kistowski, J{ó}akim
%A Grohmann, Johannes
%A Schmitt, Norbert
%A Kounev, Samuel
%B Proceedings of the 19th ACM/SPEC International Conference on Performance Engineering
%C New York, NY, USA
%D 2019
%I Association for Computing Machinery (ACM)
%P 301--312
%R 10.1145/3297663.3310298
%T {Predicting Server Power Consumption from Standard Rating Results}
%U https://doi.org/10.1145/3297663.3310298
%X Data center providers and server operators try to reduce the power consumption of their servers. Finding an energy efficient server for a specific target application is a first step in this regard. Estimating the power consumption of an application on an unavailable server is difficult, as nameplate power values are generally overestimations. Offline power models are able to predict the consumption accurately, but are usually intended for system design, requiring very specific and detailed knowledge about the system under consideration. In this paper, we introduce an offline power prediction method that uses the results of standard power rating tools. The method predicts the power consumption of a specific application for multiple load levels on a target server that is otherwise unavailable for testing. We evaluate our approach by predicting the power consumption of three applications on different physical servers. Our method is able to achieve an average prediction error of 9.49% for three workloads running on real-world, physical servers.
%@ 9781450362399

TeaStore: A Micro-Service Reference Application for Cloud Researchers. Eismann, Simon; v. Kistowski, J{ó}akim; Grohmann, Johannes; Bauer, Andr{é}; Schmitt, Norbert; Herbst, Nikolas; Kounev, Samuel; in Proceedings of 2018 IEEE/ACM International Conference on Utility and Cloud Computing Companion (UCC Companion) (2018). 11–12. IEEE.

[ Abstract ]
[ BibTeX ]
[ EndNote ]
[ URL ]
[ DOI ]
[ Download ]

Researchers propose and employ various methods to analyze, model, optimize and manage modern distributed cloud applications. In order to demonstrate and evaluate these methods in realistic scenarios, researchers rely on reference applications. These applications should offer a range of different behaviors, degrees of freedom allowing for customization and should use a modern and representative technology stack. Existing testing and benchmarking applications are either outdated, designed for specific testing scenarios, or do not offer the necessary degrees of freedom. Further, most cloud reference applications are difficult to deploy and run. In this paper, we present the TeaStore, a micro-service-based test and reference cloud application. TeaStore offers services with various performance characteristics and a high degree of freedom regarding its deployment and configuration to be used as a cloud reference application for researchers. The TeaStore is designed for the evaluation of performance modeling and resource management techniques. We invite cloud researchers to use the TeaStore and provide it open-source, extendable, easily deployable and monitorable.

@inproceedings{EiKiGrBaScHeKo2018-UCC-TeastorePoster,
  abstract = {Researchers propose and employ various methods to analyze, model, optimize and manage modern distributed cloud applications. In order to demonstrate and evaluate these methods in realistic scenarios, researchers rely on reference applications. These applications should offer a range of different behaviors, degrees of freedom allowing for customization and should use a modern and representative technology stack. Existing testing and benchmarking applications are either outdated, designed for specific testing scenarios, or do not offer the necessary degrees of freedom. Further, most cloud reference applications are difficult to deploy and run. In this paper, we present the TeaStore, a micro-service-based test and reference cloud application. TeaStore offers services with various performance characteristics and a high degree of freedom regarding its deployment and configuration to be used as a cloud reference application for researchers. The TeaStore is designed for the evaluation of performance modeling and resource management techniques. We invite cloud researchers to use the TeaStore and provide it open-source, extendable, easily deployable and monitorable.},
  author = {Eismann, Simon and v. Kistowski, J{\'o}akim and Grohmann, Johannes and Bauer, Andr{\'e} and Schmitt, Norbert and Herbst, Nikolas and Kounev, Samuel},
  booktitle = {Proceedings of 2018 IEEE/ACM International Conference on Utility and Cloud Computing Companion (UCC Companion)},
  keywords = {t_poster},
  month = 12,
  pages = {11–12},
  publisher = {IEEE},
  title = {TeaStore: A Micro-Service Reference Application for Cloud Researchers},
  year = 2018
}

%0 Conference Paper
%1 EiKiGrBaScHeKo2018-UCC-TeastorePoster
%A Eismann, Simon
%A v. Kistowski, J{ó}akim
%A Grohmann, Johannes
%A Bauer, Andr{é}
%A Schmitt, Norbert
%A Herbst, Nikolas
%A Kounev, Samuel
%B Proceedings of 2018 IEEE/ACM International Conference on Utility and Cloud Computing Companion (UCC Companion)
%D 2018
%I IEEE
%P 11--12
%R 10.1109/UCC-Companion.2018.00021
%T TeaStore: A Micro-Service Reference Application for Cloud Researchers
%U https://doi.org/10.1109/UCC-Companion.2018.00021
%X Researchers propose and employ various methods to analyze, model, optimize and manage modern distributed cloud applications. In order to demonstrate and evaluate these methods in realistic scenarios, researchers rely on reference applications. These applications should offer a range of different behaviors, degrees of freedom allowing for customization and should use a modern and representative technology stack. Existing testing and benchmarking applications are either outdated, designed for specific testing scenarios, or do not offer the necessary degrees of freedom. Further, most cloud reference applications are difficult to deploy and run. In this paper, we present the TeaStore, a micro-service-based test and reference cloud application. TeaStore offers services with various performance characteristics and a high degree of freedom regarding its deployment and configuration to be used as a cloud reference application for researchers. The TeaStore is designed for the evaluation of performance modeling and resource management techniques. We invite cloud researchers to use the TeaStore and provide it open-source, extendable, easily deployable and monitorable.

{Run-time Prediction of Power Consumption for Component Deployments}. von Kistowski, J{ó}akim; Deffner, Maximilian; Kounev, Samuel; in Proceedings of the 15th IEEE International Conference on Autonomic Computing (ICAC 2018) (2018).

[ Abstract ]
[ BibTeX ]
[ EndNote ]
[ Download ]

The Power consumption of servers in data centers depends greatly on the software running on each server and how it interacts with the hardware. Different deployments of distributed software components on heterogeneous servers can lead to significant differences in power consumption, depending on the server allocation and the current workload. As workloads and load intensity change, components may be re-deployed or exchanged in order to reduce the power consumption for the current load profile. The decision on which component to place on which server during run-time remains difficult as the power consumption that would result from such a placement remains unknown. Existing work on component deployment optimization at run-time focuses on maximizing performance or considers power in the context of static design time decisions. In this paper, we introduce a model to predict the power consumption of component placements at run-time based on the load and power profile collected for a running distributed application in a heterogeneous environment. In addition, we present a model that enables the use of our approach without dedicated power measurement devices, predicting power consumption based on load intensity and performance counters. We show that we can predict the power consumption of two different distributed web applications with a mean absolute percentage error of 2.21% and with an error of 1.04% when predicting a previously unobserved load level.

@inproceedings{KiDeKo2018-ICAC-PowerPrediction,
  abstract = {The Power consumption of servers in data centers depends greatly on the software running on each server and how it interacts with the hardware. Different deployments of distributed software components on heterogeneous servers can lead to significant differences in power consumption, depending on the server allocation and the current workload. As workloads and load intensity change, components may be re-deployed or exchanged in order to reduce the power consumption for the current load profile. The decision on which component to place on which server during run-time remains difficult as the power consumption that would result from such a placement remains unknown. Existing work on component deployment optimization at run-time focuses on maximizing performance or considers power in the context of static design time decisions. In this paper, we introduce a model to predict the power consumption of component placements at run-time based on the load and power profile collected for a running distributed application in a heterogeneous environment. In addition, we present a model that enables the use of our approach without dedicated power measurement devices, predicting power consumption based on load intensity and performance counters. We show that we can predict the power consumption of two different distributed web applications with a mean absolute percentage error of 2.21% and with an error of 1.04% when predicting a previously unobserved load level.},
  author = {von Kistowski, J{\'o}akim and Deffner, Maximilian and Kounev, Samuel},
  booktitle = {Proceedings of the 15th IEEE International Conference on Autonomic Computing (ICAC 2018)},
  keywords = {Power},
  month = {09},
  title = {{Run-time Prediction of Power Consumption for Component Deployments}},
  year = 2018
}

%0 Conference Paper
%1 KiDeKo2018-ICAC-PowerPrediction
%A von Kistowski, J{ó}akim
%A Deffner, Maximilian
%A Kounev, Samuel
%B Proceedings of the 15th IEEE International Conference on Autonomic Computing (ICAC 2018)
%D 2018
%T {Run-time Prediction of Power Consumption for Component Deployments}
%X The Power consumption of servers in data centers depends greatly on the software running on each server and how it interacts with the hardware. Different deployments of distributed software components on heterogeneous servers can lead to significant differences in power consumption, depending on the server allocation and the current workload. As workloads and load intensity change, components may be re-deployed or exchanged in order to reduce the power consumption for the current load profile. The decision on which component to place on which server during run-time remains difficult as the power consumption that would result from such a placement remains unknown. Existing work on component deployment optimization at run-time focuses on maximizing performance or considers power in the context of static design time decisions. In this paper, we introduce a model to predict the power consumption of component placements at run-time based on the load and power profile collected for a running distributed application in a heterogeneous environment. In addition, we present a model that enables the use of our approach without dedicated power measurement devices, predicting power consumption based on load intensity and performance counters. We show that we can predict the power consumption of two different distributed web applications with a mean absolute percentage error of 2.21% and with an error of 1.04% when predicting a previously unobserved load level.

{TeaStore: A Micro-Service Reference Application for Benchmarking, Modeling and Resource Management Research}. von Kistowski, J{ó}akim; Eismann, Simon; Schmitt, Norbert; Bauer, Andr{é}; Grohmann, Johannes; Kounev, Samuel; in Proceedings of the 26th IEEE International Symposium on the Modelling, Analysis, and Simulation of Computer and Telecommunication Systems (2018). 223–236. IEEE Computer Society.

{Acceptance Rate: 29.5\% (23/78)}

[ Abstract ]
[ BibTeX ]
[ EndNote ]
[ URL ]
[ DOI ]
[ Download ]

Modern distributed applications offer complex performance behavior and many degrees of freedom regarding deployment and configuration. Researchers employ various methods of analysis, modeling, and management that leverage these degrees of freedom to predict or improve non-functional properties of the software under consideration. In order to demonstrate and evaluate their applicability in the real world, methods resulting from such research areas require test and reference applications that offer a range of different behaviors, as well as the necessary degrees of freedom. Existing production software is often inaccessible for researchers or closed off to instrumentation. Existing testing and benchmarking frameworks, on the other hand, are either designed for specific testing scenarios, or they do not offer the necessary degrees of freedom. Further, most test applications are difficult to deploy and run, or are outdated. In this paper, we introduce the TeaStore, a state-of-the-art micro-service-based test and reference application. TeaStore offers services with different performance characteristics and many degrees of freedom regarding deployment and configuration to be used as a benchmarking framework for researchers. The TeaStore allows evaluating performance modeling and resource management techniques; it also offers instrumented variants to enable extensive run-time analysis. We demonstrate TeaStore's use in three contexts: performance modeling, cloud resource management, and energy efficiency analysis. Our experiments show that TeaStore can be used for evaluating novel approaches in these contexts and also motivates further research in the areas of performance modeling and resource management.

@inproceedings{KiEiScBaGrKo2018-MASCOTS-TeaStore,
  abstract = {Modern distributed applications offer complex performance behavior and many degrees of freedom regarding deployment and configuration. Researchers employ various methods of analysis, modeling, and management that leverage these degrees of freedom to predict or improve non-functional properties of the software under consideration. In order to demonstrate and evaluate their applicability in the real world, methods resulting from such research areas require test and reference applications that offer a range of different behaviors, as well as the necessary degrees of freedom. Existing production software is often inaccessible for researchers or closed off to instrumentation. Existing testing and benchmarking frameworks, on the other hand, are either designed for specific testing scenarios, or they do not offer the necessary degrees of freedom. Further, most test applications are difficult to deploy and run, or are outdated. In this paper, we introduce the TeaStore, a state-of-the-art micro-service-based test and reference application. TeaStore offers services with different performance characteristics and many degrees of freedom regarding deployment and configuration to be used as a benchmarking framework for researchers. The TeaStore allows evaluating performance modeling and resource management techniques; it also offers instrumented variants to enable extensive run-time analysis. We demonstrate TeaStore's use in three contexts: performance modeling, cloud resource management, and energy efficiency analysis. Our experiments show that TeaStore can be used for evaluating novel approaches in these contexts and also motivates further research in the areas of performance modeling and resource management.},
  author = {von Kistowski, J{\'o}akim and Eismann, Simon and Schmitt, Norbert and Bauer, Andr{\'e} and Grohmann, Johannes and Kounev, Samuel},
  booktitle = {Proceedings of the 26th IEEE International Symposium on the Modelling, Analysis, and Simulation of Computer and Telecommunication Systems},
  keywords = {Power},
  month = {09},
  note = {{Acceptance Rate: 29.5\% (23/78)}},
  pages = {223–236},
  publisher = {IEEE Computer Society},
  series = {MASCOTS '18},
  title = {{TeaStore: A Micro-Service Reference Application for Benchmarking, Modeling and Resource Management Research}},
  year = 2018
}

%0 Conference Paper
%1 KiEiScBaGrKo2018-MASCOTS-TeaStore
%A von Kistowski, J{ó}akim
%A Eismann, Simon
%A Schmitt, Norbert
%A Bauer, Andr{é}
%A Grohmann, Johannes
%A Kounev, Samuel
%B Proceedings of the 26th IEEE International Symposium on the Modelling, Analysis, and Simulation of Computer and Telecommunication Systems
%D 2018
%I IEEE Computer Society
%P 223--236
%R 10.1109/MASCOTS.2018.00030
%T {TeaStore: A Micro-Service Reference Application for Benchmarking, Modeling and Resource Management Research}
%U https://doi.org/10.1109/MASCOTS.2018.00030
%X Modern distributed applications offer complex performance behavior and many degrees of freedom regarding deployment and configuration. Researchers employ various methods of analysis, modeling, and management that leverage these degrees of freedom to predict or improve non-functional properties of the software under consideration. In order to demonstrate and evaluate their applicability in the real world, methods resulting from such research areas require test and reference applications that offer a range of different behaviors, as well as the necessary degrees of freedom. Existing production software is often inaccessible for researchers or closed off to instrumentation. Existing testing and benchmarking frameworks, on the other hand, are either designed for specific testing scenarios, or they do not offer the necessary degrees of freedom. Further, most test applications are difficult to deploy and run, or are outdated. In this paper, we introduce the TeaStore, a state-of-the-art micro-service-based test and reference application. TeaStore offers services with different performance characteristics and many degrees of freedom regarding deployment and configuration to be used as a benchmarking framework for researchers. The TeaStore allows evaluating performance modeling and resource management techniques; it also offers instrumented variants to enable extensive run-time analysis. We demonstrate TeaStore's use in three contexts: performance modeling, cloud resource management, and energy efficiency analysis. Our experiments show that TeaStore can be used for evaluating novel approaches in these contexts and also motivates further research in the areas of performance modeling and resource management.

{Modeling of Parametric Dependencies for Performance Prediction of Component-based Software Systems at Run-time}. Eismann, Simon; Walter, J{ü}rgen; von Kistowski, J{ó}akim; Kounev, Samuel; in 2018 IEEE International Conference on Software Architecture (ICSA) (2018). 135–144.

Acceptance Rate: 25,6\% (22/86)

[ Abstract ]
[ BibTeX ]
[ EndNote ]
[ URL ]
[ Download ]

Model-based performance analysis can be leveraged to explore performance properties of software systems. To capture the behavior of varying workload mixes, configurations, and deployments of a software system requires formal modeling of the impact of configuration parameters and user input on the system behavior. Such influences are represented as parametric dependencies in software performance models. Existing modeling approaches focus on modeling parametric dependencies at design-time. This paper identifies runtime specific parametric dependency features, which are not supported by existing work. Therefore, this paper proposes a novel modeling methodology for parametric dependencies and a corresponding graph-based resolution algorithm. This algorithm enables the solution of models containing component instance-level dependencies, variables with multiple descriptions in parallel, and correlations modeled as parametric dependencies. We integrate our work into the Descartes Modeling Language (DML), allowing for accurate and efficient modeling and analysis of parametric dependencies. These performance predictions are valuable for various purposes such as capacity planning, bottleneck analysis, configuration optimization and proactive auto-scaling. Our evaluation analyzes a video store application. The prediction for varying language mixes and video sizes shows a mean error below 5% for utilization and below 10% for response time.

@inproceedings{EiWaKiKo2018-ParametricDependencies,
  abstract = {Model-based performance analysis can be leveraged to explore performance properties of software systems. To capture the behavior of varying workload mixes, configurations, and deployments of a software system requires formal modeling of the impact of configuration parameters and user input on the system behavior. Such influences are represented as parametric dependencies in software performance models. Existing modeling approaches focus on modeling parametric dependencies at design-time. This paper identifies runtime specific parametric dependency features, which are not supported by existing work. Therefore, this paper proposes a novel modeling methodology for parametric dependencies and a corresponding graph-based resolution algorithm. This algorithm enables the solution of models containing component instance-level dependencies, variables with multiple descriptions in parallel, and correlations modeled as parametric dependencies. We integrate our work into the Descartes Modeling Language (DML), allowing for accurate and efficient modeling and analysis of parametric dependencies. These performance predictions are valuable for various purposes such as capacity planning, bottleneck analysis, configuration optimization and proactive auto-scaling. Our evaluation analyzes a video store application. The prediction for varying language mixes and video sizes shows a mean error below 5% for utilization and below 10% for response time.},
  author = {Eismann, Simon and Walter, J{\"u}rgen and von Kistowski, J{\'o}akim and Kounev, Samuel},
  booktitle = {2018 IEEE International Conference on Software Architecture (ICSA)},
  keywords = {Meta-models},
  month = {04},
  note = {Acceptance Rate: 25,6\% (22/86)},
  pages = {135-144},
  title = {{Modeling of Parametric Dependencies for Performance Prediction of Component-based Software Systems at Run-time}},
  year = 2018
}

%0 Conference Paper
%1 EiWaKiKo2018-ParametricDependencies
%A Eismann, Simon
%A Walter, J{ü}rgen
%A von Kistowski, J{ó}akim
%A Kounev, Samuel
%B 2018 IEEE International Conference on Software Architecture (ICSA)
%D 2018
%P 135-144
%T {Modeling of Parametric Dependencies for Performance Prediction of Component-based Software Systems at Run-time}
%U https://ieeexplore.ieee.org/document/8417146
%X Model-based performance analysis can be leveraged to explore performance properties of software systems. To capture the behavior of varying workload mixes, configurations, and deployments of a software system requires formal modeling of the impact of configuration parameters and user input on the system behavior. Such influences are represented as parametric dependencies in software performance models. Existing modeling approaches focus on modeling parametric dependencies at design-time. This paper identifies runtime specific parametric dependency features, which are not supported by existing work. Therefore, this paper proposes a novel modeling methodology for parametric dependencies and a corresponding graph-based resolution algorithm. This algorithm enables the solution of models containing component instance-level dependencies, variables with multiple descriptions in parallel, and correlations modeled as parametric dependencies. We integrate our work into the Descartes Modeling Language (DML), allowing for accurate and efficient modeling and analysis of parametric dependencies. These performance predictions are valuable for various purposes such as capacity planning, bottleneck analysis, configuration optimization and proactive auto-scaling. Our evaluation analyzes a video store application. The prediction for varying language mixes and video sizes shows a mean error below 5% for utilization and below 10% for response time.

{TeaStore - A Micro-Service Application for Benchmarking, Modeling and Resource Management Research}. von Kistowski, J{ó}akim; (2018, April).

[ BibTeX ]
[ EndNote ]
[ Download ]

@misc{talk-Kistowski-HotCloudPerf18,
  author = {von Kistowski, J{\'o}akim},
  howpublished = {Talk at Workshop on Hot Topics in Cloud Computing Performance 2018},
  keywords = {Power},
  month = {04},
  title = {{TeaStore - A Micro-Service Application for Benchmarking, Modeling and Resource Management Research}},
  year = 2018
}

%0 Generic
%1 talk-Kistowski-HotCloudPerf18
%A von Kistowski, J{ó}akim
%D 2018
%T {TeaStore - A Micro-Service Application for Benchmarking, Modeling and Resource Management Research}

{SPEC CPU2017: Next-Generation Compute Benchmark}. Bucek, James; Lange, Klaus-Dieter; von Kistowski, J{ó}akim; in {Companion of the 2018 ACM/SPEC International Conference on Performance Engineering} (2018). 41–42. ACM, New York, NY, USA.

[ BibTeX ]
[ EndNote ]
[ URL ]

@inproceedings{BuLaKi2018-SPECCPU,
  address = {New York, NY, USA},
  author = {Bucek, James and Lange, Klaus-Dieter and von Kistowski, J{\'o}akim},
  booktitle = {{Companion of the 2018 ACM/SPEC International Conference on Performance Engineering}},
  keywords = {t_poster},
  pages = {41–42},
  publisher = {ACM},
  series = {ICPE '18},
  title = {{SPEC CPU2017: Next-Generation Compute Benchmark}},
  year = 2018
}

%0 Conference Paper
%1 BuLaKi2018-SPECCPU
%A Bucek, James
%A Lange, Klaus-Dieter
%A von Kistowski, J{ó}akim
%B {Companion of the 2018 ACM/SPEC International Conference on Performance Engineering}
%C New York, NY, USA
%D 2018
%I ACM
%P 41--42
%T {SPEC CPU2017: Next-Generation Compute Benchmark}
%U http://doi.acm.org/10.1145/3185768.3185771

{Measuring and Benchmarking Power Consumption and Energy Efficiency}. von Kistowski, J{ó}akim; Lange, Klaus-Dieter; Arnold, Jeremy A.; Sharma, Sanjay; Pais, Johann; Block, Hansfried; in Companion of the 2018 ACM/SPEC International Conference on Performance Engineering (2018). 57–65. ACM, New York, NY, USA.

[ BibTeX ]
[ EndNote ]
[ URL ]
[ Download ]

@inproceedings{KiLaArShPaBl2018-ICPE-Power-Tutorial,
  address = {New York, NY, USA},
  author = {von Kistowski, J{\'o}akim and Lange, Klaus-Dieter and Arnold, Jeremy A. and Sharma, Sanjay and Pais, Johann and Block, Hansfried},
  booktitle = {Companion of the 2018 ACM/SPEC International Conference on Performance Engineering},
  keywords = {Power},
  pages = {57–65},
  publisher = {ACM},
  series = {ICPE '18},
  title = {{Measuring and Benchmarking Power Consumption and Energy Efficiency}},
  year = 2018
}

%0 Conference Paper
%1 KiLaArShPaBl2018-ICPE-Power-Tutorial
%A von Kistowski, J{ó}akim
%A Lange, Klaus-Dieter
%A Arnold, Jeremy A.
%A Sharma, Sanjay
%A Pais, Johann
%A Block, Hansfried
%B Companion of the 2018 ACM/SPEC International Conference on Performance Engineering
%C New York, NY, USA
%D 2018
%I ACM
%P 57--65
%T {Measuring and Benchmarking Power Consumption and Energy Efficiency}
%U http://doi.acm.org/10.1145/3185768.3185775

Energy Benchmarking. Lange, Klaus-Dieter; von Kistowski, J{ó}akim; in Encyclopedia of Big Data Technologies, S. Sakr, A. Zomaya (eds.) (2018). 1–6. Springer International Publishing, Cham.

[ BibTeX ]
[ EndNote ]
[ URL ]

@incollection{LaKi2018-BigDataEnergy,
  address = {Cham},
  author = {Lange, Klaus-Dieter and von Kistowski, J{\'o}akim},
  booktitle = {Encyclopedia of Big Data Technologies},
  editor = {Sakr, Sherif and Zomaya, Albert},
  keywords = {Power},
  pages = {1–6},
  publisher = {Springer International Publishing},
  title = {Energy Benchmarking},
  year = 2018
}

%0 Book Section
%1 LaKi2018-BigDataEnergy
%A Lange, Klaus-Dieter
%A von Kistowski, J{ó}akim
%B Encyclopedia of Big Data Technologies
%C Cham
%D 2018
%E Sakr, Sherif
%E Zomaya, Albert
%I Springer International Publishing
%P 1--6
%T Energy Benchmarking
%U https://doi.org/10.1007/978-3-319-63962-8_119-1

{Emulating the Power Consumption Behavior of Server Workloads using CPU Performance Counters}. Schmitt, Norbert; von Kistowski, J{ó}akim; Kounev, Samuel; in Proceedings of the 25th IEEE International Symposium on the Modelling, Analysis, and Simulation of Computer and Telecommunication Systems (2017).

{(Short Paper) Acceptance Rate: 26/84 = 31\%}

[ Abstract ]
[ BibTeX ]
[ EndNote ]
[ Download ]

The accurate measurement of a server's power consumption when running realistic workloads enables characterization of its energy efficiency and helps to make better provisioning and workload placement decisions. Information on the energy efficiency of a server for a given target workload can greatly influence such decisions and thus the final energy efficiency of a cluster or data center. However, measuring energy efficiency and power consumption of server applications has become challenging as applications are often distributed or require work intensive configuration, setup, and specialized load drivers for reproducible testing. As a result, it may be not feasible to perform tests using the actual workload that is to be deployed. We introduce an approach to create small-scale workloads that emulate the power consumption-relevant behavior of an application by deliberately triggering specific power relevant performance counter events. These workloads can then be easily deployed on a target server for fast and efficient power characterization. We validate the proposed approach by approximating the power consumption behavior of different workloads at multiple load levels. We show that our approach is capable of producing small-scale workloads that reflect the power consumption behavior of their reference applications over multiple load levels with a minimum error of less than 1%.

@inproceedings{ScKiKo2017-MASCOTS-Pet,
  abstract = {The accurate measurement of a server's power consumption when running realistic workloads enables characterization of its energy efficiency and helps to make better provisioning and workload placement decisions. Information on the energy efficiency of a server for a given target workload can greatly influence such decisions and thus the final energy efficiency of a cluster or data center. However, measuring energy efficiency and power consumption of server applications has become challenging as applications are often distributed or require work intensive configuration, setup, and specialized load drivers for reproducible testing. As a result, it may be not feasible to perform tests using the actual workload that is to be deployed. We introduce an approach to create small-scale workloads that emulate the power consumption-relevant behavior of an application by deliberately triggering specific power relevant performance counter events. These workloads can then be easily deployed on a target server for fast and efficient power characterization. We validate the proposed approach by approximating the power consumption behavior of different workloads at multiple load levels. We show that our approach is capable of producing small-scale workloads that reflect the power consumption behavior of their reference applications over multiple load levels with a minimum error of less than 1%.},
  author = {Schmitt, Norbert and von Kistowski, J{\'o}akim and Kounev, Samuel},
  booktitle = {Proceedings of the 25th IEEE International Symposium on the Modelling, Analysis, and Simulation of Computer and Telecommunication Systems},
  keywords = {Power},
  month = {09},
  note = {{(Short Paper) Acceptance Rate: 26/84 = 31\%}},
  series = {MASCOTS '17},
  title = {{Emulating the Power Consumption Behavior of Server Workloads using CPU Performance Counters}},
  year = 2017
}

%0 Conference Paper
%1 ScKiKo2017-MASCOTS-Pet
%A Schmitt, Norbert
%A von Kistowski, J{ó}akim
%A Kounev, Samuel
%B Proceedings of the 25th IEEE International Symposium on the Modelling, Analysis, and Simulation of Computer and Telecommunication Systems
%D 2017
%T {Emulating the Power Consumption Behavior of Server Workloads using CPU Performance Counters}
%X The accurate measurement of a server's power consumption when running realistic workloads enables characterization of its energy efficiency and helps to make better provisioning and workload placement decisions. Information on the energy efficiency of a server for a given target workload can greatly influence such decisions and thus the final energy efficiency of a cluster or data center. However, measuring energy efficiency and power consumption of server applications has become challenging as applications are often distributed or require work intensive configuration, setup, and specialized load drivers for reproducible testing. As a result, it may be not feasible to perform tests using the actual workload that is to be deployed. We introduce an approach to create small-scale workloads that emulate the power consumption-relevant behavior of an application by deliberately triggering specific power relevant performance counter events. These workloads can then be easily deployed on a target server for fast and efficient power characterization. We validate the proposed approach by approximating the power consumption behavior of different workloads at multiple load levels. We show that our approach is capable of producing small-scale workloads that reflect the power consumption behavior of their reference applications over multiple load levels with a minimum error of less than 1%.

{Towards a Scalability and Energy Efficiency Benchmark for VNF}. Schmitt, Norbert; von Kistowski, J{ó}akim; Kounev, Samuel; in Proceedings of the 9th TPC Technology Conference on Performance Engineering and Benchmarking (2017).

[ Abstract ]
[ BibTeX ]
[ EndNote ]
[ Download ]

Network Function Virtualization (NFV) is the transfer of network functions from dedicated devices to high-volume commodity servers. It opens opportunities for flexibility and energy savings. Concrete insights on the flexibility of specific NFV environments require measurement methodologies and benchmarks. However, current benchmarks are not measuring the ability of a virtual network function (VNF) to scale either horizontally or vertically. We therefore envision a new benchmark that measures a VNF's ability to scale while evaluating its energy efficiency at the same time. Such a benchmark would enable the selection of a suitable VNF for changing demands, deployed at an existing or new resource landscape, while minimizing energy costs.

@inproceedings{ScKiKo2017-TPCTC-BenchmarkVision,
  abstract = {Network Function Virtualization (NFV) is the transfer of network functions from dedicated devices to high-volume commodity servers. It opens opportunities for flexibility and energy savings. Concrete insights on the flexibility of specific NFV environments require measurement methodologies and benchmarks. However, current benchmarks are not measuring the ability of a virtual network function (VNF) to scale either horizontally or vertically. We therefore envision a new benchmark that measures a VNF's ability to scale while evaluating its energy efficiency at the same time. Such a benchmark would enable the selection of a suitable VNF for changing demands, deployed at an existing or new resource landscape, while minimizing energy costs.},
  author = {Schmitt, Norbert and von Kistowski, J{\'o}akim and Kounev, Samuel},
  booktitle = {Proceedings of the 9th TPC Technology Conference on Performance Engineering and Benchmarking},
  keywords = {Power},
  month = {08},
  series = {TPCTC '17},
  title = {{Towards a Scalability and Energy Efficiency Benchmark for VNF}},
  year = 2017
}

%0 Conference Paper
%1 ScKiKo2017-TPCTC-BenchmarkVision
%A Schmitt, Norbert
%A von Kistowski, J{ó}akim
%A Kounev, Samuel
%B Proceedings of the 9th TPC Technology Conference on Performance Engineering and Benchmarking
%D 2017
%T {Towards a Scalability and Energy Efficiency Benchmark for VNF}
%X Network Function Virtualization (NFV) is the transfer of network functions from dedicated devices to high-volume commodity servers. It opens opportunities for flexibility and energy savings. Concrete insights on the flexibility of specific NFV environments require measurement methodologies and benchmarks. However, current benchmarks are not measuring the ability of a virtual network function (VNF) to scale either horizontally or vertically. We therefore envision a new benchmark that measures a VNF's ability to scale while evaluating its energy efficiency at the same time. Such a benchmark would enable the selection of a suitable VNF for changing demands, deployed at an existing or new resource landscape, while minimizing energy costs.

{SPECpower Research Working Group}. von Kistowski, J{ó}akim; (2017, April).

[ BibTeX ]
[ EndNote ]

@misc{talk-Kistowski-SPECRG2017,
  author = {von Kistowski, J{\'o}akim},
  howpublished = {Talk at SPEC RG Annual Meeting 2017},
  keywords = {Power},
  month = {04},
  title = {{SPECpower Research Working Group}},
  year = 2017
}

%0 Generic
%1 talk-Kistowski-SPECRG2017
%A von Kistowski, J{ó}akim
%D 2017
%T {SPECpower Research Working Group}

{Autopilot: Enabling easy Benchmarking of Workload Energy Efficiency}. von Kistowski, J{ó}akim; Deffner, Maximilian; Arnold, Jeremy A.; Lange, Klaus-Dieter; Beckett, John; Kounev, Samuel; in Proceedings of the 8th ACM/SPEC International Conference on Performance Engineering (ICPE 2017) (2017). ACM, New York, NY, USA.

Best Demo Award

[ Abstract ]
[ BibTeX ]
[ EndNote ]
[ URL ]
[ Download ]

Benchmarking of energy efficiency is important as it helps researchers, customers, and developers to evaluate and compare the energy efficiency of software and hardware solutions. Developing and deploying energy-efficiency benchmarking workloads are challenging tasks, as work must be able to be executed in a power measurement environment using an energy-efficiency measurement methodology. The existing SPEC Chauffeur Worklet Development Kit (WDK) enables the development and use of custom workloads (called worklets) within a standardized power measurement methodology. However, it features no integration in development environments, making building and deployment of workloads challenging. We address this challenge by proposing Autopilot, a plugin for the Eclipse IDE. Autopilot enables fast and easy building and deployment of a workload under development on a system for testing. It also enables benchmark execution directly from the development environment.

@inproceedings{KiDeArLaBeKo2017-ICPE-Autopilot,
  abstract = {Benchmarking of energy efficiency is important as it helps researchers, customers, and developers to evaluate and compare the energy efficiency of software and hardware solutions. Developing and deploying energy-efficiency benchmarking workloads are challenging tasks, as work must be able to be executed in a power measurement environment using an energy-efficiency measurement methodology. The existing SPEC Chauffeur Worklet Development Kit (WDK) enables the development and use of custom workloads (called worklets) within a standardized power measurement methodology. However, it features no integration in development environments, making building and deployment of workloads challenging. We address this challenge by proposing Autopilot, a plugin for the Eclipse IDE. Autopilot enables fast and easy building and deployment of a workload under development on a system for testing. It also enables benchmark execution directly from the development environment.},
  address = {New York, NY, USA},
  author = {von Kistowski, J{\'o}akim and Deffner, Maximilian and Arnold, Jeremy A. and Lange, Klaus-Dieter and Beckett, John and Kounev, Samuel},
  booktitle = {Proceedings of the 8th ACM/SPEC International Conference on Performance Engineering (ICPE 2017)},
  keywords = {Power},
  month = {04},
  note = {<b>Best Demo Award</b>},
  publisher = {ACM},
  title = {{Autopilot: Enabling easy Benchmarking of Workload Energy Efficiency}},
  year = 2017
}

%0 Conference Paper
%1 KiDeArLaBeKo2017-ICPE-Autopilot
%A von Kistowski, J{ó}akim
%A Deffner, Maximilian
%A Arnold, Jeremy A.
%A Lange, Klaus-Dieter
%A Beckett, John
%A Kounev, Samuel
%B Proceedings of the 8th ACM/SPEC International Conference on Performance Engineering (ICPE 2017)
%C New York, NY, USA
%D 2017
%I ACM
%T {Autopilot: Enabling easy Benchmarking of Workload Energy Efficiency}
%U https://doi.org/10.1145/3030207.3053667
%X Benchmarking of energy efficiency is important as it helps researchers, customers, and developers to evaluate and compare the energy efficiency of software and hardware solutions. Developing and deploying energy-efficiency benchmarking workloads are challenging tasks, as work must be able to be executed in a power measurement environment using an energy-efficiency measurement methodology. The existing SPEC Chauffeur Worklet Development Kit (WDK) enables the development and use of custom workloads (called worklets) within a standardized power measurement methodology. However, it features no integration in development environments, making building and deployment of workloads challenging. We address this challenge by proposing Autopilot, a plugin for the Eclipse IDE. Autopilot enables fast and easy building and deployment of a workload under development on a system for testing. It also enables benchmark execution directly from the development environment.

{The SERT Metric and the Impact of Server Configuration} von Kistowski, J{ó}akim; Lange, Klaus-Dieter; Arnold, Jeremy A.; Block, Hansfried; Darnell, Greg; Beckett, John; Tricker, Mike; (2017). Standard Performance Evaluation Corporation (SPEC), 7001 Heritage Village Plaza, Suite 225, Gainesville, VA 20155, USA.

[ Abstract ]
[ BibTeX ]
[ EndNote ]
[ URL ]

{The Server Efficiency Rating Tool (SERT) is an established tool for measuring and analyzing the energy efficiency of servers. The SERT suite measuresserver efficiency using multiple workloads, which in turn consist ofsmallscale miniâ€workloads called worklets. Using multiple worklets enables the SERT suite to holistically explore the behavior of many different systems and enables thorough analysis. However, multiple workloads also result in multiple energy efficiency scores.Â Â This document introduces the single SERT metric (SERT Efficiency Score), which can be used to easily compare systems using a single number. This document explains how the SERT metric is calculated. It also illustrates how a system under test (SUT) configuration and changes to this configuration can impact the SERT Efficiency Score and demonstrates this using a running example.}

@techreport{KiLaArBlDaBeTr2017-SERT-metric,
  abstract = {{The Server Efficiency Rating Tool (SERT) is an established tool for measuring and analyzing the energy efficiency of servers. The SERT suite measuresserver efficiency using multiple workloads, which in turn consist ofsmallscale miniâ€workloads called worklets. Using multiple worklets enables the SERT suite to holistically explore the behavior of many different systems and enables thorough analysis. However, multiple workloads also result in multiple energy efficiency scores.Â Â This document introduces the single SERT metric (SERT Efficiency Score), which can be used to easily compare systems using a single number. This document explains how the SERT metric is calculated. It also illustrates how a system under test (SUT) configuration and changes to this configuration can impact the SERT Efficiency Score and demonstrates this using a running example.}},
  address = {7001 Heritage Village Plaza, Suite 225, Gainesville, VA 20155, USA},
  author = {von Kistowski, J{\'o}akim and Lange, Klaus-Dieter and Arnold, Jeremy A. and Block, Hansfried and Darnell, Greg and Beckett, John and Tricker, Mike},
  institution = {Standard Performance Evaluation Corporation (SPEC)},
  keywords = {Power},
  month = {03},
  title = {{The SERT Metric and the Impact of Server Configuration}},
  year = 2017
}

%0 Report
%1 KiLaArBlDaBeTr2017-SERT-metric
%A von Kistowski, J{ó}akim
%A Lange, Klaus-Dieter
%A Arnold, Jeremy A.
%A Block, Hansfried
%A Darnell, Greg
%A Beckett, John
%A Tricker, Mike
%C 7001 Heritage Village Plaza, Suite 225, Gainesville, VA 20155, USA
%D 2017
%T {The SERT Metric and the Impact of Server Configuration}
%U http://spec.org/sert2/SERT-metric.pdf
%X {The Server Efficiency Rating Tool (SERT) is an established tool for measuring and analyzing the energy efficiency of servers. The SERT suite measuresserver efficiency using multiple workloads, which in turn consist ofsmallscale miniâ€workloads called worklets. Using multiple worklets enables the SERT suite to holistically explore the behavior of many different systems and enables thorough analysis. However, multiple workloads also result in multiple energy efficiency scores.Â Â This document introduces the single SERT metric (SERT Efficiency Score), which can be used to easily compare systems using a single number. This document explains how the SERT metric is calculated. It also illustrates how a system under test (SUT) configuration and changes to this configuration can impact the SERT Efficiency Score and demonstrates this using a running example.}

{Modeling and Extracting Load Intensity Profiles}. von Kistowski, J{ó}akim; Herbst, Nikolas; Kounev, Samuel; Groenda, Henning; Stier, Christian; Lehrig, Sebastian; in ACM Transactions on Autonomous and Adaptive Systems (TAAS) (2017). 11(4) 23:1–23:28. ACM, New York, NY, USA.

[ Abstract ]
[ BibTeX ]
[ EndNote ]
[ URL ]
[ Download ]

Today's system developers and operators face the challenge of creating software systems that make efficient use of dynamically allocated resources under highly variable and dynamic load profiles, while at the same time delivering reliable performance. Autonomic controllers, e.g., an advanced auto-scaling mechanism in a cloud computing context, can benefit from an abstracted load model as knowledge to reconfigure on time and precisely. Existing workload characterization approaches have limited support to capture variations the inter-arrival times of incoming work units over time (i.e., a variable load profile). For example, industrial and scientific benchmarks support constant or stepwise increasing load, or inter-arrival times defined by statistical distributions or recorded traces. These options show shortcomings either in representative character of load variation patterns or in abstraction and flexibility of their format. In this article, we present the Descartes Load Intensity Model (DLIM) approach addressing these issues. DLIM provides a modeling formalism for describing load intensity variations over time. A DLIM instance is a compact formal description of a load intensity trace. DLIM-based tools provide features for benchmarking, performance and recorded load intensity trace analysis. As manually obtaining and maintaining DLIM instances becomes time consuming, we contribute three automated extraction methods and devised metrics for comparison and method selection. We discuss how these features are used to enhance system management approaches for adaptations during run-time, and how they are integrated into simulation contexts and enable benchmarking of elastic or adaptive behavior. We show that automatically extracted DLIM instances exhibit an average modeling error of 15.2\% over ten different real-world traces that cover between two weeks and seven months. These results underline DLIM model expressiveness. In terms of accuracy and processing speed, our proposed extraction methods for the descriptive models are comparable to existing time series decomposition methods. Additionally, we illustrate DLIM applicability by outlining approaches of workload modeling in systems engineering that employ or rely on our proposed load intensity modeling formalism.

@article{KiHeKo-TAAS17-ModelExtractLoadProfiles,
  abstract = {Today's system developers and operators face the challenge of creating software systems that make efficient use of dynamically allocated resources under highly variable and dynamic load profiles, while at the same time delivering reliable performance. Autonomic controllers, e.g., an advanced auto-scaling mechanism in a cloud computing context, can benefit from an abstracted load model as knowledge to reconfigure on time and precisely. Existing workload characterization approaches have limited support to capture variations the inter-arrival times of incoming work units over time (i.e., a variable load profile). For example, industrial and scientific benchmarks support constant or stepwise increasing load, or inter-arrival times defined by statistical distributions or recorded traces. These options show shortcomings either in representative character of load variation patterns or in abstraction and flexibility of their format. In this article, we present the Descartes Load Intensity Model (DLIM) approach addressing these issues. DLIM provides a modeling formalism for describing load intensity variations over time. A DLIM instance is a compact formal description of a load intensity trace. DLIM-based tools provide features for benchmarking, performance and recorded load intensity trace analysis. As manually obtaining and maintaining DLIM instances becomes time consuming, we contribute three automated extraction methods and devised metrics for comparison and method selection. We discuss how these features are used to enhance system management approaches for adaptations during run-time, and how they are integrated into simulation contexts and enable benchmarking of elastic or adaptive behavior. We show that automatically extracted DLIM instances exhibit an average modeling error of 15.2\% over ten different real-world traces that cover between two weeks and seven months. These results underline DLIM model expressiveness. In terms of accuracy and processing speed, our proposed extraction methods for the descriptive models are comparable to existing time series decomposition methods. Additionally, we illustrate DLIM applicability by outlining approaches of workload modeling in systems engineering that employ or rely on our proposed load intensity modeling formalism.},
  address = {New York, NY, USA},
  author = {von Kistowski, J{\'o}akim and Herbst, Nikolas and Kounev, Samuel and Groenda, Henning and Stier, Christian and Lehrig, Sebastian},
  journal = {ACM Transactions on Autonomous and Adaptive Systems (TAAS)},
  keywords = {Self-adaptive-systems},
  month = {01},
  number = 4,
  pages = {23:1–23:28},
  publisher = {ACM},
  title = {{Modeling and Extracting Load Intensity Profiles}},
  volume = 11,
  year = 2017
}

%0 Journal Article
%1 KiHeKo-TAAS17-ModelExtractLoadProfiles
%A von Kistowski, J{ó}akim
%A Herbst, Nikolas
%A Kounev, Samuel
%A Groenda, Henning
%A Stier, Christian
%A Lehrig, Sebastian
%C New York, NY, USA
%D 2017
%I ACM
%J ACM Transactions on Autonomous and Adaptive Systems (TAAS)
%N 4
%P 23:1--23:28
%T {Modeling and Extracting Load Intensity Profiles}
%U http://doi.acm.org/10.1145/3019596
%V 11
%X Today's system developers and operators face the challenge of creating software systems that make efficient use of dynamically allocated resources under highly variable and dynamic load profiles, while at the same time delivering reliable performance. Autonomic controllers, e.g., an advanced auto-scaling mechanism in a cloud computing context, can benefit from an abstracted load model as knowledge to reconfigure on time and precisely. Existing workload characterization approaches have limited support to capture variations the inter-arrival times of incoming work units over time (i.e., a variable load profile). For example, industrial and scientific benchmarks support constant or stepwise increasing load, or inter-arrival times defined by statistical distributions or recorded traces. These options show shortcomings either in representative character of load variation patterns or in abstraction and flexibility of their format. In this article, we present the Descartes Load Intensity Model (DLIM) approach addressing these issues. DLIM provides a modeling formalism for describing load intensity variations over time. A DLIM instance is a compact formal description of a load intensity trace. DLIM-based tools provide features for benchmarking, performance and recorded load intensity trace analysis. As manually obtaining and maintaining DLIM instances becomes time consuming, we contribute three automated extraction methods and devised metrics for comparison and method selection. We discuss how these features are used to enhance system management approaches for adaptations during run-time, and how they are integrated into simulation contexts and enable benchmarking of elastic or adaptive behavior. We show that automatically extracted DLIM instances exhibit an average modeling error of 15.2\% over ten different real-world traces that cover between two weeks and seven months. These results underline DLIM model expressiveness. In terms of accuracy and processing speed, our proposed extraction methods for the descriptive models are comparable to existing time series decomposition methods. Additionally, we illustrate DLIM applicability by outlining approaches of workload modeling in systems engineering that employ or rely on our proposed load intensity modeling formalism.

{Predicting Power Consumption of High-Memory-Bandwidth Workloads}. Schmitt, Norbert; von Kistowski, J{ó}akim; Kounev, Samuel; in Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering (2017). 353–356. ACM, New York, NY, USA.

[ Abstract ]
[ BibTeX ]
[ EndNote ]
[ URL ]
[ Download ]

{High performance workloads with high bandwidth memory utilization are among the most power consuming software applications. When writing such applications, developers can directly influence power consumption of the final software through their choice of data size and traversal method, mostly due to caching characteristics. Explicit knowledge on how choices influence power consumption can thus lead to greater overall energy efficiency. In existing work, power prediction for memory accesses and high bandwidth applications requires either detailed measurement information on the system on which the software is executed or it is too generic, not taking significant aspects, such as caching and data size into account. In this paper, we propose a power model that bridges this gap by modeling power consumption based on concrete software properties, while considering hardware characteristics on a more abstract level, characterizing it primarily using publicly available data. The model is designed to enable developers to compare power consumption of implementation alternatives for high memory bandwidth software components. We validate our model by measuring modified versions of the high bandwidth benchmark stream. We show that our model can predict the relative change of power consumption due to implementation changes and the power consumption of a concrete system under test with an average error of 19 percent.}

@inproceedings{ScKiKo2017-ICPE-MemoryPower,
  abstract = {{High performance workloads with high bandwidth memory utilization are among the most power consuming software applications. When writing such applications, developers can directly influence power consumption of the final software through their choice of data size and traversal method, mostly due to caching characteristics. Explicit knowledge on how choices influence power consumption can thus lead to greater overall energy efficiency. In existing work, power prediction for memory accesses and high bandwidth applications requires either detailed measurement information on the system on which the software is executed or it is too generic, not taking significant aspects, such as caching and data size into account. In this paper, we propose a power model that bridges this gap by modeling power consumption based on concrete software properties, while considering hardware characteristics on a more abstract level, characterizing it primarily using publicly available data. The model is designed to enable developers to compare power consumption of implementation alternatives for high memory bandwidth software components. We validate our model by measuring modified versions of the high bandwidth benchmark stream. We show that our model can predict the relative change of power consumption due to implementation changes and the power consumption of a concrete system under test with an average error of 19 percent.}},
  address = {New York, NY, USA},
  author = {Schmitt, Norbert and von Kistowski, J{\'o}akim and Kounev, Samuel},
  booktitle = {Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering},
  keywords = {Power},
  pages = {353–356},
  publisher = {ACM},
  series = {ICPE '17},
  title = {{Predicting Power Consumption of High-Memory-Bandwidth Workloads}},
  year = 2017
}

%0 Conference Paper
%1 ScKiKo2017-ICPE-MemoryPower
%A Schmitt, Norbert
%A von Kistowski, J{ó}akim
%A Kounev, Samuel
%B Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering
%C New York, NY, USA
%D 2017
%I ACM
%P 353--356
%T {Predicting Power Consumption of High-Memory-Bandwidth Workloads}
%U http://doi.acm.org/10.1145/3030207.3030241
%X {High performance workloads with high bandwidth memory utilization are among the most power consuming software applications. When writing such applications, developers can directly influence power consumption of the final software through their choice of data size and traversal method, mostly due to caching characteristics. Explicit knowledge on how choices influence power consumption can thus lead to greater overall energy efficiency. In existing work, power prediction for memory accesses and high bandwidth applications requires either detailed measurement information on the system on which the software is executed or it is too generic, not taking significant aspects, such as caching and data size into account. In this paper, we propose a power model that bridges this gap by modeling power consumption based on concrete software properties, while considering hardware characteristics on a more abstract level, characterizing it primarily using publicly available data. The model is designed to enable developers to compare power consumption of implementation alternatives for high memory bandwidth software components. We validate our model by measuring modified versions of the high bandwidth benchmark stream. We show that our model can predict the relative change of power consumption due to implementation changes and the power consumption of a concrete system under test with an average error of 19 percent.}

Predicting Power Consumption in Virtualized Environments. von Kistowski, Joakim; Schreck, Marco; Kounev, Samuel; in Computer Performance Engineering: 13th European Workshop, EPEW 2016, Chios, Greece, October 5-7, 2016, Proceedings, D. Fiems, M. Paolieri, N. A. Platis (eds.) (2016). 79–93. Springer International Publishing, Cham.

[ Abstract ]
[ BibTeX ]
[ EndNote ]
[ URL ]

Energy efficiency and power consumption of data centers can be improved through conscientious placement of workloads on specific servers. Virtualization is commonly employed nowadays, as it allows for dynamic reallocation of work and abstraction from the concrete server hardware. The ability to predict the power consumption of workloads at different load levels is essential in this context. Prediction approaches can help to make better placement choices at run-time, as well as when purchasing new server equipment, by showing which servers are better suited for the execution of a given target workload. In existing work, power prediction for servers is limited to non-virtualized contexts or it does not take multiple load levels into account. Existing approaches also fail to leverage publicly available data on server efficiency and instead require experiments to be conducted on the target system. This makes these approaches unwieldy when making decision regarding systems that are not yet available to the decision maker. In this paper, we use the readily available data provided by the SPEC SERT to predict the power consumption of workloads for different load levels in virtualized environments. We evaluate our approach comparing predicted results against measurements of power consumption in multiple virtualized environment configurations on a target server that differs significantly from the reference system used for experimentation. We show that power consumption of CPU and storage loads can be reliably predicted with a prediction error of less than 15% across all tested virtualized environment configurations.

@inproceedings{KiScKo-EPEW-VMPower,
  abstract = {Energy efficiency and power consumption of data centers can be improved through conscientious placement of workloads on specific servers. Virtualization is commonly employed nowadays, as it allows for dynamic reallocation of work and abstraction from the concrete server hardware. The ability to predict the power consumption of workloads at different load levels is essential in this context. Prediction approaches can help to make better placement choices at run-time, as well as when purchasing new server equipment, by showing which servers are better suited for the execution of a given target workload. In existing work, power prediction for servers is limited to non-virtualized contexts or it does not take multiple load levels into account. Existing approaches also fail to leverage publicly available data on server efficiency and instead require experiments to be conducted on the target system. This makes these approaches unwieldy when making decision regarding systems that are not yet available to the decision maker. In this paper, we use the readily available data provided by the SPEC SERT to predict the power consumption of workloads for different load levels in virtualized environments. We evaluate our approach comparing predicted results against measurements of power consumption in multiple virtualized environment configurations on a target server that differs significantly from the reference system used for experimentation. We show that power consumption of CPU and storage loads can be reliably predicted with a prediction error of less than 15% across all tested virtualized environment configurations.},
  address = {Cham},
  author = {von Kistowski, Joakim and Schreck, Marco and Kounev, Samuel},
  booktitle = {Computer Performance Engineering: 13th European Workshop, EPEW 2016, Chios, Greece, October 5-7, 2016, Proceedings},
  editor = {Fiems, Dieter and Paolieri, Marco and Platis, N. Agapios},
  keywords = {Power},
  month = {{October}},
  pages = {79–93},
  publisher = {Springer International Publishing},
  title = {Predicting Power Consumption in Virtualized Environments},
  year = 2016
}

%0 Conference Paper
%1 KiScKo-EPEW-VMPower
%A von Kistowski, Joakim
%A Schreck, Marco
%A Kounev, Samuel
%B Computer Performance Engineering: 13th European Workshop, EPEW 2016, Chios, Greece, October 5-7, 2016, Proceedings
%C Cham
%D 2016
%E Fiems, Dieter
%E Paolieri, Marco
%E Platis, N. Agapios
%I Springer International Publishing
%P 79--93
%T Predicting Power Consumption in Virtualized Environments
%U http://dx.doi.org/10.1007/978-3-319-46433-6_6
%X Energy efficiency and power consumption of data centers can be improved through conscientious placement of workloads on specific servers. Virtualization is commonly employed nowadays, as it allows for dynamic reallocation of work and abstraction from the concrete server hardware. The ability to predict the power consumption of workloads at different load levels is essential in this context. Prediction approaches can help to make better placement choices at run-time, as well as when purchasing new server equipment, by showing which servers are better suited for the execution of a given target workload. In existing work, power prediction for servers is limited to non-virtualized contexts or it does not take multiple load levels into account. Existing approaches also fail to leverage publicly available data on server efficiency and instead require experiments to be conducted on the target system. This makes these approaches unwieldy when making decision regarding systems that are not yet available to the decision maker. In this paper, we use the readily available data provided by the SPEC SERT to predict the power consumption of workloads for different load levels in virtualized environments. We evaluate our approach comparing predicted results against measurements of power consumption in multiple virtualized environment configurations on a target server that differs significantly from the reference system used for experimentation. We show that power consumption of CPU and storage loads can be reliably predicted with a prediction error of less than 15% across all tested virtualized environment configurations.

{Variations in CPU Power Consumption}. von Kistowski, J{ó}akim; Block, Hansfried; Beckett, John; Spradling, Cloyce; Lange, Klaus-Dieter; Kounev, Samuel; in Proceedings of the 7th ACM/SPEC International Conference on Performance Engineering (ICPE 2016) (2016). ACM, New York, NY, USA.

[ Abstract ]
[ BibTeX ]
[ EndNote ]
[ Download ]

{Experimental analysis of computer systems' power consumption has become an integral part of system performance evaluation, efficiency management, and model-based analysis. As with all measurements, repeatability and reproducibility of power measurements is a major challenge. Nominally identical systems can have different power consumption running the same workload under otherwise identical conditions. This behavior can also be observed for individual system components. Specifically, CPU power consumption can vary amongst different samples of nominally identical CPUs. This in turn has a significant impact on the overall system power, considering that a system's processor is the largest and most dynamic power consumer of the overall system. The concrete impact of CPU sample power variations is unknown, as comprehensive studies about differences in power consumption for nominally identical systems are currently missing. We address this lack of studies by conducting measurements on four different processor types from two different architectures. For each of these types, we compare up to 30 physical processor samples with a total sum of 90 samples over all processor types. We analyze the variations in power consumption for the different samples using six different workloads over five load levels. Additionally, we analyze how these variations change for different processor core counts and architectures. The results of this paper show that selection of a processor sample can have a statistically significant impact on power consumption. With no correlation to performance, power consumption for nominally identical processors can differ as much as 29.6\% in idle and 19.5\% at full load. We also show that these variations change over different architectures and processor types.}

@inproceedings{KiBlBeSpLaKo2016-ICPE-PowerVariation,
  abstract = {{Experimental analysis of computer systems' power consumption has become an integral part of system performance evaluation, efficiency management, and model-based analysis. As with all measurements, repeatability and reproducibility of power measurements is a major challenge. Nominally identical systems can have different power consumption running the same workload under otherwise identical conditions. This behavior can also be observed for individual system components. Specifically, CPU power consumption can vary amongst different samples of nominally identical CPUs. This in turn has a significant impact on the overall system power, considering that a system's processor is the largest and most dynamic power consumer of the overall system. The concrete impact of CPU sample power variations is unknown, as comprehensive studies about differences in power consumption for nominally identical systems are currently missing. We address this lack of studies by conducting measurements on four different processor types from two different architectures. For each of these types, we compare up to 30 physical processor samples with a total sum of 90 samples over all processor types. We analyze the variations in power consumption for the different samples using six different workloads over five load levels. Additionally, we analyze how these variations change for different processor core counts and architectures. The results of this paper show that selection of a processor sample can have a statistically significant impact on power consumption. With no correlation to performance, power consumption for nominally identical processors can differ as much as 29.6\% in idle and 19.5\% at full load. We also show that these variations change over different architectures and processor types.}},
  address = {New York, NY, USA},
  author = {von Kistowski, J{\'o}akim and Block, Hansfried and Beckett, John and Spradling, Cloyce and Lange, Klaus-Dieter and Kounev, Samuel},
  booktitle = {Proceedings of the 7th ACM/SPEC International Conference on Performance Engineering (ICPE 2016)},
  keywords = {Power},
  month = {03},
  publisher = {ACM},
  title = {{Variations in CPU Power Consumption}},
  year = 2016
}

%0 Conference Paper
%1 KiBlBeSpLaKo2016-ICPE-PowerVariation
%A von Kistowski, J{ó}akim
%A Block, Hansfried
%A Beckett, John
%A Spradling, Cloyce
%A Lange, Klaus-Dieter
%A Kounev, Samuel
%B Proceedings of the 7th ACM/SPEC International Conference on Performance Engineering (ICPE 2016)
%C New York, NY, USA
%D 2016
%I ACM
%T {Variations in CPU Power Consumption}
%X {Experimental analysis of computer systems' power consumption has become an integral part of system performance evaluation, efficiency management, and model-based analysis. As with all measurements, repeatability and reproducibility of power measurements is a major challenge. Nominally identical systems can have different power consumption running the same workload under otherwise identical conditions. This behavior can also be observed for individual system components. Specifically, CPU power consumption can vary amongst different samples of nominally identical CPUs. This in turn has a significant impact on the overall system power, considering that a system's processor is the largest and most dynamic power consumer of the overall system. The concrete impact of CPU sample power variations is unknown, as comprehensive studies about differences in power consumption for nominally identical systems are currently missing. We address this lack of studies by conducting measurements on four different processor types from two different architectures. For each of these types, we compare up to 30 physical processor samples with a total sum of 90 samples over all processor types. We analyze the variations in power consumption for the different samples using six different workloads over five load levels. Additionally, we analyze how these variations change for different processor core counts and architectures. The results of this paper show that selection of a processor sample can have a statistically significant impact on power consumption. With no correlation to performance, power consumption for nominally identical processors can differ as much as 29.6\% in idle and 19.5\% at full load. We also show that these variations change over different architectures and processor types.}

{Use Cases for Chauffeur}. von Kistowski, J{ó}akim; (2016, January).

[ BibTeX ]
[ EndNote ]

@misc{talk-Kistowski-SPECSymposium2016,
  author = {von Kistowski, J{\'o}akim},
  howpublished = {Invited Talk at SPEC Symposium North America 2016},
  keywords = {Power},
  month = {01},
  title = {{Use Cases for Chauffeur}},
  year = 2016
}

%0 Generic
%1 talk-Kistowski-SPECSymposium2016
%A von Kistowski, J{ó}akim
%D 2016
%T {Use Cases for Chauffeur}

{Univariate Interpolation-based Modeling of Power and Performance}. von Kistowski, J{ó}akim; Kounev, Samuel; in Proceedings of the 9th EAI International Conference on Performance Evaluation Methodologies and Tools (VALUETOOLS 2015) (2015).

[ Abstract ]
[ BibTeX ]
[ EndNote ]
[ Download ]

{Performance and power scale non-linearly with device utilization, making characterization and prediction of energy efficiency at a given load level a challenging issue. A common approach to address this problem is the creation of power or performance state tables for a pre-measured subset of all possible system states. Approaches to determine performance and power for a state not included in the measured subset use simple interpolation, such as nearest neighbor interpolation, or define state switching rules. This leads to a loss in accuracy, as unmeasured system states are not considered. In this paper, we compare different interpolation functions and automatically configure and select functions for a given domain or measurement set. We evaluate our approach by comparing interpolation of measurement data subsets against power and performance measurements on a commodity server. We show that for non-extrapolating models interpolation is significantly more accurate than regression, with our automatically configured interpolation function improving modeling accuracy up to 43.6\%.}

@inproceedings{KiKo2015-VALUETOOLS,
  abstract = {{Performance and power scale non-linearly with device utilization, making characterization and prediction of energy efficiency at a given load level a challenging issue. A common approach to address this problem is the creation of power or performance state tables for a pre-measured subset of all possible system states. Approaches to determine performance and power for a state not included in the measured subset use simple interpolation, such as nearest neighbor interpolation, or define state switching rules. This leads to a loss in accuracy, as unmeasured system states are not considered. In this paper, we compare different interpolation functions and automatically configure and select functions for a given domain or measurement set. We evaluate our approach by comparing interpolation of measurement data subsets against power and performance measurements on a commodity server. We show that for non-extrapolating models interpolation is significantly more accurate than regression, with our automatically configured interpolation function improving modeling accuracy up to 43.6\%.}},
  author = {von Kistowski, J{\'o}akim and Kounev, Samuel},
  booktitle = {Proceedings of the 9th EAI International Conference on Performance Evaluation Methodologies and Tools (VALUETOOLS 2015)},
  keywords = {Power},
  month = 12,
  title = {{Univariate Interpolation-based Modeling of Power and Performance}},
  year = 2015
}

%0 Conference Paper
%1 KiKo2015-VALUETOOLS
%A von Kistowski, J{ó}akim
%A Kounev, Samuel
%B Proceedings of the 9th EAI International Conference on Performance Evaluation Methodologies and Tools (VALUETOOLS 2015)
%D 2015
%T {Univariate Interpolation-based Modeling of Power and Performance}
%X {Performance and power scale non-linearly with device utilization, making characterization and prediction of energy efficiency at a given load level a challenging issue. A common approach to address this problem is the creation of power or performance state tables for a pre-measured subset of all possible system states. Approaches to determine performance and power for a state not included in the measured subset use simple interpolation, such as nearest neighbor interpolation, or define state switching rules. This leads to a loss in accuracy, as unmeasured system states are not considered. In this paper, we compare different interpolation functions and automatically configure and select functions for a given domain or measurement set. We evaluate our approach by comparing interpolation of measurement data subsets against power and performance measurements on a commodity server. We show that for non-extrapolating models interpolation is significantly more accurate than regression, with our automatically configured interpolation function improving modeling accuracy up to 43.6\%.}

{Common Errors and Assumptions in Energy Measurement and Management}. von Kistowski, J{ó}akim; (2015, November).

[ BibTeX ]
[ EndNote ]
[ Download ]

@misc{talk-Kistowski-SSP15,
  author = {von Kistowski, J{\'o}akim},
  howpublished = {Technical Talk at Symposium on Software Performance (SSP) 2015},
  keywords = {Power},
  month = 11,
  title = {{Common Errors and Assumptions in Energy Measurement and Management}},
  year = 2015
}

%0 Generic
%1 talk-Kistowski-SSP15
%A von Kistowski, J{ó}akim
%D 2015
%T {Common Errors and Assumptions in Energy Measurement and Management}

{Energy Efficiency of Hierarchical Server Load Distribution Strategies}. von Kistowski, J{ó}akim; Beckett, John; Lange, Klaus-Dieter; Block, Hansfried; Arnold, Jeremy A.; Kounev, Samuel; in Proceedings of the IEEE 23nd International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS 2015) (2015). IEEE.

Full paper acceptance rate: 19\%

[ Abstract ]
[ BibTeX ]
[ EndNote ]
[ Download ]

{Energy efficiency of servers has become a significant issue over the last years. Load distribution plays a crucial role in the improvement of energy efficiency as (un-)balancing strategies can be leveraged to distribute load over one or multiple systems in a way in which resources are utilized at high performance, yet low overall power consumption. This can be achieved on multiple levels, from load distribution on single CPU cores to machine level load balancing on distributed systems. With modern day server architectures providing load balancing opportunities at several layers, answering the question of optimal load distribution has become non-trivial. Work has to be distributed hierarchically in a fashion that enables maximum energy efficiency at each level. Current approaches balance load based on generalized assumptions about the energy efficiency of servers. These assumptions are based either on very machine-specific or highly generalized observations that may or may not hold true over a variety of systems and configurations. In this paper, we use a modified version of the SPEC SERT suite to measure the energy efficiency of a variety of hierarchical load distribution strategies on single and multi-node systems. We introduce a new strategy and evaluate energy efficiency for homogeneous and heterogeneous workloads over different hardware configurations. Our results show that the selection of a load distribution strategy depends heavily on workload, system utilization, as well as hardware. Used in conjunction with existing strategies, our new load distribution strategy can reduce a single system's power consumption by up to 10.7%.}

@inproceedings{KiBeLaBlArKo2015-MASCOTS,
  abstract = {{Energy efficiency of servers has become a significant issue over the last years. Load distribution plays a crucial role in the improvement of energy efficiency as (un-)balancing strategies can be leveraged to distribute load over one or multiple systems in a way in which resources are utilized at high performance, yet low overall power consumption. This can be achieved on multiple levels, from load distribution on single CPU cores to machine level load balancing on distributed systems. With modern day server architectures providing load balancing opportunities at several layers, answering the question of optimal load distribution has become non-trivial. Work has to be distributed hierarchically in a fashion that enables maximum energy efficiency at each level. Current approaches balance load based on generalized assumptions about the energy efficiency of servers. These assumptions are based either on very machine-specific or highly generalized observations that may or may not hold true over a variety of systems and configurations. In this paper, we use a modified version of the SPEC SERT suite to measure the energy efficiency of a variety of hierarchical load distribution strategies on single and multi-node systems. We introduce a new strategy and evaluate energy efficiency for homogeneous and heterogeneous workloads over different hardware configurations. Our results show that the selection of a load distribution strategy depends heavily on workload, system utilization, as well as hardware. Used in conjunction with existing strategies, our new load distribution strategy can reduce a single system's power consumption by up to 10.7%.}},
  author = {von Kistowski, J{\'o}akim and Beckett, John and Lange, Klaus-Dieter and Block, Hansfried and Arnold, Jeremy A. and Kounev, Samuel},
  booktitle = {Proceedings of the IEEE 23nd International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS 2015)},
  keywords = {Power},
  month = 10,
  note = {Full paper acceptance rate: 19\%},
  publisher = {IEEE},
  title = {{Energy Efficiency of Hierarchical Server Load Distribution Strategies}},
  year = 2015
}

%0 Conference Paper
%1 KiBeLaBlArKo2015-MASCOTS
%A von Kistowski, J{ó}akim
%A Beckett, John
%A Lange, Klaus-Dieter
%A Block, Hansfried
%A Arnold, Jeremy A.
%A Kounev, Samuel
%B Proceedings of the IEEE 23nd International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS 2015)
%D 2015
%I IEEE
%T {Energy Efficiency of Hierarchical Server Load Distribution Strategies}
%X {Energy efficiency of servers has become a significant issue over the last years. Load distribution plays a crucial role in the improvement of energy efficiency as (un-)balancing strategies can be leveraged to distribute load over one or multiple systems in a way in which resources are utilized at high performance, yet low overall power consumption. This can be achieved on multiple levels, from load distribution on single CPU cores to machine level load balancing on distributed systems. With modern day server architectures providing load balancing opportunities at several layers, answering the question of optimal load distribution has become non-trivial. Work has to be distributed hierarchically in a fashion that enables maximum energy efficiency at each level. Current approaches balance load based on generalized assumptions about the energy efficiency of servers. These assumptions are based either on very machine-specific or highly generalized observations that may or may not hold true over a variety of systems and configurations. In this paper, we use a modified version of the SPEC SERT suite to measure the energy efficiency of a variety of hierarchical load distribution strategies on single and multi-node systems. We introduce a new strategy and evaluate energy efficiency for homogeneous and heterogeneous workloads over different hardware configurations. Our results show that the selection of a load distribution strategy depends heavily on workload, system utilization, as well as hardware. Used in conjunction with existing strategies, our new load distribution strategy can reduce a single system's power consumption by up to 10.7%.}

Performance-oriented {DevOps}: {A} Research Agenda. (SPEC-RG-2015-01), Brunnert, Andreas; van Hoorn, Andre; Willnecker, Felix; Danciu, Alexandru; Hasselbring, Wilhelm; Heger, Christoph; Herbst, Nikolas; Jamshidi, Pooyan; Jung, Reiner; von Kistowski, Joakim; Koziolek, Anne; Kross, Johannes; Spinner, Simon; V{ö}gele, Christian; Walter, J{ü}rgen; Wert, Alexander; (2015). SPEC Research Group --- DevOps Performance Working Group, Standard Performance Evaluation Corporation (SPEC).

[ Abstract ]
[ BibTeX ]
[ EndNote ]
[ URL ]

DevOps is a trend towards a tighter integration between development (Dev) and operations (Ops) teams. The need for such an integration is driven by the requirement to continuously adapt enterprise applications (EAs) to changes in the business environment. As of today, DevOps concepts have been primarily introduced to ensure a constant flow of features and bug fixes into new releases from a functional perspective. In order to integrate a non-functional perspective into these DevOps concepts this report focuses on tools, activities, and processes to ensure one of the most important quality attributes of a software system, namely performance. Performance describes system properties concerning its timeliness and use of resources. Common metrics are response time, throughput, and resource utilization. Performance goals for EAs are typically defined by setting upper and/or lower bounds for these metrics and specific business transactions. In order to ensure that such performance goals can be met, several activities are required during development and operation of these systems as well as during the transition from Dev to Ops. Activities during development are typically summarized by the term Software Performance Engineering (SPE), whereas activities during operations are called Application Performance Management (APM). SPE and APM were historically tackled independently from each other, but the newly emerging DevOps concepts require and enable a tighter integration between both activity streams. This report presents existing solutions to support this integration as well as open research challenges in this area. The report starts by defining EAs and summarizes their characteristics that make performance evaluations for these systems particularly challenging. It continues by describing our understanding of DevOps and explaining the roots of this trend to set the context for the remaining parts of the report. Afterwards, performance management activities that are common in both life cycle phases are explained, until the particularities of SPE and APM are discussed in separate sections. Finally, the report concludes by outlining activities and challenges to support the rapid iteration between Dev and Ops.

@techreport{SPEC-RG-2015-001-DevOpsPerformanceResearchAgenda,
  abstract = {DevOps is a trend towards a tighter integration between development (Dev) and operations (Ops) teams. The need for such an integration is driven by the requirement to continuously adapt enterprise applications (EAs) to changes in the business environment. As of today, DevOps concepts have been primarily introduced to ensure a constant flow of features and bug fixes into new releases from a functional perspective. In order to integrate a non-functional perspective into these DevOps concepts this report focuses on tools, activities, and processes to ensure one of the most important quality attributes of a software system, namely performance. Performance describes system properties concerning its timeliness and use of resources. Common metrics are response time, throughput, and resource utilization. Performance goals for EAs are typically defined by setting upper and/or lower bounds for these metrics and specific business transactions. In order to ensure that such performance goals can be met, several activities are required during development and operation of these systems as well as during the transition from Dev to Ops. Activities during development are typically summarized by the term Software Performance Engineering (SPE), whereas activities during operations are called Application Performance Management (APM). SPE and APM were historically tackled independently from each other, but the newly emerging DevOps concepts require and enable a tighter integration between both activity streams. This report presents existing solutions to support this integration as well as open research challenges in this area. The report starts by defining EAs and summarizes their characteristics that make performance evaluations for these systems particularly challenging. It continues by describing our understanding of DevOps and explaining the roots of this trend to set the context for the remaining parts of the report. Afterwards, performance management activities that are common in both life cycle phases are explained, until the particularities of SPE and APM are discussed in separate sections. Finally, the report concludes by outlining activities and challenges to support the rapid iteration between Dev and Ops.},
  author = {Brunnert, Andreas and van Hoorn, Andre and Willnecker, Felix and Danciu, Alexandru and Hasselbring, Wilhelm and Heger, Christoph and Herbst, Nikolas and Jamshidi, Pooyan and Jung, Reiner and von Kistowski, Joakim and Koziolek, Anne and Kross, Johannes and Spinner, Simon and V{\"o}gele, Christian and Walter, J{\"u}rgen and Wert, Alexander},
  institution = {SPEC Research Group — DevOps Performance Working Group, Standard Performance Evaluation Corporation (SPEC)},
  keywords = {SPEC},
  month = {08},
  number = {SPEC-RG-2015-01},
  title = {Performance-oriented {DevOps}: {A} Research Agenda},
  year = 2015
}

%0 Report
%1 SPEC-RG-2015-001-DevOpsPerformanceResearchAgenda
%A Brunnert, Andreas
%A van Hoorn, Andre
%A Willnecker, Felix
%A Danciu, Alexandru
%A Hasselbring, Wilhelm
%A Heger, Christoph
%A Herbst, Nikolas
%A Jamshidi, Pooyan
%A Jung, Reiner
%A von Kistowski, Joakim
%A Koziolek, Anne
%A Kross, Johannes
%A Spinner, Simon
%A V{ö}gele, Christian
%A Walter, J{ü}rgen
%A Wert, Alexander
%D 2015
%N SPEC-RG-2015-01
%T Performance-oriented {DevOps}: {A} Research Agenda
%U https://research.spec.org/fileadmin/user_upload/documents/wg_devops/endorsed_publications/SPEC-RG-2015-001-DevOpsPerformanceResearchAgenda.pdf
%X DevOps is a trend towards a tighter integration between development (Dev) and operations (Ops) teams. The need for such an integration is driven by the requirement to continuously adapt enterprise applications (EAs) to changes in the business environment. As of today, DevOps concepts have been primarily introduced to ensure a constant flow of features and bug fixes into new releases from a functional perspective. In order to integrate a non-functional perspective into these DevOps concepts this report focuses on tools, activities, and processes to ensure one of the most important quality attributes of a software system, namely performance. Performance describes system properties concerning its timeliness and use of resources. Common metrics are response time, throughput, and resource utilization. Performance goals for EAs are typically defined by setting upper and/or lower bounds for these metrics and specific business transactions. In order to ensure that such performance goals can be met, several activities are required during development and operation of these systems as well as during the transition from Dev to Ops. Activities during development are typically summarized by the term Software Performance Engineering (SPE), whereas activities during operations are called Application Performance Management (APM). SPE and APM were historically tackled independently from each other, but the newly emerging DevOps concepts require and enable a tighter integration between both activity streams. This report presents existing solutions to support this integration as well as open research challenges in this area. The report starts by defining EAs and summarizes their characteristics that make performance evaluations for these systems particularly challenging. It continues by describing our understanding of DevOps and explaining the roots of this trend to set the context for the remaining parts of the report. Afterwards, performance management activities that are common in both life cycle phases are explained, until the particularities of SPE and APM are discussed in separate sections. Finally, the report concludes by outlining activities and challenges to support the rapid iteration between Dev and Ops.

{Modeling and Extracting Load Intensity Profiles}. von Kistowski, J{ó}akim; Herbst, Nikolas Roman; Zoller, Daniel; Kounev, Samuel; Hotho, Andreas; in Proceedings of the 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS 2015) (2015).

Acceptance rate: 29\%

[ Abstract ]
[ BibTeX ]
[ EndNote ]
[ Download ]

Today's system developers and operators face the challenge of creating software systems that make efficient use of dynamically allocated resources under highly variable and dynamic load profiles, while at the same time delivering reliable performance. Benchmarking of systems under these constraints is difficult, as state-of-the-art benchmarking frameworks provide only limited support for emulating such dynamic and highly variable load profiles for the creation of realistic workload scenarios. Industrial benchmarks typically confine themselves to workloads with constant or stepwise increasing loads. Alternatively, they support replaying of recorded load traces. Statistical load intensity descriptions also do not sufficiently capture concrete pattern load profile variations over time. To address these issues, we present the Descartes Load Intensity Model (DLIM). DLIM provides a modeling formalism for describing load intensity variations over time. A DLIM instance can be used as a compact representation of a recorded load intensity trace, providing a powerful tool for benchmarking and performance analysis. As manually obtaining DLIM instances can be time consuming, we present three different automated extraction methods, which also help to enable autonomous system analysis for self-adaptive systems. Model expressiveness is validated using the presented extraction methods. Extracted DLIM instances exhibit a median modeling error of 12.4% on average over nine different real-world traces covering between two weeks and seven months. Additionally, extraction methods perform orders of magnitude faster than existing time series decomposition approaches.

@inproceedings{KiHeZoKoHo2015_SEAMS_DLIMExtraction,
  abstract = {Today's system developers and operators face the challenge of creating software systems that make efficient use of dynamically allocated resources under highly variable and dynamic load profiles, while at the same time delivering reliable performance. Benchmarking of systems under these constraints is difficult, as state-of-the-art benchmarking frameworks provide only limited support for emulating such dynamic and highly variable load profiles for the creation of realistic workload scenarios. Industrial benchmarks typically confine themselves to workloads with constant or stepwise increasing loads. Alternatively, they support replaying of recorded load traces. Statistical load intensity descriptions also do not sufficiently capture concrete pattern load profile variations over time. To address these issues, we present the Descartes Load Intensity Model (DLIM). DLIM provides a modeling formalism for describing load intensity variations over time. A DLIM instance can be used as a compact representation of a recorded load intensity trace, providing a powerful tool for benchmarking and performance analysis. As manually obtaining DLIM instances can be time consuming, we present three different automated extraction methods, which also help to enable autonomous system analysis for self-adaptive systems. Model expressiveness is validated using the presented extraction methods. Extracted DLIM instances exhibit a median modeling error of 12.4% on average over nine different real-world traces covering between two weeks and seven months. Additionally, extraction methods perform orders of magnitude faster than existing time series decomposition approaches.},
  author = {von Kistowski, J{\'o}akim and Herbst, Nikolas Roman and Zoller, Daniel and Kounev, Samuel and Hotho, Andreas},
  booktitle = {Proceedings of the 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS 2015)},
  keywords = {LIMBO},
  month = {05},
  note = {Acceptance rate: 29\%},
  title = {{Modeling and Extracting Load Intensity Profiles}},
  year = 2015
}

%0 Conference Paper
%1 KiHeZoKoHo2015_SEAMS_DLIMExtraction
%A von Kistowski, J{ó}akim
%A Herbst, Nikolas Roman
%A Zoller, Daniel
%A Kounev, Samuel
%A Hotho, Andreas
%B Proceedings of the 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS 2015)
%D 2015
%T {Modeling and Extracting Load Intensity Profiles}
%X Today's system developers and operators face the challenge of creating software systems that make efficient use of dynamically allocated resources under highly variable and dynamic load profiles, while at the same time delivering reliable performance. Benchmarking of systems under these constraints is difficult, as state-of-the-art benchmarking frameworks provide only limited support for emulating such dynamic and highly variable load profiles for the creation of realistic workload scenarios. Industrial benchmarks typically confine themselves to workloads with constant or stepwise increasing loads. Alternatively, they support replaying of recorded load traces. Statistical load intensity descriptions also do not sufficiently capture concrete pattern load profile variations over time. To address these issues, we present the Descartes Load Intensity Model (DLIM). DLIM provides a modeling formalism for describing load intensity variations over time. A DLIM instance can be used as a compact representation of a recorded load intensity trace, providing a powerful tool for benchmarking and performance analysis. As manually obtaining DLIM instances can be time consuming, we present three different automated extraction methods, which also help to enable autonomous system analysis for self-adaptive systems. Model expressiveness is validated using the presented extraction methods. Extracted DLIM instances exhibit a median modeling error of 12.4% on average over nine different real-world traces covering between two weeks and seven months. Additionally, extraction methods perform orders of magnitude faster than existing time series decomposition approaches.

{How to Build a Benchmark}. von Kistowski, J{ó}akim; Arnold, Jeremy A.; Huppler, Karl; Lange, Klaus-Dieter; Henning, John L.; Cao, Paul; in Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering (ICPE 2015) (2015). ACM, New York, NY, USA.

[ Abstract ]
[ BibTeX ]
[ EndNote ]
[ Download ]

{Standardized benchmarks have become widely accepted tools for the comparison of products and evaluation of methodologies. These benchmarks are created by consortia like SPEC and TPC under confidentiality agreements which provide little opportunity for outside observers to get a look at the processes and concerns that are prevalent in benchmark development. This paper introduces the primary concerns of benchmark development from the perspectives of SPEC and TPC committees. We provide a benchmark definition, outline the types of benchmarks, and explain the characteristics of a good benchmark. We focus on the characteristics important for a standardized benchmark, as created by the SPEC and TPC consortia. To this end, we specify the primary criteria to be employed for benchmark design and workload selection. We use multiple standardized benchmarks as examples to demonstrate how these criteria are ensured.}

@inproceedings{KiArHuLaHeCa2015-ICPE-Benchmark,
  abstract = {{Standardized benchmarks have become widely accepted tools for the comparison of products and evaluation of methodologies. These benchmarks are created by consortia like SPEC and TPC under confidentiality agreements which provide little opportunity for outside observers to get a look at the processes and concerns that are prevalent in benchmark development. This paper introduces the primary concerns of benchmark development from the perspectives of SPEC and TPC committees. We provide a benchmark definition, outline the types of benchmarks, and explain the characteristics of a good benchmark. We focus on the characteristics important for a standardized benchmark, as created by the SPEC and TPC consortia. To this end, we specify the primary criteria to be employed for benchmark design and workload selection. We use multiple standardized benchmarks as examples to demonstrate how these criteria are ensured.}},
  address = {New York, NY, USA},
  author = {von Kistowski, J{\'o}akim and Arnold, Jeremy A. and Huppler, Karl and Lange, Klaus-Dieter and Henning, John L. and Cao, Paul},
  booktitle = {Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering (ICPE 2015)},
  keywords = {Power},
  month = {02},
  publisher = {ACM},
  series = {ICPE '15},
  title = {{How to Build a Benchmark}},
  year = 2015
}

%0 Conference Paper
%1 KiArHuLaHeCa2015-ICPE-Benchmark
%A von Kistowski, J{ó}akim
%A Arnold, Jeremy A.
%A Huppler, Karl
%A Lange, Klaus-Dieter
%A Henning, John L.
%A Cao, Paul
%B Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering (ICPE 2015)
%C New York, NY, USA
%D 2015
%I ACM
%T {How to Build a Benchmark}
%X {Standardized benchmarks have become widely accepted tools for the comparison of products and evaluation of methodologies. These benchmarks are created by consortia like SPEC and TPC under confidentiality agreements which provide little opportunity for outside observers to get a look at the processes and concerns that are prevalent in benchmark development. This paper introduces the primary concerns of benchmark development from the perspectives of SPEC and TPC committees. We provide a benchmark definition, outline the types of benchmarks, and explain the characteristics of a good benchmark. We focus on the characteristics important for a standardized benchmark, as created by the SPEC and TPC consortia. To this end, we specify the primary criteria to be employed for benchmark design and workload selection. We use multiple standardized benchmarks as examples to demonstrate how these criteria are ensured.}

{Analysis of the Influences on Server Power Consumption and Energy Efficiency for CPU-Intensive Workloads}. von Kistowski, J{ó}akim; Block, Hansfried; Beckett, John; Lange, Klaus-Dieter; Arnold, Jeremy A.; Kounev, Samuel; in Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering (ICPE 2015) (2015). ACM, New York, NY, USA.

acceptance rate: 27\%

[ Abstract ]
[ BibTeX ]
[ EndNote ]
[ Download ]

{ Energy efficiency of servers has become a significant research topic over the last years, as server energy consumption varies depending on multiple factors, such as server utilization and workload type. Server energy analysis and estimation must take all relevant factors into account to ensure reliable estimates and conclusions. Thorough system analysis requires benchmarks capable of testing different system resources at different load levels using multiple workload types. Server energy estimation approaches, on the other hand, require knowledge about the interactions of these factors for the creation of accurate power models. Common approaches to energy-aware workload classification classify workloads depending on the resource types used by the different workloads. However, they rarely take into account differences in workloads targeting the same resources. Industrial energy-efficiency benchmarks typically do not evaluate the system's energy consumption at different resource load levels, and they only provide data for system analysis at maximum system load. In this paper, we benchmark multiple server configurations using the CPU worklets included in SPEC's Server Efficiency Rating Tool (SERT). We evaluate the impact of load levels and different CPU workloads on power consumption and energy efficiency. We analyze how functions approximating the measured power consumption differ over multiple server configurations and architectures. We show that workloads targeting the same resource can differ significantly in their power draw and energy efficiency. The power consumption of a given workload type varies depending on utilization, hardware and software configuration. The power consumption of CPU-intensive workloads does not scale uniformly with increased load, nor do hardware or software configuration changes affect it in a uniform manner.}

@inproceedings{KiBlBeLaArKo2015-ICPE-SERT,
  abstract = {{ Energy efficiency of servers has become a significant research topic over the last years, as server energy consumption varies depending on multiple factors, such as server utilization and workload type. Server energy analysis and estimation must take all relevant factors into account to ensure reliable estimates and conclusions. Thorough system analysis requires benchmarks capable of testing different system resources at different load levels using multiple workload types. Server energy estimation approaches, on the other hand, require knowledge about the interactions of these factors for the creation of accurate power models. Common approaches to energy-aware workload classification classify workloads depending on the resource types used by the different workloads. However, they rarely take into account differences in workloads targeting the same resources. Industrial energy-efficiency benchmarks typically do not evaluate the system's energy consumption at different resource load levels, and they only provide data for system analysis at maximum system load. In this paper, we benchmark multiple server configurations using the CPU worklets included in SPEC's Server Efficiency Rating Tool (SERT). We evaluate the impact of load levels and different CPU workloads on power consumption and energy efficiency. We analyze how functions approximating the measured power consumption differ over multiple server configurations and architectures. We show that workloads targeting the same resource can differ significantly in their power draw and energy efficiency. The power consumption of a given workload type varies depending on utilization, hardware and software configuration. The power consumption of CPU-intensive workloads does not scale uniformly with increased load, nor do hardware or software configuration changes affect it in a uniform manner.}},
  address = {New York, NY, USA},
  author = {von Kistowski, J{\'o}akim and Block, Hansfried and Beckett, John and Lange, Klaus-Dieter and Arnold, Jeremy A. and Kounev, Samuel},
  booktitle = {Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering (ICPE 2015)},
  keywords = {Power},
  month = {02},
  note = {acceptance rate: 27\%},
  publisher = {ACM},
  series = {ICPE '15},
  title = {{Analysis of the Influences on Server Power Consumption and Energy Efficiency for CPU-Intensive Workloads}},
  year = 2015
}

%0 Conference Paper
%1 KiBlBeLaArKo2015-ICPE-SERT
%A von Kistowski, J{ó}akim
%A Block, Hansfried
%A Beckett, John
%A Lange, Klaus-Dieter
%A Arnold, Jeremy A.
%A Kounev, Samuel
%B Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering (ICPE 2015)
%C New York, NY, USA
%D 2015
%I ACM
%T {Analysis of the Influences on Server Power Consumption and Energy Efficiency for CPU-Intensive Workloads}
%X { Energy efficiency of servers has become a significant research topic over the last years, as server energy consumption varies depending on multiple factors, such as server utilization and workload type. Server energy analysis and estimation must take all relevant factors into account to ensure reliable estimates and conclusions. Thorough system analysis requires benchmarks capable of testing different system resources at different load levels using multiple workload types. Server energy estimation approaches, on the other hand, require knowledge about the interactions of these factors for the creation of accurate power models. Common approaches to energy-aware workload classification classify workloads depending on the resource types used by the different workloads. However, they rarely take into account differences in workloads targeting the same resources. Industrial energy-efficiency benchmarks typically do not evaluate the system's energy consumption at different resource load levels, and they only provide data for system analysis at maximum system load. In this paper, we benchmark multiple server configurations using the CPU worklets included in SPEC's Server Efficiency Rating Tool (SERT). We evaluate the impact of load levels and different CPU workloads on power consumption and energy efficiency. We analyze how functions approximating the measured power consumption differ over multiple server configurations and architectures. We show that workloads targeting the same resource can differ significantly in their power draw and energy efficiency. The power consumption of a given workload type varies depending on utilization, hardware and software configuration. The power consumption of CPU-intensive workloads does not scale uniformly with increased load, nor do hardware or software configuration changes affect it in a uniform manner.}

{Using and Extending LIMBO for the Descriptive Modeling of Arrival Behaviors}. von Kistowski, J{ó}akim; Herbst, Nikolas Roman; Kounev, Samuel; in Proceedings of the Symposium on Software Performance 2014 (2014). 131–140. University of Stuttgart, Faculty of Computer Science, Electrical Engineering, and Information Technology.

Best Poster Award

[ Abstract ]
[ BibTeX ]
[ EndNote ]
[ Download ]

{LIMBO is a tool for the creation of load profiles with variable intensity over time both from scratch and from existing data. Primarily, LIMBO's intended use is the description of load arrival behaviours in open workloads. Specifically, LIMBO can be employed for the creation of custom request or user arrival time-stamps or for the re-parameterization of existing traces. LIMBO bases on the Descartes Load Intensity Model (DLIM) for the formalized description of its load intensity profiles. The DLIM formalism can be understood as a structure for piece-wise defined and combined mathematical functions. We outline DLIM and its elements and demonstrate its integration within LIMBO. LIMBO is capable of generating request or user arrival time stamps from DLIM instances. In a next step, these generated time-stamps can be used for both open workload based benchmarking and simulations. The TimestampTimer plug-in for JMeter already allows the former. LIMBO also offers a range of tools for easy load intensity modeling and analysis, including, but not limited to, a visual decomposition of load intensity time-series into seasonal and trend parts, a simplified load intensity model as part of a model creation wizard, and an automated model instance extractor. As part of our LIMBO tutorial, we explain these features in detail. We demonstrate common use cases for LIMBO and show how they are supported. We also focus on LIMBOs extensible architecture and discuss how to use LIMBO as part of another tool-chain.}

@inproceedings{KiHeKo2014-SOSP-LIMBO,
  abstract = {{LIMBO is a tool for the creation of load profiles with variable intensity over time both from scratch and from existing data. Primarily, LIMBO's intended use is the description of load arrival behaviours in open workloads. Specifically, LIMBO can be employed for the creation of custom request or user arrival time-stamps or for the re-parameterization of existing traces. LIMBO bases on the Descartes Load Intensity Model (DLIM) for the formalized description of its load intensity profiles. The DLIM formalism can be understood as a structure for piece-wise defined and combined mathematical functions. We outline DLIM and its elements and demonstrate its integration within LIMBO. LIMBO is capable of generating request or user arrival time stamps from DLIM instances. In a next step, these generated time-stamps can be used for both open workload based benchmarking and simulations. The TimestampTimer plug-in for JMeter already allows the former. LIMBO also offers a range of tools for easy load intensity modeling and analysis, including, but not limited to, a visual decomposition of load intensity time-series into seasonal and trend parts, a simplified load intensity model as part of a model creation wizard, and an automated model instance extractor. As part of our LIMBO tutorial, we explain these features in detail. We demonstrate common use cases for LIMBO and show how they are supported. We also focus on LIMBOs extensible architecture and discuss how to use LIMBO as part of another tool-chain.}},
  author = {von Kistowski, J{\'o}akim and Herbst, Nikolas Roman and Kounev, Samuel},
  booktitle = {Proceedings of the Symposium on Software Performance 2014},
  keywords = {t_poster},
  month = 11,
  note = {<b>Best Poster Award</b>},
  pages = {131–140},
  publisher = {University of Stuttgart, Faculty of Computer Science, Electrical Engineering, and Information Technology},
  title = {{Using and Extending LIMBO for the Descriptive Modeling of Arrival Behaviors}},
  year = 2014
}

%0 Conference Paper
%1 KiHeKo2014-SOSP-LIMBO
%A von Kistowski, J{ó}akim
%A Herbst, Nikolas Roman
%A Kounev, Samuel
%B Proceedings of the Symposium on Software Performance 2014
%D 2014
%I University of Stuttgart, Faculty of Computer Science, Electrical Engineering, and Information Technology
%P 131--140
%T {Using and Extending LIMBO for the Descriptive Modeling of Arrival Behaviors}
%X {LIMBO is a tool for the creation of load profiles with variable intensity over time both from scratch and from existing data. Primarily, LIMBO's intended use is the description of load arrival behaviours in open workloads. Specifically, LIMBO can be employed for the creation of custom request or user arrival time-stamps or for the re-parameterization of existing traces. LIMBO bases on the Descartes Load Intensity Model (DLIM) for the formalized description of its load intensity profiles. The DLIM formalism can be understood as a structure for piece-wise defined and combined mathematical functions. We outline DLIM and its elements and demonstrate its integration within LIMBO. LIMBO is capable of generating request or user arrival time stamps from DLIM instances. In a next step, these generated time-stamps can be used for both open workload based benchmarking and simulations. The TimestampTimer plug-in for JMeter already allows the former. LIMBO also offers a range of tools for easy load intensity modeling and analysis, including, but not limited to, a visual decomposition of load intensity time-series into seasonal and trend parts, a simplified load intensity model as part of a model creation wizard, and an automated model instance extractor. As part of our LIMBO tutorial, we explain these features in detail. We demonstrate common use cases for LIMBO and show how they are supported. We also focus on LIMBOs extensible architecture and discuss how to use LIMBO as part of another tool-chain.}

{Modeling Variations in Load Intensity over Time}. von Kistowski, J{ó}akim; Herbst, Nikolas Roman; Kounev, Samuel; in Proceedings of the 3rd International Workshop on Large-Scale Testing (LT 2014), co-located with the 5th ACM/SPEC International Conference on Performance Engineering (ICPE 2014) (2014). 1–4. ACM, New York, NY, USA.

[ Abstract ]
[ BibTeX ]
[ EndNote ]
[ URL ]
[ Download ]

{Today's software systems are expected to deliver reliable performance under highly variable load intensities while at the same time making efficient use of dynamically allocated resources. Conventional benchmarking frameworks provide limited support for emulating such highly variable and dynamic load profiles and workload scenarios. Industrial benchmarks typically use workloads with constant or stepwise increasing load intensity, or they simply replay recorded workload traces. Based on this observation, we identify the need for means allowing flexible definition of load profiles and address this by introducing two meta-models at different abstraction levels. At the lower abstraction level, the Descartes Load Intensity Meta-Model (DLIM) offers a structured and accessible way of describing the load intensity over time by editing and combining mathematical functions. The High-Level Descartes Load Intensity Meta-Model (HLDLIM) allows the description of load variations using few defined parameters that characterize the seasonal patterns, trends, bursts and noise parts. We demonstrate that both meta-models are capable of capturing real-world load profiles with acceptable accuracy through comparison with a real life trace.}

@inproceedings{KiHeKo2014-LT-DLIM,
  abstract = {{Today's software systems are expected to deliver reliable performance under highly variable load intensities while at the same time making efficient use of dynamically allocated resources. Conventional benchmarking frameworks provide limited support for emulating such highly variable and dynamic load profiles and workload scenarios. Industrial benchmarks typically use workloads with constant or stepwise increasing load intensity, or they simply replay recorded workload traces. Based on this observation, we identify the need for means allowing flexible definition of load profiles and address this by introducing two meta-models at different abstraction levels. At the lower abstraction level, the Descartes Load Intensity Meta-Model (DLIM) offers a structured and accessible way of describing the load intensity over time by editing and combining mathematical functions. The High-Level Descartes Load Intensity Meta-Model (HLDLIM) allows the description of load variations using few defined parameters that characterize the seasonal patterns, trends, bursts and noise parts. We demonstrate that both meta-models are capable of capturing real-world load profiles with acceptable accuracy through comparison with a real life trace.}},
  address = {New York, NY, USA},
  author = {von Kistowski, J{\'o}akim and Herbst, Nikolas Roman and Kounev, Samuel},
  booktitle = {Proceedings of the 3rd International Workshop on Large-Scale Testing (LT 2014), co-located with the 5th ACM/SPEC International Conference on Performance Engineering (ICPE 2014)},
  keywords = {t_workshop},
  month = {03},
  pages = {1–4},
  publisher = {ACM},
  title = {{Modeling Variations in Load Intensity over Time}},
  year = 2014
}

%0 Conference Paper
%1 KiHeKo2014-LT-DLIM
%A von Kistowski, J{ó}akim
%A Herbst, Nikolas Roman
%A Kounev, Samuel
%B Proceedings of the 3rd International Workshop on Large-Scale Testing (LT 2014), co-located with the 5th ACM/SPEC International Conference on Performance Engineering (ICPE 2014)
%C New York, NY, USA
%D 2014
%I ACM
%P 1--4
%T {Modeling Variations in Load Intensity over Time}
%U http://doi.acm.org/10.1145/2577036.2577037
%X {Today's software systems are expected to deliver reliable performance under highly variable load intensities while at the same time making efficient use of dynamically allocated resources. Conventional benchmarking frameworks provide limited support for emulating such highly variable and dynamic load profiles and workload scenarios. Industrial benchmarks typically use workloads with constant or stepwise increasing load intensity, or they simply replay recorded workload traces. Based on this observation, we identify the need for means allowing flexible definition of load profiles and address this by introducing two meta-models at different abstraction levels. At the lower abstraction level, the Descartes Load Intensity Meta-Model (DLIM) offers a structured and accessible way of describing the load intensity over time by editing and combining mathematical functions. The High-Level Descartes Load Intensity Meta-Model (HLDLIM) allows the description of load variations using few defined parameters that characterize the seasonal patterns, trends, bursts and noise parts. We demonstrate that both meta-models are capable of capturing real-world load profiles with acceptable accuracy through comparison with a real life trace.}

{LIMBO - A Load Intensity Modeling Platform}. von Kistowski, J{ó}akim; (2014, March).

[ BibTeX ]
[ EndNote ]

@misc{talk-Kistowski-SPECRG2014,
  author = {von Kistowski, J{\'o}akim},
  howpublished = {Talk at SPEC RG Annual Meeting 2014},
  keywords = {LIMBO},
  month = {03},
  title = {{LIMBO - A Load Intensity Modeling Platform}},
  year = 2014
}

%0 Generic
%1 talk-Kistowski-SPECRG2014
%A von Kistowski, J{ó}akim
%D 2014
%T {LIMBO - A Load Intensity Modeling Platform}

{Modeling Variations in Load Intensity Profiles}. von Kistowski, J{ó}akim; (2014, March). {Karlsruhe Institute of Technology (KIT)}, Am Fasanengarten 5, 76131 Karlsruhe, Germany.

Gesellschaft zur F{ö}rderung des Forschungstransfers (GFFT) Prize "Best Master Thesis"

[ Abstract ]
[ BibTeX ]
[ EndNote ]
[ Download ]

{Today's software systems are expected to deliver reliable performance under highly variable load intensities while at the same time making efficient use of dynamically allocated resources. Conventional benchmarking frameworks provide limited support for emulating such highly variable and dynamic load profiles and workload scenarios. Industrial benchmarks typically use workloads with constant or stepwise increasing load intensity, or they simply replay recorded workload traces. Based on this observation, I identify the need for means allowing flexible definition of load profiles and address this by introducing two meta-models at different abstraction levels. At the lower abstraction level, the Descartes Load Intensity Meta-Model (DLIM) offers a structured and accessible way of describing the load intensity over time by editing and combining mathematical functions. The high-level Descartes Load Intensity Meta-Model (hl-DLIM) allows the description of load variations using few defined parameters that characterize the seasonal patterns, trends, bursts, and noise parts. Using these parameters, hl-DLIM is capable of describing a subset of most common load intensity variations. During the work on this thesis I developed LIMBO - an Eclipse-based tool for modeling variable load intensity profiles based on DLIM and hl-DLIM as underlying modeling formalisms. LIMBO provides visualization for DLIM instances and a model creation wizard based on hl-DLIM parameters. It also offers three automated model extraction processes with which to extract DLIM and hl-DLIM instances from existing arrival rate traces. It also offers a model-to-model transformation from hl-DLIM to DLIM. I demonstrate that both meta-models are capable of capturing real-world load profiles with acceptable accuracy, having an average median deviation of 19.9\% from the original trace. This is done by comparing nine different real life traces, which were measured over duration of two weeks to seven months, to their respective model instances as extracted by the automated model extraction processes. I also evaluate the usability and accessibility of LIMBO based on a questionnaire which was answered by eight computer scientists from five different organizations within the performance engineering community.}

@mastersthesis{Kistowski2014,
  abstract = {{Today's software systems are expected to deliver reliable performance under highly variable load intensities while at the same time making efficient use of dynamically allocated resources. Conventional benchmarking frameworks provide limited support for emulating such highly variable and dynamic load profiles and workload scenarios. Industrial benchmarks typically use workloads with constant or stepwise increasing load intensity, or they simply replay recorded workload traces. Based on this observation, I identify the need for means allowing flexible definition of load profiles and address this by introducing two meta-models at different abstraction levels. At the lower abstraction level, the Descartes Load Intensity Meta-Model (DLIM) offers a structured and accessible way of describing the load intensity over time by editing and combining mathematical functions. The high-level Descartes Load Intensity Meta-Model (hl-DLIM) allows the description of load variations using few defined parameters that characterize the seasonal patterns, trends, bursts, and noise parts. Using these parameters, hl-DLIM is capable of describing a subset of most common load intensity variations. During the work on this thesis I developed LIMBO - an Eclipse-based tool for modeling variable load intensity profiles based on DLIM and hl-DLIM as underlying modeling formalisms. LIMBO provides visualization for DLIM instances and a model creation wizard based on hl-DLIM parameters. It also offers three automated model extraction processes with which to extract DLIM and hl-DLIM instances from existing arrival rate traces. It also offers a model-to-model transformation from hl-DLIM to DLIM. I demonstrate that both meta-models are capable of capturing real-world load profiles with acceptable accuracy, having an average median deviation of 19.9\% from the original trace. This is done by comparing nine different real life traces, which were measured over duration of two weeks to seven months, to their respective model instances as extracted by the automated model extraction processes. I also evaluate the usability and accessibility of LIMBO based on a questionnaire which was answered by eight computer scientists from five different organizations within the performance engineering community.}},
  address = {Am Fasanengarten 5, 76131 Karlsruhe, Germany},
  author = {von Kistowski, J{\'o}akim},
  keywords = {LIMBO},
  month = {03},
  note = {<b>Gesellschaft zur F{\"o}rderung des Forschungstransfers (GFFT) Prize "Best Master Thesis"</b>},
  school = {{Karlsruhe Institute of Technology (KIT)}},
  title = {{Modeling Variations in Load Intensity Profiles}},
  type = {{Master Thesis}},
  year = 2014
}

%0 Thesis
%1 Kistowski2014
%A von Kistowski, J{ó}akim
%C Am Fasanengarten 5, 76131 Karlsruhe, Germany
%D 2014
%T {Modeling Variations in Load Intensity Profiles}
%X {Today's software systems are expected to deliver reliable performance under highly variable load intensities while at the same time making efficient use of dynamically allocated resources. Conventional benchmarking frameworks provide limited support for emulating such highly variable and dynamic load profiles and workload scenarios. Industrial benchmarks typically use workloads with constant or stepwise increasing load intensity, or they simply replay recorded workload traces. Based on this observation, I identify the need for means allowing flexible definition of load profiles and address this by introducing two meta-models at different abstraction levels. At the lower abstraction level, the Descartes Load Intensity Meta-Model (DLIM) offers a structured and accessible way of describing the load intensity over time by editing and combining mathematical functions. The high-level Descartes Load Intensity Meta-Model (hl-DLIM) allows the description of load variations using few defined parameters that characterize the seasonal patterns, trends, bursts, and noise parts. Using these parameters, hl-DLIM is capable of describing a subset of most common load intensity variations. During the work on this thesis I developed LIMBO - an Eclipse-based tool for modeling variable load intensity profiles based on DLIM and hl-DLIM as underlying modeling formalisms. LIMBO provides visualization for DLIM instances and a model creation wizard based on hl-DLIM parameters. It also offers three automated model extraction processes with which to extract DLIM and hl-DLIM instances from existing arrival rate traces. It also offers a model-to-model transformation from hl-DLIM to DLIM. I demonstrate that both meta-models are capable of capturing real-world load profiles with acceptable accuracy, having an average median deviation of 19.9\% from the original trace. This is done by comparing nine different real life traces, which were measured over duration of two weeks to seven months, to their respective model instances as extracted by the automated model extraction processes. I also evaluate the usability and accessibility of LIMBO based on a questionnaire which was answered by eight computer scientists from five different organizations within the performance engineering community.}

{LIMBO: A Tool For Modeling Variable Load Intensities}. von Kistowski, J{ó}akim; Herbst, Nikolas Roman; Kounev, Samuel; in Proceedings of the 5th ACM/SPEC International Conference on Performance Engineering (ICPE 2014) (2014). 225–226. ACM, New York, NY, USA.

[ Abstract ]
[ BibTeX ]
[ EndNote ]
[ URL ]
[ Download ]

{Modern software systems are expected to deliver reliable performance under highly variable load intensities while at the same time making efficient use of dynamically allocated resources. Conventional benchmarking frameworks provide limited support for emulating such highly variable and dynamic load profiles and workload scenarios. Industrial benchmarks typically use workloads with constant or stepwise increasing load intensity, or they simply replay recorded workload traces. In this paper, we present LIMBO - an Eclipse-based tool for modeling variable load intensity profiles based on the Descartes Load Intensity Model as an underlying modeling formalism.}

@inproceedings{KiHeKo2014-ICPEDemo-LIMBO,
  abstract = {{Modern software systems are expected to deliver reliable performance under highly variable load intensities while at the same time making efficient use of dynamically allocated resources. Conventional benchmarking frameworks provide limited support for emulating such highly variable and dynamic load profiles and workload scenarios. Industrial benchmarks typically use workloads with constant or stepwise increasing load intensity, or they simply replay recorded workload traces. In this paper, we present LIMBO - an Eclipse-based tool for modeling variable load intensity profiles based on the Descartes Load Intensity Model as an underlying modeling formalism.}},
  address = {New York, NY, USA},
  author = {von Kistowski, J{\'o}akim and Herbst, Nikolas Roman and Kounev, Samuel},
  booktitle = {Proceedings of the 5th ACM/SPEC International Conference on Performance Engineering (ICPE 2014)},
  keywords = {LIMBO},
  month = {03},
  pages = {225–226},
  publisher = {ACM},
  series = {ICPE '14},
  title = {{LIMBO: A Tool For Modeling Variable Load Intensities}},
  year = 2014
}

%0 Conference Paper
%1 KiHeKo2014-ICPEDemo-LIMBO
%A von Kistowski, J{ó}akim
%A Herbst, Nikolas Roman
%A Kounev, Samuel
%B Proceedings of the 5th ACM/SPEC International Conference on Performance Engineering (ICPE 2014)
%C New York, NY, USA
%D 2014
%I ACM
%P 225--226
%T {LIMBO: A Tool For Modeling Variable Load Intensities}
%U http://doi.acm.org/10.1145/2568088.2576092
%X {Modern software systems are expected to deliver reliable performance under highly variable load intensities while at the same time making efficient use of dynamically allocated resources. Conventional benchmarking frameworks provide limited support for emulating such highly variable and dynamic load profiles and workload scenarios. Industrial benchmarks typically use workloads with constant or stepwise increasing load intensity, or they simply replay recorded workload traces. In this paper, we present LIMBO - an Eclipse-based tool for modeling variable load intensity profiles based on the Descartes Load Intensity Model as an underlying modeling formalism.}

{Defining and Measuring Workloads for Elasticity Benchmarking}. von Kistowski, J{ó}akim; (2011). {Karlsruhe Institute of Technology (KIT)}, Am Fasanengarten 5, 76131 Karlsruhe, Germany.

[ BibTeX ]
[ EndNote ]

@mastersthesis{Kistowski2011a,
  address = {Am Fasanengarten 5, 76131 Karlsruhe, Germany},
  author = {von Kistowski, J{\'o}akim},
  keywords = {Elasticity},
  school = {{Karlsruhe Institute of Technology (KIT)}},
  title = {{Defining and Measuring Workloads for Elasticity Benchmarking}},
  type = {{Bachelor Thesis}},
  year = 2011
}

%0 Thesis
%1 Kistowski2011a
%A von Kistowski, J{ó}akim
%C Am Fasanengarten 5, 76131 Karlsruhe, Germany
%D 2011
%T {Defining and Measuring Workloads for Elasticity Benchmarking}

{Defining and Quantifying Elasticity of Resources in Cloud Computing and Scalable Platforms} Kuperberg, Michael; Herbst, Nikolas Roman; von Kistowski, J{ó}akim Gunnarsson; Reussner, Ralf; (2011). (Vol. 16) Karlsruhe Institute of Technology (KIT), Am Fasanengarten 5, 76131 Karlsruhe, Germany.

[ Abstract ]
[ BibTeX ]
[ EndNote ]
[ URL ]
[ Download ]

{Elasticity is the ability of a software system to dynamically scale the amount of the resources it provides to clients as their workloads increase or decrease. Elasticity is praised as a key advantage of cloud computing, where computing resources are dynamically added and released. However, there exists no concise or formal definition of elasticity, and thus no approaches to quantify it have been developed so far. Existing work on cloud computing is limited to the technical view of implementing elastic systems, and definitions or scalability have not been extended to cover elasticity. In this report, we present a detailed discussion of elasticity, propose techniques for quantifying and measuring it, and outline next steps to be taken for enabling comparisons between cloud computing offerings on the basis of elasticity. We also present preliminary work on measuring elasticity of resource pools provided by the Java Virtual Machine.}

@techreport{KuHeKiRe2011-ResourceElasticity,
  abstract = {{Elasticity is the ability of a software system to dynamically scale the amount of the resources it provides to clients as their workloads increase or decrease. Elasticity is praised as a key advantage of cloud computing, where computing resources are dynamically added and released. However, there exists no concise or formal definition of elasticity, and thus no approaches to quantify it have been developed so far. Existing work on cloud computing is limited to the technical view of implementing elastic systems, and definitions or scalability have not been extended to cover elasticity. In this report, we present a detailed discussion of elasticity, propose techniques for quantifying and measuring it, and outline next steps to be taken for enabling comparisons between cloud computing offerings on the basis of elasticity. We also present preliminary work on measuring elasticity of resource pools provided by the Java Virtual Machine.}},
  address = {Am Fasanengarten 5, 76131 Karlsruhe, Germany},
  author = {Kuperberg, Michael and Herbst, Nikolas Roman and von Kistowski, J{\'o}akim Gunnarsson and Reussner, Ralf},
  institution = {Karlsruhe Institute of Technology (KIT)},
  keywords = {t_techreport},
  series = {Karlsruhe Reports in Informatics},
  title = {{Defining and Quantifying Elasticity of Resources in Cloud Computing and Scalable Platforms}},
  volume = 16,
  year = 2011
}

%0 Report
%1 KuHeKiRe2011-ResourceElasticity
%A Kuperberg, Michael
%A Herbst, Nikolas Roman
%A von Kistowski, J{ó}akim Gunnarsson
%A Reussner, Ralf
%B Karlsruhe Reports in Informatics
%C Am Fasanengarten 5, 76131 Karlsruhe, Germany
%D 2011
%T {Defining and Quantifying Elasticity of Resources in Cloud Computing and Scalable Platforms}
%U http://digbib.ubka.uni-karlsruhe.de/volltexte/1000023476
%V 16
%X {Elasticity is the ability of a software system to dynamically scale the amount of the resources it provides to clients as their workloads increase or decrease. Elasticity is praised as a key advantage of cloud computing, where computing resources are dynamically added and released. However, there exists no concise or formal definition of elasticity, and thus no approaches to quantify it have been developed so far. Existing work on cloud computing is limited to the technical view of implementing elastic systems, and definitions or scalability have not been extended to cover elasticity. In this report, we present a detailed discussion of elasticity, propose techniques for quantifying and measuring it, and outline next steps to be taken for enabling comparisons between cloud computing offerings on the basis of elasticity. We also present preliminary work on measuring elasticity of resource pools provided by the Java Virtual Machine.}

Publications of Jóakim v. Kistowski

Publications

Hinweis zum Datenschutz

Hinweis zum Datenschutz

Bildnachweise