Publications

• [ 2020 ]
• [ 2019 ]
• [ 2018 ]
• [ 2017 ]
• [ 2016 ]
• [ 2012 ]

• 
  {Energy Efficiency Analysis of Compiler Optimizations on the SPEC CPU 2017 Benchmark Suite}. Schmitt, Norbert; Bucek, James; Lange, Klaus-Dieter; Kounev, Samuel; in Proceedings of the 11th ACM/SPEC International Conference on Performance Engineering (ICPE 2020)
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  @inproceedings{ScLaBuKo-ICPE20-SPEC-CPU2017, address = {New York, NY, USA}, author = {Schmitt, Norbert and Bucek, James and Lange, Klaus-Dieter and Kounev, Samuel}, booktitle = {Proceedings of the 11th ACM/SPEC International Conference on Performance Engineering (ICPE 2020)}, month = {April}, publisher = {ACM}, title = {{Energy Efficiency Analysis of Compiler Optimizations on the SPEC CPU 2017 Benchmark Suite}}, year = 2020 }

  %0 Conference Paper %1 ScLaBuKo-ICPE20-SPEC-CPU2017 %A Schmitt, Norbert %A Bucek, James %A Lange, Klaus-Dieter %A Kounev, Samuel %B Proceedings of the 11th ACM/SPEC International Conference on Performance Engineering (ICPE 2020) %C New York, NY, USA %D 2020 %I ACM %T {Energy Efficiency Analysis of Compiler Optimizations on the SPEC CPU 2017 Benchmark Suite} %U https://doi.org/10.1145/3375555.3383759
• 
  {An Automated Forecasting Framework based on Method Recommendation for Seasonal Time Series}. Bauer, Andr{\’e}; Z{\"u}fle, Marwin; Grohmann, Johannes; Schmitt, Norbert; Herbst, Nikolas; Kounev, Samuel; in Proceedings of the ACM/SPEC International Conference on Performance Engineering 48–55.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Due to the fast-paced and changing demands of their users, computing systems require autonomic resource management. To enable proactive and accurate decision-making for changes causing a particular overhead, reliable forecasts are needed. In fact, choosing the best performing forecasting method for a given time series scenario is a crucial task. Taking the "No-Free-Lunch Theorem" into account, there exists no forecasting method that performs best on all types of time series. To this end, we propose an automated approach that (i) extracts characteristics from a given time series, (ii) selects the best-suited machine learning method based on recommendation, and finally, (iii) performs the forecast. Our approach offers the benefit of not relying on a single method with its possibly inaccurate forecasts. In an extensive evaluation, our approach achieves the best forecasting accuracy.

  @inproceedings{BaZuGrScHeKo-ICPE20-Seasonal-Forecast, abstract = {Due to the fast-paced and changing demands of their users, computing systems require autonomic resource management. To enable proactive and accurate decision-making for changes causing a particular overhead, reliable forecasts are needed. In fact, choosing the best performing forecasting method for a given time series scenario is a crucial task. Taking the "No-Free-Lunch Theorem" into account, there exists no forecasting method that performs best on all types of time series. To this end, we propose an automated approach that (i) extracts characteristics from a given time series, (ii) selects the best-suited machine learning method based on recommendation, and finally, (iii) performs the forecast. Our approach offers the benefit of not relying on a single method with its possibly inaccurate forecasts. In an extensive evaluation, our approach achieves the best forecasting accuracy.}, address = {New York, NY, USA}, author = {Bauer, Andr{\'e} and Z{\"u}fle, Marwin and Grohmann, Johannes and Schmitt, Norbert and Herbst, Nikolas and Kounev, Samuel}, booktitle = {Proceedings of the ACM/SPEC International Conference on Performance Engineering}, month = {April}, pages = {48--55}, publisher = {Association for Computing Machinery (ACM)}, series = {ICPE '20}, title = {{An Automated Forecasting Framework based on Method Recommendation for Seasonal Time Series}}, year = 2020 }

  %0 Conference Paper %1 BaZuGrScHeKo-ICPE20-Seasonal-Forecast %A Bauer, Andr{\'e} %A Z{\"u}fle, Marwin %A Grohmann, Johannes %A Schmitt, Norbert %A Herbst, Nikolas %A Kounev, Samuel %B Proceedings of the ACM/SPEC International Conference on Performance Engineering %C New York, NY, USA %D 2020 %I Association for Computing Machinery (ACM) %P 48--55 %T {An Automated Forecasting Framework based on Method Recommendation for Seasonal Time Series} %U https://doi.org/10.1145/3358960.3379123 %X Due to the fast-paced and changing demands of their users, computing systems require autonomic resource management. To enable proactive and accurate decision-making for changes causing a particular overhead, reliable forecasts are needed. In fact, choosing the best performing forecasting method for a given time series scenario is a crucial task. Taking the "No-Free-Lunch Theorem" into account, there exists no forecasting method that performs best on all types of time series. To this end, we propose an automated approach that (i) extracts characteristics from a given time series, (ii) selects the best-suited machine learning method based on recommendation, and finally, (iii) performs the forecast. Our approach offers the benefit of not relying on a single method with its possibly inaccurate forecasts. In an extensive evaluation, our approach achieves the best forecasting accuracy.

• 
  The SPEC-RG Reference Architecture for FaaS: From Microservices and Containers to Serverless Platforms. van Eyk, Erwin; Grohmann, Johannes; Eismann, Simon; Bauer, Andr{\’e}; Versluis, Laurens; Toader, Lucian; Schmitt, Norbert; Herbst, Nikolas; Abad, Cristina L.; Iosup, Alexandru; in IEEE Internet Computing 23(6) 7–18.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Microservices, containers, and serverless computing belong to a trend toward applications composed of many small, self-contained, and automatically managed components. Core to serverless computing, Function-as-a-Service (FaaS) platforms employ state-of-the-art container technology and microservices-based architectures to enable users to manage complex applications without the need for systems-level expertise. Victim of its own success, and partially due to proprietary technology, currently the community has a limited overview of these platforms. To address this, we propose a reference architecture and ecosystem for FaaS platforms. Based on a year-long survey of real-world platforms conducted within the SPEC-RG Cloud Group, we highlight specific components and identify common operational patterns.

  @article{vEGrEiBaVeToScHeAbIo-IC-FaaS, abstract = {Microservices, containers, and serverless computing belong to a trend toward applications composed of many small, self-contained, and automatically managed components. Core to serverless computing, Function-as-a-Service (FaaS) platforms employ state-of-the-art container technology and microservices-based architectures to enable users to manage complex applications without the need for systems-level expertise. Victim of its own success, and partially due to proprietary technology, currently the community has a limited overview of these platforms. To address this, we propose a reference architecture and ecosystem for FaaS platforms. Based on a year-long survey of real-world platforms conducted within the SPEC-RG Cloud Group, we highlight specific components and identify common operational patterns.}, author = {van Eyk, Erwin and Grohmann, Johannes and Eismann, Simon and Bauer, Andr{\'e} and Versluis, Laurens and Toader, Lucian and Schmitt, Norbert and Herbst, Nikolas and Abad, Cristina L. and Iosup, Alexandru}, journal = {IEEE Internet Computing}, month = {nov}, number = 6, pages = {7--18}, publisher = {IEEE}, title = {The SPEC-RG Reference Architecture for FaaS: From Microservices and Containers to Serverless Platforms}, volume = 23, year = 2019 }

  %0 Journal Article %1 vEGrEiBaVeToScHeAbIo-IC-FaaS %A van Eyk, Erwin %A Grohmann, Johannes %A Eismann, Simon %A Bauer, Andr{\'e} %A Versluis, Laurens %A Toader, Lucian %A Schmitt, Norbert %A Herbst, Nikolas %A Abad, Cristina L. %A Iosup, Alexandru %D 2019 %I IEEE %J IEEE Internet Computing %N 6 %P 7--18 %T The SPEC-RG Reference Architecture for FaaS: From Microservices and Containers to Serverless Platforms %U https://doi.org/10.1109/MIC.2019.2952061 %V 23 %X Microservices, containers, and serverless computing belong to a trend toward applications composed of many small, self-contained, and automatically managed components. Core to serverless computing, Function-as-a-Service (FaaS) platforms employ state-of-the-art container technology and microservices-based architectures to enable users to manage complex applications without the need for systems-level expertise. Victim of its own success, and partially due to proprietary technology, currently the community has a limited overview of these platforms. To address this, we propose a reference architecture and ecosystem for FaaS platforms. Based on a year-long survey of real-world platforms conducted within the SPEC-RG Cloud Group, we highlight specific components and identify common operational patterns.
• 
  {Towards Edge Benchmarking: A Methodology for Characterizing Edge Workloads}. Tocze, Klervie; Schmitt, Norbert; Brandic, Ivona; Aral, Atakan; Nadjm-Tehrani, Simin; in Proceedings of Workshop on Hot Topics in Cloud Computing Performance (HotCloudPerf) as part of FAS*(IEEE ICAC/SASO) conferences companion
  • [ BibTeX ]
  • [ EndNote ]
  @inproceedings{ToScBrArNa2019TowardsEdgeBench, author = {Tocze, Klervie and Schmitt, Norbert and Brandic, Ivona and Aral, Atakan and Nadjm-Tehrani, Simin}, booktitle = {Proceedings of Workshop on Hot Topics in Cloud Computing Performance (HotCloudPerf) as part of FAS*(IEEE ICAC/SASO) conferences companion}, month = {June}, publisher = {IEEE}, title = {{Towards Edge Benchmarking: A Methodology for Characterizing Edge Workloads}}, year = 2019 }

  %0 Conference Paper %1 ToScBrArNa2019TowardsEdgeBench %A Tocze, Klervie %A Schmitt, Norbert %A Brandic, Ivona %A Aral, Atakan %A Nadjm-Tehrani, Simin %B Proceedings of Workshop on Hot Topics in Cloud Computing Performance (HotCloudPerf) as part of FAS*(IEEE ICAC/SASO) conferences companion %D 2019 %I IEEE %T {Towards Edge Benchmarking: A Methodology for Characterizing Edge Workloads}
• 
  {Predicting Server Power Consumption from Standard Rating Results}. von Kistowski, J{\’o}akim; Grohmann, Johannes; Schmitt, Norbert; Kounev, Samuel; in Proceedings of the 19th ACM/SPEC International Conference on Performance Engineering 301–312.

  {Full Paper Acceptance Rate: 18.6\% (13/70)}
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Data center providers and server operators try to reduce the power consumption of their servers. Finding an energy efficient server for a specific target application is a first step in this regard. Estimating the power consumption of an application on an unavailable server is difficult, as nameplate power values are generally overestimations. Offline power models are able to predict the consumption accurately, but are usually intended for system design, requiring very specific and detailed knowledge about the system under consideration. In this paper, we introduce an offline power prediction method that uses the results of standard power rating tools. The method predicts the power consumption of a specific application for multiple load levels on a target server that is otherwise unavailable for testing. We evaluate our approach by predicting the power consumption of three applications on different physical servers. Our method is able to achieve an average prediction error of 9.49% for three workloads running on real-world, physical servers.

  @inproceedings{KiGrScKo2019-ICPE-PowerPrediction, abstract = {Data center providers and server operators try to reduce the power consumption of their servers. Finding an energy efficient server for a specific target application is a first step in this regard. Estimating the power consumption of an application on an unavailable server is difficult, as nameplate power values are generally overestimations. Offline power models are able to predict the consumption accurately, but are usually intended for system design, requiring very specific and detailed knowledge about the system under consideration. In this paper, we introduce an offline power prediction method that uses the results of standard power rating tools. The method predicts the power consumption of a specific application for multiple load levels on a target server that is otherwise unavailable for testing. We evaluate our approach by predicting the power consumption of three applications on different physical servers. Our method is able to achieve an average prediction error of 9.49% for three workloads running on real-world, physical servers.}, address = {New York, NY, USA}, author = {von Kistowski, J{\'o}akim and Grohmann, Johannes and Schmitt, Norbert and Kounev, Samuel}, booktitle = {Proceedings of the 19th ACM/SPEC International Conference on Performance Engineering}, note = {{Full Paper Acceptance Rate: 18.6\% (13/70)}}, pages = {301--312}, publisher = {Association for Computing Machinery (ACM)}, series = {ICPE '19}, title = {{Predicting Server Power Consumption from Standard Rating Results}}, year = 2019 }

  %0 Conference Paper %1 KiGrScKo2019-ICPE-PowerPrediction %A von Kistowski, J{\'o}akim %A Grohmann, Johannes %A Schmitt, Norbert %A Kounev, Samuel %B Proceedings of the 19th ACM/SPEC International Conference on Performance Engineering %C New York, NY, USA %D 2019 %I Association for Computing Machinery (ACM) %P 301--312 %T {Predicting Server Power Consumption from Standard Rating Results} %U https://doi.org/10.1145/3297663.3310298 %X Data center providers and server operators try to reduce the power consumption of their servers. Finding an energy efficient server for a specific target application is a first step in this regard. Estimating the power consumption of an application on an unavailable server is difficult, as nameplate power values are generally overestimations. Offline power models are able to predict the consumption accurately, but are usually intended for system design, requiring very specific and detailed knowledge about the system under consideration. In this paper, we introduce an offline power prediction method that uses the results of standard power rating tools. The method predicts the power consumption of a specific application for multiple load levels on a target server that is otherwise unavailable for testing. We evaluate our approach by predicting the power consumption of three applications on different physical servers. Our method is able to achieve an average prediction error of 9.49% for three workloads running on real-world, physical servers.
• 
  {Improving the Energy Efficiency of IoT-Systems and its Software}. Schmitt, Norbert; in {Organic Computing: Doctoral Dissertation Colloquium 2018}, S. Tomforde, B. Sick (Hrsg.)
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @incollection{Sc2015-EEIoTSoftware, author = {Schmitt, Norbert}, booktitle = {{Organic Computing: Doctoral Dissertation Colloquium 2018}}, editor = {Tomforde, Sven and Sick, Bernhard}, month = {April}, publisher = {kassel university press GmbH}, title = {{Improving the Energy Efficiency of IoT-Systems and its Software}}, year = 2019 }

  %0 Book Section %1 Sc2015-EEIoTSoftware %A Schmitt, Norbert %B {Organic Computing: Doctoral Dissertation Colloquium 2018} %D 2019 %E Tomforde, Sven %E Sick, Bernhard %I kassel university press GmbH %T {Improving the Energy Efficiency of IoT-Systems and its Software}
• 
  Online Power Consumption Estimation for Functions in Cloud Applications. Schmitt, Norbert; Iffländer, Lukas; Bauer, André; Kounev, Samuel; in Proceedings of the 16th IEEE International Conference on Autonomic Computing (ICAC)
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @inproceedings{ScIfBaKo2019-ICAC-OnlinePowerEstimation, author = {Schmitt, Norbert and Iffländer, Lukas and Bauer, André and Kounev, Samuel}, booktitle = {Proceedings of the 16th IEEE International Conference on Autonomic Computing (ICAC)}, month = {June}, publisher = {IEEE}, title = {Online Power Consumption Estimation for Functions in Cloud Applications}, year = 2019 }

  %0 Conference Paper %1 ScIfBaKo2019-ICAC-OnlinePowerEstimation %A Schmitt, Norbert %A Iffländer, Lukas %A Bauer, André %A Kounev, Samuel %B Proceedings of the 16th IEEE International Conference on Autonomic Computing (ICAC) %D 2019 %I IEEE %T Online Power Consumption Estimation for Functions in Cloud Applications

• 
  TeaStore: A Micro-Service Reference Application for Cloud Researchers. Eismann, Simon; v. Kistowski, J{\’o}akim; Grohmann, Johannes; Bauer, Andr{\’e}; Schmitt, Norbert; Herbst, Nikolas; Kounev, Samuel; in Proceedings of 2018 IEEE/ACM International Conference on Utility and Cloud Computing Companion (UCC Companion) 11–12.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Researchers propose and employ various methods to analyze, model, optimize and manage modern distributed cloud applications. In order to demonstrate and evaluate these methods in realistic scenarios, researchers rely on reference applications. These applications should offer a range of different behaviors, degrees of freedom allowing for customization and should use a modern and representative technology stack. Existing testing and benchmarking applications are either outdated, designed for specific testing scenarios, or do not offer the necessary degrees of freedom. Further, most cloud reference applications are difficult to deploy and run. In this paper, we present the TeaStore, a micro-service-based test and reference cloud application. TeaStore offers services with various performance characteristics and a high degree of freedom regarding its deployment and configuration to be used as a cloud reference application for researchers. The TeaStore is designed for the evaluation of performance modeling and resource management techniques. We invite cloud researchers to use the TeaStore and provide it open-source, extendable, easily deployable and monitorable.

  @inproceedings{EiKiGrBaScHeKo2018-UCC-TeastorePoster, abstract = {Researchers propose and employ various methods to analyze, model, optimize and manage modern distributed cloud applications. In order to demonstrate and evaluate these methods in realistic scenarios, researchers rely on reference applications. These applications should offer a range of different behaviors, degrees of freedom allowing for customization and should use a modern and representative technology stack. Existing testing and benchmarking applications are either outdated, designed for specific testing scenarios, or do not offer the necessary degrees of freedom. Further, most cloud reference applications are difficult to deploy and run. In this paper, we present the TeaStore, a micro-service-based test and reference cloud application. TeaStore offers services with various performance characteristics and a high degree of freedom regarding its deployment and configuration to be used as a cloud reference application for researchers. The TeaStore is designed for the evaluation of performance modeling and resource management techniques. We invite cloud researchers to use the TeaStore and provide it open-source, extendable, easily deployable and monitorable.}, author = {Eismann, Simon and v. Kistowski, J{\'o}akim and Grohmann, Johannes and Bauer, Andr{\'e} and Schmitt, Norbert and Herbst, Nikolas and Kounev, Samuel}, booktitle = {Proceedings of 2018 IEEE/ACM International Conference on Utility and Cloud Computing Companion (UCC Companion)}, month = {dec}, pages = {11--12}, publisher = {IEEE}, title = {TeaStore: A Micro-Service Reference Application for Cloud Researchers}, year = 2018 }

  %0 Conference Paper %1 EiKiGrBaScHeKo2018-UCC-TeastorePoster %A Eismann, Simon %A v. Kistowski, J{\'o}akim %A Grohmann, Johannes %A Bauer, Andr{\'e} %A Schmitt, Norbert %A Herbst, Nikolas %A Kounev, Samuel %B Proceedings of 2018 IEEE/ACM International Conference on Utility and Cloud Computing Companion (UCC Companion) %D 2018 %I IEEE %P 11--12 %T TeaStore: A Micro-Service Reference Application for Cloud Researchers %U https://doi.org/10.1109/UCC-Companion.2018.00021 %X Researchers propose and employ various methods to analyze, model, optimize and manage modern distributed cloud applications. In order to demonstrate and evaluate these methods in realistic scenarios, researchers rely on reference applications. These applications should offer a range of different behaviors, degrees of freedom allowing for customization and should use a modern and representative technology stack. Existing testing and benchmarking applications are either outdated, designed for specific testing scenarios, or do not offer the necessary degrees of freedom. Further, most cloud reference applications are difficult to deploy and run. In this paper, we present the TeaStore, a micro-service-based test and reference cloud application. TeaStore offers services with various performance characteristics and a high degree of freedom regarding its deployment and configuration to be used as a cloud reference application for researchers. The TeaStore is designed for the evaluation of performance modeling and resource management techniques. We invite cloud researchers to use the TeaStore and provide it open-source, extendable, easily deployable and monitorable.
• 
  {TeaStore: A Micro-Service Reference Application for Benchmarking, Modeling and Resource Management Research}. von Kistowski, J{\’o}akim; Eismann, Simon; Schmitt, Norbert; Bauer, Andr{\’e}; Grohmann, Johannes; Kounev, Samuel; in Proceedings of the 26th IEEE International Symposium on the Modelling, Analysis, and Simulation of Computer and Telecommunication Systems 223–236.

  {Acceptance Rate: 29.5\% (23/78)}
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Modern distributed applications offer complex performance behavior and many degrees of freedom regarding deployment and configuration. Researchers employ various methods of analysis, modeling, and management that leverage these degrees of freedom to predict or improve non-functional properties of the software under consideration. In order to demonstrate and evaluate their applicability in the real world, methods resulting from such research areas require test and reference applications that offer a range of different behaviors, as well as the necessary degrees of freedom. Existing production software is often inaccessible for researchers or closed off to instrumentation. Existing testing and benchmarking frameworks, on the other hand, are either designed for specific testing scenarios, or they do not offer the necessary degrees of freedom. Further, most test applications are difficult to deploy and run, or are outdated. In this paper, we introduce the TeaStore, a state-of-the-art micro-service-based test and reference application. TeaStore offers services with different performance characteristics and many degrees of freedom regarding deployment and configuration to be used as a benchmarking framework for researchers. The TeaStore allows evaluating performance modeling and resource management techniques; it also offers instrumented variants to enable extensive run-time analysis. We demonstrate TeaStore's use in three contexts: performance modeling, cloud resource management, and energy efficiency analysis. Our experiments show that TeaStore can be used for evaluating novel approaches in these contexts and also motivates further research in the areas of performance modeling and resource management.

  @inproceedings{KiEiScBaGrKo2018-MASCOTS-TeaStore, abstract = {Modern distributed applications offer complex performance behavior and many degrees of freedom regarding deployment and configuration. Researchers employ various methods of analysis, modeling, and management that leverage these degrees of freedom to predict or improve non-functional properties of the software under consideration. In order to demonstrate and evaluate their applicability in the real world, methods resulting from such research areas require test and reference applications that offer a range of different behaviors, as well as the necessary degrees of freedom. Existing production software is often inaccessible for researchers or closed off to instrumentation. Existing testing and benchmarking frameworks, on the other hand, are either designed for specific testing scenarios, or they do not offer the necessary degrees of freedom. Further, most test applications are difficult to deploy and run, or are outdated. In this paper, we introduce the TeaStore, a state-of-the-art micro-service-based test and reference application. TeaStore offers services with different performance characteristics and many degrees of freedom regarding deployment and configuration to be used as a benchmarking framework for researchers. The TeaStore allows evaluating performance modeling and resource management techniques; it also offers instrumented variants to enable extensive run-time analysis. We demonstrate TeaStore's use in three contexts: performance modeling, cloud resource management, and energy efficiency analysis. Our experiments show that TeaStore can be used for evaluating novel approaches in these contexts and also motivates further research in the areas of performance modeling and resource management.}, author = {von Kistowski, J{\'o}akim and Eismann, Simon and Schmitt, Norbert and Bauer, Andr{\'e} and Grohmann, Johannes and Kounev, Samuel}, booktitle = {Proceedings of the 26th IEEE International Symposium on the Modelling, Analysis, and Simulation of Computer and Telecommunication Systems}, month = {September}, note = {{Acceptance Rate: 29.5\% (23/78)}}, pages = {223--236}, publisher = {IEEE Computer Society}, series = {MASCOTS '18}, title = {{TeaStore: A Micro-Service Reference Application for Benchmarking, Modeling and Resource Management Research}}, year = 2018 }

  %0 Conference Paper %1 KiEiScBaGrKo2018-MASCOTS-TeaStore %A von Kistowski, J{\'o}akim %A Eismann, Simon %A Schmitt, Norbert %A Bauer, Andr{\'e} %A Grohmann, Johannes %A Kounev, Samuel %B Proceedings of the 26th IEEE International Symposium on the Modelling, Analysis, and Simulation of Computer and Telecommunication Systems %D 2018 %I IEEE Computer Society %P 223--236 %T {TeaStore: A Micro-Service Reference Application for Benchmarking, Modeling and Resource Management Research} %U https://doi.org/10.1109/MASCOTS.2018.00030 %X Modern distributed applications offer complex performance behavior and many degrees of freedom regarding deployment and configuration. Researchers employ various methods of analysis, modeling, and management that leverage these degrees of freedom to predict or improve non-functional properties of the software under consideration. In order to demonstrate and evaluate their applicability in the real world, methods resulting from such research areas require test and reference applications that offer a range of different behaviors, as well as the necessary degrees of freedom. Existing production software is often inaccessible for researchers or closed off to instrumentation. Existing testing and benchmarking frameworks, on the other hand, are either designed for specific testing scenarios, or they do not offer the necessary degrees of freedom. Further, most test applications are difficult to deploy and run, or are outdated. In this paper, we introduce the TeaStore, a state-of-the-art micro-service-based test and reference application. TeaStore offers services with different performance characteristics and many degrees of freedom regarding deployment and configuration to be used as a benchmarking framework for researchers. The TeaStore allows evaluating performance modeling and resource management techniques; it also offers instrumented variants to enable extensive run-time analysis. We demonstrate TeaStore's use in three contexts: performance modeling, cloud resource management, and energy efficiency analysis. Our experiments show that TeaStore can be used for evaluating novel approaches in these contexts and also motivates further research in the areas of performance modeling and resource management.

• 
  {Local Approximation of Server Workloads}. Schmitt, Norbert;
  • [ BibTeX ]
  • [ EndNote ]
  @misc{talk-Schmitt-SPECRG2017, author = {Schmitt, Norbert}, howpublished = {Talk at SPEC RG Annual Meeting 2017}, month = {April}, title = {{Local Approximation of Server Workloads}}, year = 2017 }

  %0 Generic %1 talk-Schmitt-SPECRG2017 %A Schmitt, Norbert %D 2017 %T {Local Approximation of Server Workloads}
• 
  {Towards a Scalability and Energy Efficiency Benchmark for VNF}. Schmitt, Norbert; von Kistowski, J{\’o}akim; Kounev, Samuel; in Proceedings of the 9th TPC Technology Conference on Performance Engineering and Benchmarking
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Network Function Virtualization (NFV) is the transfer of network functions from dedicated devices to high-volume commodity servers. It opens opportunities for flexibility and energy savings. Concrete insights on the flexibility of specific NFV environments require measurement methodologies and benchmarks. However, current benchmarks are not measuring the ability of a virtual network function (VNF) to scale either horizontally or vertically. We therefore envision a new benchmark that measures a VNF's ability to scale while evaluating its energy efficiency at the same time. Such a benchmark would enable the selection of a suitable VNF for changing demands, deployed at an existing or new resource landscape, while minimizing energy costs.

  @inproceedings{ScKiKo2017-TPCTC-BenchmarkVision, abstract = {Network Function Virtualization (NFV) is the transfer of network functions from dedicated devices to high-volume commodity servers. It opens opportunities for flexibility and energy savings. Concrete insights on the flexibility of specific NFV environments require measurement methodologies and benchmarks. However, current benchmarks are not measuring the ability of a virtual network function (VNF) to scale either horizontally or vertically. We therefore envision a new benchmark that measures a VNF's ability to scale while evaluating its energy efficiency at the same time. Such a benchmark would enable the selection of a suitable VNF for changing demands, deployed at an existing or new resource landscape, while minimizing energy costs.}, author = {Schmitt, Norbert and von Kistowski, J{\'o}akim and Kounev, Samuel}, booktitle = {Proceedings of the 9th TPC Technology Conference on Performance Engineering and Benchmarking}, month = {August}, series = {TPCTC '17}, title = {{Towards a Scalability and Energy Efficiency Benchmark for VNF}}, year = 2017 }

  %0 Conference Paper %1 ScKiKo2017-TPCTC-BenchmarkVision %A Schmitt, Norbert %A von Kistowski, J{\'o}akim %A Kounev, Samuel %B Proceedings of the 9th TPC Technology Conference on Performance Engineering and Benchmarking %D 2017 %T {Towards a Scalability and Energy Efficiency Benchmark for VNF} %X Network Function Virtualization (NFV) is the transfer of network functions from dedicated devices to high-volume commodity servers. It opens opportunities for flexibility and energy savings. Concrete insights on the flexibility of specific NFV environments require measurement methodologies and benchmarks. However, current benchmarks are not measuring the ability of a virtual network function (VNF) to scale either horizontally or vertically. We therefore envision a new benchmark that measures a VNF's ability to scale while evaluating its energy efficiency at the same time. Such a benchmark would enable the selection of a suitable VNF for changing demands, deployed at an existing or new resource landscape, while minimizing energy costs.
• 
  {Emulating the Power Consumption Behavior of Server Workloads using CPU Performance Counters}. Schmitt, Norbert; von Kistowski, J{\’o}akim; Kounev, Samuel; in Proceedings of the 25th IEEE International Symposium on the Modelling, Analysis, and Simulation of Computer and Telecommunication Systems

  {(Short Paper) Acceptance Rate: 26/84 = 31\%}
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  The accurate measurement of a server's power consumption when running realistic workloads enables characterization of its energy efficiency and helps to make better provisioning and workload placement decisions. Information on the energy efficiency of a server for a given target workload can greatly influence such decisions and thus the final energy efficiency of a cluster or data center. However, measuring energy efficiency and power consumption of server applications has become challenging as applications are often distributed or require work intensive configuration, setup, and specialized load drivers for reproducible testing. As a result, it may be not feasible to perform tests using the actual workload that is to be deployed. We introduce an approach to create small-scale workloads that emulate the power consumption-relevant behavior of an application by deliberately triggering specific power relevant performance counter events. These workloads can then be easily deployed on a target server for fast and efficient power characterization. We validate the proposed approach by approximating the power consumption behavior of different workloads at multiple load levels. We show that our approach is capable of producing small-scale workloads that reflect the power consumption behavior of their reference applications over multiple load levels with a minimum error of less than 1%.

  @inproceedings{ScKiKo2017-MASCOTS-Pet, abstract = {The accurate measurement of a server's power consumption when running realistic workloads enables characterization of its energy efficiency and helps to make better provisioning and workload placement decisions. Information on the energy efficiency of a server for a given target workload can greatly influence such decisions and thus the final energy efficiency of a cluster or data center. However, measuring energy efficiency and power consumption of server applications has become challenging as applications are often distributed or require work intensive configuration, setup, and specialized load drivers for reproducible testing. As a result, it may be not feasible to perform tests using the actual workload that is to be deployed. We introduce an approach to create small-scale workloads that emulate the power consumption-relevant behavior of an application by deliberately triggering specific power relevant performance counter events. These workloads can then be easily deployed on a target server for fast and efficient power characterization. We validate the proposed approach by approximating the power consumption behavior of different workloads at multiple load levels. We show that our approach is capable of producing small-scale workloads that reflect the power consumption behavior of their reference applications over multiple load levels with a minimum error of less than 1%.}, author = {Schmitt, Norbert and von Kistowski, J{\'o}akim and Kounev, Samuel}, booktitle = {Proceedings of the 25th IEEE International Symposium on the Modelling, Analysis, and Simulation of Computer and Telecommunication Systems}, month = {September}, note = {{(Short Paper) Acceptance Rate: 26/84 = 31\%}}, series = {MASCOTS '17}, title = {{Emulating the Power Consumption Behavior of Server Workloads using CPU Performance Counters}}, year = 2017 }

  %0 Conference Paper %1 ScKiKo2017-MASCOTS-Pet %A Schmitt, Norbert %A von Kistowski, J{\'o}akim %A Kounev, Samuel %B Proceedings of the 25th IEEE International Symposium on the Modelling, Analysis, and Simulation of Computer and Telecommunication Systems %D 2017 %T {Emulating the Power Consumption Behavior of Server Workloads using CPU Performance Counters} %X The accurate measurement of a server's power consumption when running realistic workloads enables characterization of its energy efficiency and helps to make better provisioning and workload placement decisions. Information on the energy efficiency of a server for a given target workload can greatly influence such decisions and thus the final energy efficiency of a cluster or data center. However, measuring energy efficiency and power consumption of server applications has become challenging as applications are often distributed or require work intensive configuration, setup, and specialized load drivers for reproducible testing. As a result, it may be not feasible to perform tests using the actual workload that is to be deployed. We introduce an approach to create small-scale workloads that emulate the power consumption-relevant behavior of an application by deliberately triggering specific power relevant performance counter events. These workloads can then be easily deployed on a target server for fast and efficient power characterization. We validate the proposed approach by approximating the power consumption behavior of different workloads at multiple load levels. We show that our approach is capable of producing small-scale workloads that reflect the power consumption behavior of their reference applications over multiple load levels with a minimum error of less than 1%.
• 
  {Predicting Power Consumption of High-Memory-Bandwidth Workloads}. Schmitt, Norbert; von Kistowski, J{\’o}akim; Kounev, Samuel; in Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering 353–356.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  {High performance workloads with high bandwidth memory utilization are among the most power consuming software applications. When writing such applications, developers can directly influence power consumption of the final software through their choice of data size and traversal method, mostly due to caching characteristics. Explicit knowledge on how choices influence power consumption can thus lead to greater overall energy efficiency. In existing work, power prediction for memory accesses and high bandwidth applications requires either detailed measurement information on the system on which the software is executed or it is too generic, not taking significant aspects, such as caching and data size into account. In this paper, we propose a power model that bridges this gap by modeling power consumption based on concrete software properties, while considering hardware characteristics on a more abstract level, characterizing it primarily using publicly available data. The model is designed to enable developers to compare power consumption of implementation alternatives for high memory bandwidth software components. We validate our model by measuring modified versions of the high bandwidth benchmark stream. We show that our model can predict the relative change of power consumption due to implementation changes and the power consumption of a concrete system under test with an average error of 19 percent.}

  @inproceedings{ScKiKo2017-ICPE-MemoryPower, abstract = {{High performance workloads with high bandwidth memory utilization are among the most power consuming software applications. When writing such applications, developers can directly influence power consumption of the final software through their choice of data size and traversal method, mostly due to caching characteristics. Explicit knowledge on how choices influence power consumption can thus lead to greater overall energy efficiency. In existing work, power prediction for memory accesses and high bandwidth applications requires either detailed measurement information on the system on which the software is executed or it is too generic, not taking significant aspects, such as caching and data size into account. In this paper, we propose a power model that bridges this gap by modeling power consumption based on concrete software properties, while considering hardware characteristics on a more abstract level, characterizing it primarily using publicly available data. The model is designed to enable developers to compare power consumption of implementation alternatives for high memory bandwidth software components. We validate our model by measuring modified versions of the high bandwidth benchmark stream. We show that our model can predict the relative change of power consumption due to implementation changes and the power consumption of a concrete system under test with an average error of 19 percent.}}, address = {New York, NY, USA}, author = {Schmitt, Norbert and von Kistowski, J{\'o}akim and Kounev, Samuel}, booktitle = {Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering}, pages = {353--356}, publisher = {ACM}, series = {ICPE '17}, title = {{Predicting Power Consumption of High-Memory-Bandwidth Workloads}}, year = 2017 }

  %0 Conference Paper %1 ScKiKo2017-ICPE-MemoryPower %A Schmitt, Norbert %A von Kistowski, J{\'o}akim %A Kounev, Samuel %B Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering %C New York, NY, USA %D 2017 %I ACM %P 353--356 %T {Predicting Power Consumption of High-Memory-Bandwidth Workloads} %U http://doi.acm.org/10.1145/3030207.3030241 %X {High performance workloads with high bandwidth memory utilization are among the most power consuming software applications. When writing such applications, developers can directly influence power consumption of the final software through their choice of data size and traversal method, mostly due to caching characteristics. Explicit knowledge on how choices influence power consumption can thus lead to greater overall energy efficiency. In existing work, power prediction for memory accesses and high bandwidth applications requires either detailed measurement information on the system on which the software is executed or it is too generic, not taking significant aspects, such as caching and data size into account. In this paper, we propose a power model that bridges this gap by modeling power consumption based on concrete software properties, while considering hardware characteristics on a more abstract level, characterizing it primarily using publicly available data. The model is designed to enable developers to compare power consumption of implementation alternatives for high memory bandwidth software components. We validate our model by measuring modified versions of the high bandwidth benchmark stream. We show that our model can predict the relative change of power consumption due to implementation changes and the power consumption of a concrete system under test with an average error of 19 percent.}

• 
  {Local Approximation of NFV Workloads}. Schmitt, Norbert; Thesis; University of W{\"u}rzburg; Am Hubland, Informatikgeb{\"a}ude, 97074 W{\"u}rzburg, Germany.
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @mastersthesis{Schmitt2015, address = {Am Hubland, Informatikgeb{\"a}ude, 97074 W{\"u}rzburg, Germany}, author = {Schmitt, Norbert}, month = {October}, school = {University of W{\"u}rzburg}, title = {{Local Approximation of NFV Workloads}}, type = {{Master Thesis}}, year = 2016 }

  %0 Thesis %1 Schmitt2015 %A Schmitt, Norbert %C Am Hubland, Informatikgeb{\"a}ude, 97074 W{\"u}rzburg, Germany %D 2016 %T {Local Approximation of NFV Workloads}
• 
  {Triggering Performance Counters for Energy Efficiency Measurements}. Schmitt, Norbert;
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @misc{talk-Schmitt-SSP2016, author = {Schmitt, Norbert}, howpublished = {Experience Talk at SSP 2016}, month = {November}, title = {{Triggering Performance Counters for Energy Efficiency Measurements}}, year = 2016 }

  %0 Generic %1 talk-Schmitt-SSP2016 %A Schmitt, Norbert %D 2016 %T {Triggering Performance Counters for Energy Efficiency Measurements}

• 
  {Intelligentes Mapping f{\"u}r Indoor-Quadrocopter}. Schmitt, Norbert; Thesis; University of W{\"u}rzburg; Am Hubland, Informatikgeb{\"a}ude, 97074 W{\"u}rzburg, Germany.
  • [ BibTeX ]
  • [ EndNote ]
  @mastersthesis{Schmitt2012, address = {Am Hubland, Informatikgeb{\"a}ude, 97074 W{\"u}rzburg, Germany}, author = {Schmitt, Norbert}, month = {December}, school = {University of W{\"u}rzburg}, title = {{Intelligentes Mapping f{\"u}r Indoor-Quadrocopter}}, type = {{Bachelor Thesis}}, year = 2012 }

  %0 Thesis %1 Schmitt2012 %A Schmitt, Norbert %C Am Hubland, Informatikgeb{\"a}ude, 97074 W{\"u}rzburg, Germany %D 2012 %T {Intelligentes Mapping f{\"u}r Indoor-Quadrocopter}