Publications

• [ 2020 ]
• [ 2017 ]
• [ 2015 ]
• [ 2013 ]
• [ 2011 ]
• [ 2008 ]
• [ 2005 ]

• 
  Systems Benchmarking Kounev, Samuel; Lange, Klaus-Dieter; von Kistowski, Jóakim; (2020). (1st ed.) Springer International Publishing.
  • [ BibTeX ]
  • [ EndNote ]
  @book{KoLaKi-2020-SystemsBenchmarking, author = {Kounev, Samuel and Lange, Klaus-Dieter and von Kistowski, Jóakim}, edition = 1, keywords = {t_book}, publisher = {Springer International Publishing}, title = {{Systems Benchmarking}}, year = 2020 }

  %0 Book %1 KoLaKi-2020-SystemsBenchmarking %A Kounev, Samuel %A Lange, Klaus-Dieter %A von Kistowski, Jóakim %D 2020 %I Springer International Publishing %T Systems Benchmarking %7 1

• 
  Self-Aware Computing Systems Kounev, Samuel; Kephart, Jeffrey O.; Milenkoski, Aleksandar; Zhu, Xiaoyun; (2017). Springer Verlag, Berlin Heidelberg, Germany.
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  @book{KoKeMiZh2017-self, address = {{Berlin Heidelberg, Germany}}, author = {Kounev, Samuel and Kephart, Jeffrey O. and Milenkoski, Aleksandar and Zhu, Xiaoyun}, keywords = {Self-adaptive-systems}, publisher = {{Springer Verlag}}, title = {{Self-Aware Computing Systems}}, year = 2017 }

  %0 Book %1 KoKeMiZh2017-self %A Kounev, Samuel %A Kephart, Jeffrey O. %A Milenkoski, Aleksandar %A Zhu, Xiaoyun %C Berlin Heidelberg, Germany %D 2017 %I Springer Verlag %R 10.1007/978-3-319-47474-8 %T Self-Aware Computing Systems %U http://www.springer.com/de/book/9783319474724 %@ 978-3-319-47474-8

• 
  Proceeding of IEEE International Conference on Autonomic Computing (ICAC2015), Grenoble, France, July 8--10, 2015 Kounev, Samuel; (P. Lalanda; A. Diaconescu; L. Cherkasova; S. Kounev, eds.) (2015). IEEE Computer Society Conference Publishing Services (CPS), Los Alamitos, California.
  • [ BibTeX ]
  • [ EndNote ]
  @proceedings{proceedings-icac-2015, address = {Los Alamitos, California}, author = {Kounev, Samuel}, editor = {Lalanda, Philippe and Diaconescu, Ada and Cherkasova, Lucy and Kounev, Samuel}, keywords = {se2}, month = {July}, publisher = {IEEE Computer Society Conference Publishing Services (CPS)}, title = {Proceeding of IEEE International Conference on Autonomic Computing (ICAC2015), Grenoble, France, July 8–10, 2015}, year = 2015 }

  %0 Conference Proceedings %1 proceedings-icac-2015 %A Kounev, Samuel %C Los Alamitos, California %D 2015 %E Lalanda, Philippe %E Diaconescu, Ada %E Cherkasova, Lucy %E Kounev, Samuel %I IEEE Computer Society Conference Publishing Services (CPS) %T Proceeding of IEEE International Conference on Autonomic Computing (ICAC2015), Grenoble, France, July 8--10, 2015
• 
  Model-driven Algorithms and Architectures for Self-Aware Computing Systems Kounev, Samuel; in Dagstuhl Reports, (S. Kounev; X. Zhu; J. O. Kephart; M. Kwiatkowska; A. M. [Assistant Editor], eds.) (2015). Dagstuhl, Germany.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  This report documents the program and the outcomes of Dagstuhl Seminar 15041 "Model-driven Algorithms and Architectures for Self-Aware Computing Systems". The design of self-aware computing systems calls for an integrated interdisciplinary approach building on results from multiple areas of computer science and engineering, including software and systems engineering, systems modeling, simulation and analysis, autonomic and organic computing, machine learning and artificial intelligence, data center resource management, and so on. The Dagstuhl Seminar 15041 served as a platform to raise the awareness about the relevant research efforts in the respective research communities as well as existing synergies that can be exploited to advance the state-of-the-art, formulate a new research agenda that takes a broader view on the problem following an integrated and interdisciplinary approach, and establish collaborations between academia and industry.

  @proceedings{KoZhKeKw2015, abstract = {{This report documents the program and the outcomes of Dagstuhl Seminar 15041 "Model-driven Algorithms and Architectures for Self-Aware Computing Systems". The design of self-aware computing systems calls for an integrated interdisciplinary approach building on results from multiple areas of computer science and engineering, including software and systems engineering, systems modeling, simulation and analysis, autonomic and organic computing, machine learning and artificial intelligence, data center resource management, and so on. The Dagstuhl Seminar 15041 served as a platform to raise the awareness about the relevant research efforts in the respective research communities as well as existing synergies that can be exploited to advance the state-of-the-art, formulate a new research agenda that takes a broader view on the problem following an integrated and interdisciplinary approach, and establish collaborations between academia and industry.}}, address = {Dagstuhl, Germany}, author = {Kounev, Samuel}, editor = {Kounev, Samuel and Zhu, Xiaoyun and Kephart, Jeffrey O. and Kwiatkowska, Marta and Editor], Aleksandar Milenkoski [Assistant}, keywords = {Self-adaptive-systems}, month = {January}, series = {Dagstuhl Reports}, title = {{Model-driven Algorithms and Architectures for Self-Aware Computing Systems}}, year = 2015 }

  %0 Conference Proceedings %1 KoZhKeKw2015 %A Kounev, Samuel %B Dagstuhl Reports %C Dagstuhl, Germany %D 2015 %E Kounev, Samuel %E Zhu, Xiaoyun %E Kephart, Jeffrey O. %E Kwiatkowska, Marta %E Editor], Aleksandar Milenkoski [Assistant %T Model-driven Algorithms and Architectures for Self-Aware Computing Systems %U http://drops.dagstuhl.de/opus/volltexte/2015/5038/ %X This report documents the program and the outcomes of Dagstuhl Seminar 15041 "Model-driven Algorithms and Architectures for Self-Aware Computing Systems". The design of self-aware computing systems calls for an integrated interdisciplinary approach building on results from multiple areas of computer science and engineering, including software and systems engineering, systems modeling, simulation and analysis, autonomic and organic computing, machine learning and artificial intelligence, data center resource management, and so on. The Dagstuhl Seminar 15041 served as a platform to raise the awareness about the relevant research efforts in the respective research communities as well as existing synergies that can be exploited to advance the state-of-the-art, formulate a new research agenda that takes a broader view on the problem following an integrated and interdisciplinary approach, and establish collaborations between academia and industry.

• 
  Proceedings of the 2013 International Workshop on Hot Topics in Cloud Services (HotTopiCS 2013) Kounev, Samuel; (S. Kounev; S. Zschaler; K. Sachs, eds.) (2013). ACM.
  • [ BibTeX ]
  • [ EndNote ]
  @proceedings{KoZsSa2013-HOTTOPICS-Proceedings, author = {Kounev, Samuel}, editor = {Kounev, Samuel and Zschaler, Steffen and Sachs, Kai}, keywords = {t_proceedings}, month = {April}, publisher = {ACM}, title = {Proceedings of the 2013 International Workshop on Hot Topics in Cloud Services (HotTopiCS 2013)}, year = 2013 }

  %0 Conference Proceedings %1 KoZsSa2013-HOTTOPICS-Proceedings %A Kounev, Samuel %D 2013 %E Kounev, Samuel %E Zschaler, Steffen %E Sachs, Kai %I ACM %T Proceedings of the 2013 International Workshop on Hot Topics in Cloud Services (HotTopiCS 2013)

• 
  ICPE’11 - 2nd Joint ACM/SPEC International Conference on Performance Engineering, Karlsruhe, Germany, March 14--16, 2011 Kounev, Samuel; (S. Kounev; V. Cortellessa; R. Mirandola; D. J. Lilja, eds.) (2011). ACM, New York, NY, USA.
  • [ BibTeX ]
  • [ EndNote ]
  @proceedings{DBLP-conf-wosp-2011, address = {New York, NY, USA}, author = {Kounev, Samuel}, editor = {Kounev, Samuel and Cortellessa, Vittorio and Mirandola, Raffaela and Lilja, David J.}, keywords = {Power}, month = {March}, publisher = {ACM}, title = {ICPE'11 - 2nd Joint ACM/SPEC International Conference on Performance Engineering, Karlsruhe, Germany, March 14–16, 2011}, year = 2011 }

  %0 Conference Proceedings %1 DBLP-conf-wosp-2011 %A Kounev, Samuel %C New York, NY, USA %D 2011 %E Kounev, Samuel %E Cortellessa, Vittorio %E Mirandola, Raffaela %E Lilja, David J. %I ACM %T ICPE'11 - 2nd Joint ACM/SPEC International Conference on Performance Engineering, Karlsruhe, Germany, March 14--16, 2011

• 
  Performance Evaluation: Metrics, Models and Benchmarks, Proceedings of the 2008 SPEC International Performance Evaluation Workshop (SIPEW 2008), Darmstadt, Germany, June 27-28 Kounev, Samuel; in Lecture Notes in Computer Science (LNCS), (S. Kounev; I. Gorton; K. Sachs, eds.) (2008). (Vol. 5119) Springer, Heidelberg, Germany.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  This book constitutes the refereed proceedings of the SPEC International Performance Evaluation Workshop, SIPEW 2008, held in Darmstadt, Germany, in June 2008. The 17 revised full papers presented together with 3 keynote talks were carefully reviewed and selected out of 39 submissions for inclusion in the book. The papers are organized in topical sections on models for software performance engineering; benchmarks and workload characterization; Web services and service-oriented architectures; power and performance; and profiling, monitoring and optimization.

  @proceedings{kounev2008c, abstract = {This book constitutes the refereed proceedings of the SPEC International Performance Evaluation Workshop, SIPEW 2008, held in Darmstadt, Germany, in June 2008. The 17 revised full papers presented together with 3 keynote talks were carefully reviewed and selected out of 39 submissions for inclusion in the book. The papers are organized in topical sections on models for software performance engineering; benchmarks and workload characterization; Web services and service-oriented architectures; power and performance; and profiling, monitoring and optimization.}, address = {Heidelberg, Germany}, author = {Kounev, Samuel}, editor = {Kounev, Samuel and Gorton, Ian and Sachs, Kai}, keywords = {Power}, month = {June}, publisher = {Springer}, series = {Lecture Notes in Computer Science (LNCS)}, title = {{Performance Evaluation: Metrics, Models and Benchmarks, Proceedings of the 2008 SPEC International Performance Evaluation Workshop (SIPEW 2008), Darmstadt, Germany, June 27-28}}, volume = 5119, year = 2008 }

  %0 Conference Proceedings %1 kounev2008c %A Kounev, Samuel %B Lecture Notes in Computer Science (LNCS) %C Heidelberg, Germany %D 2008 %E Kounev, Samuel %E Gorton, Ian %E Sachs, Kai %I Springer %T Performance Evaluation: Metrics, Models and Benchmarks, Proceedings of the 2008 SPEC International Performance Evaluation Workshop (SIPEW 2008), Darmstadt, Germany, June 27-28 %U http://www.springer.com/computer/programming/book/978-3-540-69813-5 %V 5119 %X This book constitutes the refereed proceedings of the SPEC International Performance Evaluation Workshop, SIPEW 2008, held in Darmstadt, Germany, in June 2008. The 17 revised full papers presented together with 3 keynote talks were carefully reviewed and selected out of 39 submissions for inclusion in the book. The papers are organized in topical sections on models for software performance engineering; benchmarks and workload characterization; Web services and service-oriented architectures; power and performance; and profiling, monitoring and optimization.

• 
  Performance Engineering of Distributed Component-Based Systems - Benchmarking, Modeling and Performance Prediction Kounev, Samuel; (2005). Shaker Verlag, Ph.D. Thesis, Technische Universität Darmstadt, Germany.

  Distinguished Dissertation Award
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @book{Ko2005-TUD-PhD_Thesis, author = {Kounev, Samuel}, keywords = {QPME_Bibliography}, month = {December}, note = {<b>Distinguished Dissertation Award</b>}, publisher = {Shaker Verlag, Ph.D. Thesis, Technische Universit{\"a}t Darmstadt, Germany}, title = {Performance Engineering of Distributed Component-Based Systems - Benchmarking, Modeling and Performance Prediction}, year = 2005 }

  %0 Book %1 Ko2005-TUD-PhD_Thesis %A Kounev, Samuel %D 2005 %I Shaker Verlag, Ph.D. Thesis, Technische Universität Darmstadt, Germany %T Performance Engineering of Distributed Component-Based Systems - Benchmarking, Modeling and Performance Prediction

• [ 2023 ]
• [ 2022 ]
• [ 2021 ]
• [ 2020 ]
• [ 2019 ]
• [ 2018 ]
• [ 2017 ]
• [ 2016 ]
• [ 2015 ]
• [ 2014 ]
• [ 2013 ]
• [ 2011 ]
• [ 2009 ]
• [ 2008 ]
• [ 2006 ]
• [ 2004 ]
• [ 1999 ]

• 
  Recommendation of Secure Group Communication Schemes using Multi Objective Optimization Prantl, Thomas; Bauer, André; Iffländer, Lukas; Krupitzer, Christian; Kounev, Samuel; in International Journal of Information Security (2023).
  • [ BibTeX ]
  • [ EndNote ]
  @article{PrBaIfKrKo_IJoIS_SGC_Pareto, author = {Prantl, Thomas and Bauer, André and Iffländer, Lukas and Krupitzer, Christian and Kounev, Samuel}, journal = {International Journal of Information Security}, keywords = {myown}, title = {Recommendation of Secure Group Communication Schemes using Multi Objective Optimization}, year = 2023 }

  %0 Journal Article %1 PrBaIfKrKo_IJoIS_SGC_Pareto %A Prantl, Thomas %A Bauer, André %A Iffländer, Lukas %A Krupitzer, Christian %A Kounev, Samuel %D 2023 %J International Journal of Information Security %T Recommendation of Secure Group Communication Schemes using Multi Objective Optimization
• 
  A literature review of IoT and CPS—What they are, and what they are not Lesch, Veronika; Züfle, Marwin; Bauer, André; Iffländer, Lukas; Krupitzer, Christian; Kounev, Samuel; in Journal of Systems and Software (2023). 111631. Elsevier.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Today, we are confronted with many concepts such as Cyber–Physical-Systems (CPS), Internet of Things (IoT), Industry 4.0, Industrial Internet, Ubiquitous Computing, Pervasive Computing and many more. Some researchers use all of these terms interchangeably, while others interpret them in ways that contradict each other. The inconsistent and interchangeable usage of these terms creates the impression that authors abuse them as buzzwords to attract attention. Hence, the question arises: Is the existence of all these terms justified? In this paper, we first look at the origin of the terms. Then, we focus on Internet of Things (IoT) and Cyber-Physical-Systems (CPS) as those terms are more often used as the others and further often seen as underlying technologies. We presentthe results of a literature review, including academic, industry and gray literature, with the objective to identify and discuss several clusters of similar statements on both terms. Building on this, we present definitions for IoT and CPS that reflect the core intuition of the terms as found in the literature review while providing a clear demarcation of the two terms. Then, we illustrate the applicability of our findings on several use cases. Finally, we discuss the relation to the other topics closely related to adaptive systems, namely Industry 4.0, Industrial Internet, Ubiquitous Computing, and Pervasive Computing.

  @article{lesch2023literature, abstract = {Today, we are confronted with many concepts such as Cyber–Physical-Systems (CPS), Internet of Things (IoT), Industry 4.0, Industrial Internet, Ubiquitous Computing, Pervasive Computing and many more. Some researchers use all of these terms interchangeably, while others interpret them in ways that contradict each other. The inconsistent and interchangeable usage of these terms creates the impression that authors abuse them as buzzwords to attract attention. Hence, the question arises: Is the existence of all these terms justified? In this paper, we first look at the origin of the terms. Then, we focus on Internet of Things (IoT) and Cyber-Physical-Systems (CPS) as those terms are more often used as the others and further often seen as underlying technologies. We presentthe results of a literature review, including academic, industry and gray literature, with the objective to identify and discuss several clusters of similar statements on both terms. Building on this, we present definitions for IoT and CPS that reflect the core intuition of the terms as found in the literature review while providing a clear demarcation of the two terms. Then, we illustrate the applicability of our findings on several use cases. Finally, we discuss the relation to the other topics closely related to adaptive systems, namely Industry 4.0, Industrial Internet, Ubiquitous Computing, and Pervasive Computing.}, author = {Lesch, Veronika and Z{\"u}fle, Marwin and Bauer, Andr{\'e} and Iffl{\"a}nder, Lukas and Krupitzer, Christian and Kounev, Samuel}, journal = {Journal of Systems and Software}, keywords = {myown}, pages = 111631, publisher = {Elsevier}, title = {A literature review of IoT and CPS—What they are, and what they are not}, year = 2023 }

  %0 Journal Article %1 lesch2023literature %A Lesch, Veronika %A Züfle, Marwin %A Bauer, André %A Iffländer, Lukas %A Krupitzer, Christian %A Kounev, Samuel %D 2023 %I Elsevier %J Journal of Systems and Software %P 111631 %T A literature review of IoT and CPS—What they are, and what they are not %U https://www.sciencedirect.com/science/article/pii/S0164121223000262 %X Today, we are confronted with many concepts such as Cyber–Physical-Systems (CPS), Internet of Things (IoT), Industry 4.0, Industrial Internet, Ubiquitous Computing, Pervasive Computing and many more. Some researchers use all of these terms interchangeably, while others interpret them in ways that contradict each other. The inconsistent and interchangeable usage of these terms creates the impression that authors abuse them as buzzwords to attract attention. Hence, the question arises: Is the existence of all these terms justified? In this paper, we first look at the origin of the terms. Then, we focus on Internet of Things (IoT) and Cyber-Physical-Systems (CPS) as those terms are more often used as the others and further often seen as underlying technologies. We presentthe results of a literature review, including academic, industry and gray literature, with the objective to identify and discuss several clusters of similar statements on both terms. Building on this, we present definitions for IoT and CPS that reflect the core intuition of the terms as found in the literature review while providing a clear demarcation of the two terms. Then, we illustrate the applicability of our findings on several use cases. Finally, we discuss the relation to the other topics closely related to adaptive systems, namely Industry 4.0, Industrial Internet, Ubiquitous Computing, and Pervasive Computing.
• 
  Optimizing storage assignment, order picking, and their interaction in mezzanine warehouses Lesch, Veronika; Müller, Patrick B.M.; Krämer, Moritz; Hadry, Marius; Kounev, Samuel; Krupitzer, Christian; in Applied intelligence (2023). 25.
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  @article{lesch2023optimizing, author = {Lesch, Veronika and Müller, Patrick B.M. and Krämer, Moritz and Hadry, Marius and Kounev, Samuel and Krupitzer, Christian}, journal = {Applied intelligence}, keywords = {myown}, month = {feb}, pages = 25, title = {Optimizing storage assignment, order picking, and their interaction in mezzanine warehouses}, year = 2023 }

  %0 Journal Article %1 lesch2023optimizing %A Lesch, Veronika %A Müller, Patrick B.M. %A Krämer, Moritz %A Hadry, Marius %A Kounev, Samuel %A Krupitzer, Christian %D 2023 %J Applied intelligence %P 25 %T Optimizing storage assignment, order picking, and their interaction in mezzanine warehouses %U https://doi.org/10.1007/s10489-022-04443-x
• 
  MVNOCoreSim: A Digital Twin for Virtualized IoT-centric Mobile Core Networks Geissler, Stefan; Wamser, Florian; Bauer, Wolfgang; Gebert, Steffen; Kounev, Samuel; Hossfeld, Tobias; in IEEE Internet of Things (2023).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  The rapid growth of connected devices has led to the implementation of IoT-centric platforms designed to provide connectivity for machine-to-machine type communication. Specifically, mobile virtual network operators (MVNOs) provide global service for IoT use cases by leveraging roaming in readily deployed physical networks. This global deployment of connected devices poses a significant challenge regarding the operation of centralized core networks with respect to dimensioning, scaling as well as system survivability in case of signaling incidents. Overload control mechanisms for IoT mobile networks offer a proactive solution for mitigating excessive signaling traffic from IoT devices in mobile core networks. However, global scaling and a heterogeneous composition of devices present network operators with additional challenges. To address these challenges, this work presents a detailed, protocol level simulation framework of a real-world IoT MVNO core network. We present both models for the expected IoT signaling load and the processing of signaling messages based on measurements in a live, production environment as well as a dedicated testbed. Finally, we present a case study on various overload mechanisms and identify critical performance characteristics to compare their performance. The results of this study categorize overload control mechanisms and shed light on the necessity for MVNOs to deal with the upcoming IoT traffic by defining appropriate overload control mechanisms in mobile core networks to ensure network survivability under unforeseen conditions.

  @article{geissler2023mvnocoresim, abstract = {The rapid growth of connected devices has led to the implementation of IoT-centric platforms designed to provide connectivity for machine-to-machine type communication. Specifically, mobile virtual network operators (MVNOs) provide global service for IoT use cases by leveraging roaming in readily deployed physical networks. This global deployment of connected devices poses a significant challenge regarding the operation of centralized core networks with respect to dimensioning, scaling as well as system survivability in case of signaling incidents. Overload control mechanisms for IoT mobile networks offer a proactive solution for mitigating excessive signaling traffic from IoT devices in mobile core networks. However, global scaling and a heterogeneous composition of devices present network operators with additional challenges. To address these challenges, this work presents a detailed, protocol level simulation framework of a real-world IoT MVNO core network. We present both models for the expected IoT signaling load and the processing of signaling messages based on measurements in a live, production environment as well as a dedicated testbed. Finally, we present a case study on various overload mechanisms and identify critical performance characteristics to compare their performance. The results of this study categorize overload control mechanisms and shed light on the necessity for MVNOs to deal with the upcoming IoT traffic by defining appropriate overload control mechanisms in mobile core networks to ensure network survivability under unforeseen conditions.}, author = {Geissler, Stefan and Wamser, Florian and Bauer, Wolfgang and Gebert, Steffen and Kounev, Samuel and Hossfeld, Tobias}, journal = {IEEE Internet of Things}, keywords = {myown}, title = {MVNOCoreSim: A Digital Twin for Virtualized IoT-centric Mobile Core Networks}, year = 2023 }

  %0 Journal Article %1 geissler2023mvnocoresim %A Geissler, Stefan %A Wamser, Florian %A Bauer, Wolfgang %A Gebert, Steffen %A Kounev, Samuel %A Hossfeld, Tobias %D 2023 %J IEEE Internet of Things %T MVNOCoreSim: A Digital Twin for Virtualized IoT-centric Mobile Core Networks %U https://ieeexplore.ieee.org/document/10089151 %X The rapid growth of connected devices has led to the implementation of IoT-centric platforms designed to provide connectivity for machine-to-machine type communication. Specifically, mobile virtual network operators (MVNOs) provide global service for IoT use cases by leveraging roaming in readily deployed physical networks. This global deployment of connected devices poses a significant challenge regarding the operation of centralized core networks with respect to dimensioning, scaling as well as system survivability in case of signaling incidents. Overload control mechanisms for IoT mobile networks offer a proactive solution for mitigating excessive signaling traffic from IoT devices in mobile core networks. However, global scaling and a heterogeneous composition of devices present network operators with additional challenges. To address these challenges, this work presents a detailed, protocol level simulation framework of a real-world IoT MVNO core network. We present both models for the expected IoT signaling load and the processing of signaling messages based on measurements in a live, production environment as well as a dedicated testbed. Finally, we present a case study on various overload mechanisms and identify critical performance characteristics to compare their performance. The results of this study categorize overload control mechanisms and shed light on the necessity for MVNOs to deal with the upcoming IoT traffic by defining appropriate overload control mechanisms in mobile core networks to ensure network survivability under unforeseen conditions.

• 
  CortexVR: Immersive analysis and training of cognitive executive functions of soccer players using virtual reality and machine learning Krupitzer, Christian; Naber, Jens; Stauffert, Jan-Philipp; Mayer, Jan; Spielmann, Jan; Ehmann, Paul; Boci, Noel; Bürkle, Maurice; Ho, André; Komorek, Clemens; Heinickel, Felix; Kounev, Samuel; Becker, Christian; Latoschik, Marc Erich; in Frontiers in Psychology (2022). 5139. Frontiers.
  • [ BibTeX ]
  • [ EndNote ]
  @article{krupitzer2022cortexvr, author = {Krupitzer, Christian and Naber, Jens and Stauffert, Jan-Philipp and Mayer, Jan and Spielmann, Jan and Ehmann, Paul and Boci, Noel and B{\"u}rkle, Maurice and Ho, Andr{\'e} and Komorek, Clemens and Heinickel, Felix and Kounev, Samuel and Becker, Christian and Latoschik, Marc Erich}, journal = {Frontiers in Psychology}, keywords = {myown}, pages = 5139, publisher = {Frontiers}, title = {CortexVR: Immersive analysis and training of cognitive executive functions of soccer players using virtual reality and machine learning}, year = 2022 }

  %0 Journal Article %1 krupitzer2022cortexvr %A Krupitzer, Christian %A Naber, Jens %A Stauffert, Jan-Philipp %A Mayer, Jan %A Spielmann, Jan %A Ehmann, Paul %A Boci, Noel %A Bürkle, Maurice %A Ho, André %A Komorek, Clemens %A Heinickel, Felix %A Kounev, Samuel %A Becker, Christian %A Latoschik, Marc Erich %D 2022 %I Frontiers %J Frontiers in Psychology %P 5139 %T CortexVR: Immersive analysis and training of cognitive executive functions of soccer players using virtual reality and machine learning
• 
  Self-Aware Optimization of Adaptation Planning Strategies Lesch, Veronika; Hadry, Marius; Kounev, Samuel; Krupitzer, Christian; in ACM Transactions on Autonomous and Adaptive Systems (2022).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  In today’s world, circumstances, processes, and requirements for software systems are becoming increasingly complex. In order to operate properly in such dynamic environments, software systems must adapt to these changes, which has led to the research area of Self-Adaptive Systems (SAS). Platooning is one example of adaptive systems in Intelligent Transportation Systems, which is the ability of vehicles to travel with close inter-vehicle distances. This technology leads to an increase in road throughput and safety, which directly addresses the increased infrastructure needs due to increased traffic on the roads. However, the No-Free-Lunch theorem states that the performance of one adaptation planning strategy is not necessarily transferable to other problems. Moreover, especially in the field of SAS, the selection of the most appropriate strategy depends on the current situation of the system. In this paper, we address the problem of self-aware optimization of adaptation planning strategies by designing a framework that includes situation detection, strategy selection, and parameter optimization of the selected strategies. We apply our approach on the case study platooning coordination and evaluate the performance of the proposed framework.

  @article{lesch2022selfaware, abstract = {In today’s world, circumstances, processes, and requirements for software systems are becoming increasingly complex. In order to operate properly in such dynamic environments, software systems must adapt to these changes, which has led to the research area of Self-Adaptive Systems (SAS). Platooning is one example of adaptive systems in Intelligent Transportation Systems, which is the ability of vehicles to travel with close inter-vehicle distances. This technology leads to an increase in road throughput and safety, which directly addresses the increased infrastructure needs due to increased traffic on the roads. However, the No-Free-Lunch theorem states that the performance of one adaptation planning strategy is not necessarily transferable to other problems. Moreover, especially in the field of SAS, the selection of the most appropriate strategy depends on the current situation of the system. In this paper, we address the problem of self-aware optimization of adaptation planning strategies by designing a framework that includes situation detection, strategy selection, and parameter optimization of the selected strategies. We apply our approach on the case study platooning coordination and evaluate the performance of the proposed framework.}, author = {Lesch, Veronika and Hadry, Marius and Kounev, Samuel and Krupitzer, Christian}, journal = {ACM Transactions on Autonomous and Adaptive Systems}, keywords = {myown}, month = {oct}, title = {Self-Aware Optimization of Adaptation Planning Strategies}, year = 2022 }

  %0 Journal Article %1 lesch2022selfaware %A Lesch, Veronika %A Hadry, Marius %A Kounev, Samuel %A Krupitzer, Christian %D 2022 %J ACM Transactions on Autonomous and Adaptive Systems %T Self-Aware Optimization of Adaptation Planning Strategies %U https://doi.org/10.1145/3568680 %X In today’s world, circumstances, processes, and requirements for software systems are becoming increasingly complex. In order to operate properly in such dynamic environments, software systems must adapt to these changes, which has led to the research area of Self-Adaptive Systems (SAS). Platooning is one example of adaptive systems in Intelligent Transportation Systems, which is the ability of vehicles to travel with close inter-vehicle distances. This technology leads to an increase in road throughput and safety, which directly addresses the increased infrastructure needs due to increased traffic on the roads. However, the No-Free-Lunch theorem states that the performance of one adaptation planning strategy is not necessarily transferable to other problems. Moreover, especially in the field of SAS, the selection of the most appropriate strategy depends on the current situation of the system. In this paper, we address the problem of self-aware optimization of adaptation planning strategies by designing a framework that includes situation detection, strategy selection, and parameter optimization of the selected strategies. We apply our approach on the case study platooning coordination and evaluate the performance of the proposed framework.
• 
  A Survey on Secure Group Communication Schemes with Focus on IoT Communication Prantl, Thomas; Zeck, Timo; Bauer, André; Ten, Peter; Prantl, Dominik; Ben Yahya, Ala Eddine; Iffländer, Lukas; Dmitrienko, Alexandra; Krupitzer, Christian; Kounev, Samuel; in IEEE Access (2022). IEEE.

  Accepted but not yet published
  • [ BibTeX ]
  • [ EndNote ]
  @article{PrZeBaTePrYaIfDmKrKo_IEEEAccess_SGCSurvey, author = {Prantl, Thomas and Zeck, Timo and Bauer, André and Ten, Peter and Prantl, Dominik and Ben Yahya, Ala Eddine and Iffländer, Lukas and Dmitrienko, Alexandra and Krupitzer, Christian and Kounev, Samuel}, journal = {IEEE Access}, keywords = {myown}, note = {Accepted but not yet published}, publisher = {IEEE}, title = {A Survey on Secure Group Communication Schemes with Focus on IoT Communication}, year = 2022 }

  %0 Journal Article %1 PrZeBaTePrYaIfDmKrKo_IEEEAccess_SGCSurvey %A Prantl, Thomas %A Zeck, Timo %A Bauer, André %A Ten, Peter %A Prantl, Dominik %A Ben Yahya, Ala Eddine %A Iffländer, Lukas %A Dmitrienko, Alexandra %A Krupitzer, Christian %A Kounev, Samuel %D 2022 %I IEEE %J IEEE Access %T A Survey on Secure Group Communication Schemes with Focus on IoT Communication
• 
  A literature review on optimization techniques for adaptation planning in adaptive systems: State of the art and research directions Henrichs, Elia; Lesch, Veronika; Straesser, Martin; Kounev, Samuel; Krupitzer, Christian; in Information and Software Technology (2022). 149 106940.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Context: Recent developments in modern IT systems including internet of things, edge/fog computing, or cyber–physical systems support intelligent and seamless interaction between users and systems. This requires a reaction to changes in their environment or the system. Adaptive systems provide mechanisms for these reactions. Objective: To implement this functionality, several approaches for the planning of adaptations exist that rely on rules, utility functions, or advanced techniques, such as machine learning. As the adaptation space with possible options is often extensively huge, optimization techniques might support efficient determination of the adaptation space and identify the system’s optimal configuration. With this paper, we provide a systematic review of adaptation planning as the optimization target. Method: In this paper, we review which optimization techniques are applied for adaptation planning in adaptive systems using a systematic literature review approach. Results: We reviewed 115 paper in detail out of an initial search set of 9,588 papers. Our analysis reveals that learning techniques and genetic algorithms are by far dominant; in total, heuristics (anytime learning) are more frequently applied as exact algorithms. We observed that around 57% of the approaches target multi-objectiveness and around 30% integrate distributed optimization. As last dimension, we focused on situation-awareness, which is only supported by two approaches. Conclusion: In this paper, we provide an overview of the current state of the art of approaches that rely on optimization techniques for planning adaptations in adaptive systems and further derive open research challenges, in particular regarding the integration of distributed optimization and situation-awareness.

  @article{HENRICHS2022106940, abstract = {Context: Recent developments in modern IT systems including internet of things, edge/fog computing, or cyber–physical systems support intelligent and seamless interaction between users and systems. This requires a reaction to changes in their environment or the system. Adaptive systems provide mechanisms for these reactions. Objective: To implement this functionality, several approaches for the planning of adaptations exist that rely on rules, utility functions, or advanced techniques, such as machine learning. As the adaptation space with possible options is often extensively huge, optimization techniques might support efficient determination of the adaptation space and identify the system’s optimal configuration. With this paper, we provide a systematic review of adaptation planning as the optimization target. Method: In this paper, we review which optimization techniques are applied for adaptation planning in adaptive systems using a systematic literature review approach. Results: We reviewed 115 paper in detail out of an initial search set of 9,588 papers. Our analysis reveals that learning techniques and genetic algorithms are by far dominant; in total, heuristics (anytime learning) are more frequently applied as exact algorithms. We observed that around 57% of the approaches target multi-objectiveness and around 30% integrate distributed optimization. As last dimension, we focused on situation-awareness, which is only supported by two approaches. Conclusion: In this paper, we provide an overview of the current state of the art of approaches that rely on optimization techniques for planning adaptations in adaptive systems and further derive open research challenges, in particular regarding the integration of distributed optimization and situation-awareness.}, author = {Henrichs, Elia and Lesch, Veronika and Straesser, Martin and Kounev, Samuel and Krupitzer, Christian}, journal = {Information and Software Technology}, keywords = {myown}, pages = 106940, title = {A literature review on optimization techniques for adaptation planning in adaptive systems: State of the art and research directions}, volume = 149, year = 2022 }

  %0 Journal Article %1 HENRICHS2022106940 %A Henrichs, Elia %A Lesch, Veronika %A Straesser, Martin %A Kounev, Samuel %A Krupitzer, Christian %D 2022 %J Information and Software Technology %P 106940 %T A literature review on optimization techniques for adaptation planning in adaptive systems: State of the art and research directions %U https://www.sciencedirect.com/science/article/pii/S0950584922000891 %V 149 %X Context: Recent developments in modern IT systems including internet of things, edge/fog computing, or cyber–physical systems support intelligent and seamless interaction between users and systems. This requires a reaction to changes in their environment or the system. Adaptive systems provide mechanisms for these reactions. Objective: To implement this functionality, several approaches for the planning of adaptations exist that rely on rules, utility functions, or advanced techniques, such as machine learning. As the adaptation space with possible options is often extensively huge, optimization techniques might support efficient determination of the adaptation space and identify the system’s optimal configuration. With this paper, we provide a systematic review of adaptation planning as the optimization target. Method: In this paper, we review which optimization techniques are applied for adaptation planning in adaptive systems using a systematic literature review approach. Results: We reviewed 115 paper in detail out of an initial search set of 9,588 papers. Our analysis reveals that learning techniques and genetic algorithms are by far dominant; in total, heuristics (anytime learning) are more frequently applied as exact algorithms. We observed that around 57% of the approaches target multi-objectiveness and around 30% integrate distributed optimization. As last dimension, we focused on situation-awareness, which is only supported by two approaches. Conclusion: In this paper, we provide an overview of the current state of the art of approaches that rely on optimization techniques for planning adaptations in adaptive systems and further derive open research challenges, in particular regarding the integration of distributed optimization and situation-awareness.
• 
  A case study on the stability of performance tests for serverless applications Eismann, Simon; Costa, Diego; Liao, Lizhi; Bezemer, Cor-Paul; Shang, Weiyi; van Hoorn, André; Kounev, Samuel; in Journal of Systems and Software (JSS) (2022). 189
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Context: While in serverless computing, application resource management and operational concerns are generally delegated to the cloud provider, ensuring that serverless applications meet their performance requirements is still a responsibility of the developers. Performance testing is a commonly used performance assessment practice; however, it traditionally requires visibility of the resource environment. Objective: In this study, we investigate whether performance tests of serverless applications are stable, that is, if their results are reproducible, and what implications the serverless paradigm has for performance tests. Method: We conduct a case study where we collect two datasets of performance test results: (a) repetitions of performance tests for varying memory size and load intensities and (b) three repetitions of the same performance test every day for ten months. Results: We find that performance tests of serverless applications are comparatively stable if conducted on the same day. However, we also observe short-term performance variations and frequent long-term performance changes. Conclusion: Performance tests for serverless applications can be stable; however, the serverless model impacts the planning, execution, and analysis of performance tests.

  @article{eismann2022study, abstract = {Context: While in serverless computing, application resource management and operational concerns are generally delegated to the cloud provider, ensuring that serverless applications meet their performance requirements is still a responsibility of the developers. Performance testing is a commonly used performance assessment practice; however, it traditionally requires visibility of the resource environment. Objective: In this study, we investigate whether performance tests of serverless applications are stable, that is, if their results are reproducible, and what implications the serverless paradigm has for performance tests. Method: We conduct a case study where we collect two datasets of performance test results: (a) repetitions of performance tests for varying memory size and load intensities and (b) three repetitions of the same performance test every day for ten months. Results: We find that performance tests of serverless applications are comparatively stable if conducted on the same day. However, we also observe short-term performance variations and frequent long-term performance changes. Conclusion: Performance tests for serverless applications can be stable; however, the serverless model impacts the planning, execution, and analysis of performance tests.}, author = {Eismann, Simon and Costa, Diego and Liao, Lizhi and Bezemer, Cor-Paul and Shang, Weiyi and van Hoorn, André and Kounev, Samuel}, journal = {Journal of Systems and Software (JSS)}, keywords = {myown}, title = {A case study on the stability of performance tests for serverless applications}, volume = 189, year = 2022 }

  %0 Journal Article %1 eismann2022study %A Eismann, Simon %A Costa, Diego %A Liao, Lizhi %A Bezemer, Cor-Paul %A Shang, Weiyi %A van Hoorn, André %A Kounev, Samuel %D 2022 %J Journal of Systems and Software (JSS) %T A case study on the stability of performance tests for serverless applications %U https://www.sciencedirect.com/science/article/abs/pii/S0164121222000498 %V 189 %X Context: While in serverless computing, application resource management and operational concerns are generally delegated to the cloud provider, ensuring that serverless applications meet their performance requirements is still a responsibility of the developers. Performance testing is a commonly used performance assessment practice; however, it traditionally requires visibility of the resource environment. Objective: In this study, we investigate whether performance tests of serverless applications are stable, that is, if their results are reproducible, and what implications the serverless paradigm has for performance tests. Method: We conduct a case study where we collect two datasets of performance test results: (a) repetitions of performance tests for varying memory size and load intensities and (b) three repetitions of the same performance test every day for ten months. Results: We find that performance tests of serverless applications are comparatively stable if conducted on the same day. However, we also observe short-term performance variations and frequent long-term performance changes. Conclusion: Performance tests for serverless applications can be stable; however, the serverless model impacts the planning, execution, and analysis of performance tests.

• 
  A Machine Learning-based Workflow for Automatic Detection of Anomalies in Machine Tools Züfle, Marwin; Moog, Felix; Lesch, Veronika; Krupitzer, Christian; Kounev, Samuel; in ISA Transactions: The Journal of Automation (2021).

  (in press)
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  Despite the increased sensor-based data collection in Industry 4.0, the practical use of this data is still in its infancy. In contrast, academic literature provides several approaches to detect machine failures but, in most cases, relies on simulations and vast amounts of training data. Since it is often not practical to collect such amounts of data in an industrial context, we propose an approach to detect the current production mode and machine degradation states on a comparably small data set. Our approach integrates domain knowledge about manufacturing systems into a highly generalizable end-to-end workflow ranging from raw data processing, phase segmentation, data resampling, and feature extraction to machine tool anomaly detection. The workflow applies unsupervised clustering techniques to identify the current production mode and supervised classification models for detecting the present degradation. A resampling strategy and classical machine learning models enable the workflow to handle small data sets and distinguish between normal and abnormal machine tool behavior. To the best of our knowledge, there exists no such end-to-end workflow in the literature that uses the entire machine signal as input to identify anomalies for individual tools. Our evaluation with data from a real multi-purpose machine shows that the proposed workflow detects anomalies with an average F1-score of almost 93%.

  @article{zuefle2021anomalydetection, abstract = {Despite the increased sensor-based data collection in Industry 4.0, the practical use of this data is still in its infancy. In contrast, academic literature provides several approaches to detect machine failures but, in most cases, relies on simulations and vast amounts of training data. Since it is often not practical to collect such amounts of data in an industrial context, we propose an approach to detect the current production mode and machine degradation states on a comparably small data set. Our approach integrates domain knowledge about manufacturing systems into a highly generalizable end-to-end workflow ranging from raw data processing, phase segmentation, data resampling, and feature extraction to machine tool anomaly detection. The workflow applies unsupervised clustering techniques to identify the current production mode and supervised classification models for detecting the present degradation. A resampling strategy and classical machine learning models enable the workflow to handle small data sets and distinguish between normal and abnormal machine tool behavior. To the best of our knowledge, there exists no such end-to-end workflow in the literature that uses the entire machine signal as input to identify anomalies for individual tools. Our evaluation with data from a real multi-purpose machine shows that the proposed workflow detects anomalies with an average F1-score of almost 93%.}, author = {Züfle, Marwin and Moog, Felix and Lesch, Veronika and Krupitzer, Christian and Kounev, Samuel}, journal = {ISA Transactions: The Journal of Automation}, keywords = {myown}, note = {(in press)}, title = {A Machine Learning-based Workflow for Automatic Detection of Anomalies in Machine Tools}, year = 2021 }

  %0 Journal Article %1 zuefle2021anomalydetection %A Züfle, Marwin %A Moog, Felix %A Lesch, Veronika %A Krupitzer, Christian %A Kounev, Samuel %D 2021 %J ISA Transactions: The Journal of Automation %T A Machine Learning-based Workflow for Automatic Detection of Anomalies in Machine Tools %U https://www.sciencedirect.com/science/article/pii/S0019057821003773 %X Despite the increased sensor-based data collection in Industry 4.0, the practical use of this data is still in its infancy. In contrast, academic literature provides several approaches to detect machine failures but, in most cases, relies on simulations and vast amounts of training data. Since it is often not practical to collect such amounts of data in an industrial context, we propose an approach to detect the current production mode and machine degradation states on a comparably small data set. Our approach integrates domain knowledge about manufacturing systems into a highly generalizable end-to-end workflow ranging from raw data processing, phase segmentation, data resampling, and feature extraction to machine tool anomaly detection. The workflow applies unsupervised clustering techniques to identify the current production mode and supervised classification models for detecting the present degradation. A resampling strategy and classical machine learning models enable the workflow to handle small data sets and distinguish between normal and abnormal machine tool behavior. To the best of our knowledge, there exists no such end-to-end workflow in the literature that uses the entire machine signal as input to identify anomalies for individual tools. Our evaluation with data from a real multi-purpose machine shows that the proposed workflow detects anomalies with an average F1-score of almost 93%.
• 
  A Comparison of Mechanisms for Compensating Negative Impacts of System Integration Lesch, Veronika; Krupitzer, Christian; Stubenrauch, Kevin; Keil, Nico; Becker, Christian; Kounev, Samuel; Segata, Michele; in Future Generation Computer Systems (2021). 116 117–131.
  • [ BibTeX ]
  • [ EndNote ]
  • [ DOI ]
  • [ Download ]
  @article{lesch2020comparison, author = {Lesch, Veronika and Krupitzer, Christian and Stubenrauch, Kevin and Keil, Nico and Becker, Christian and Kounev, Samuel and Segata, Michele}, journal = {Future Generation Computer Systems}, keywords = {myown}, month = {March}, pages = {117-131}, title = {A Comparison of Mechanisms for Compensating Negative Impacts of System Integration}, volume = 116, year = 2021 }

  %0 Journal Article %1 lesch2020comparison %A Lesch, Veronika %A Krupitzer, Christian %A Stubenrauch, Kevin %A Keil, Nico %A Becker, Christian %A Kounev, Samuel %A Segata, Michele %D 2021 %J Future Generation Computer Systems %P 117-131 %R https://doi.org/10.1016/j.future.2020.10.029 %T A Comparison of Mechanisms for Compensating Negative Impacts of System Integration %V 116
• 
  SARDE: A Framework for Continuous and Self-Adaptive Resource Demand Estimation Grohmann, Johannes; Eismann, Simon; Bauer, André; Spinner, Simon; Blum, Johannes; Herbst, Nikolas; Kounev, Samuel; in ACM Transactions on Autonomous and Adaptive Systems (2021). 15(2) Association for Computing Machinery, New York, NY, USA.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Resource demands are crucial parameters for modeling and predicting the performance of software systems. Currently, resource demand estimators are usually executed once for system analysis. However, the monitored system, as well as the resource demand itself, are subject to constant change in runtime environments. These changes additionally impact the applicability, the required parametrization as well as the resulting accuracy of individual estimation approaches. Over time, this leads to invalid or outdated estimates, which in turn negatively influence the decision-making of adaptive systems. In this article, we present SARDE, a framework for self-adaptive resource demand estimation in continuous environments. SARDE dynamically and continuously tunes, selects, and executes an ensemble of resource demand estimation approaches to adapt to changes in the environment. This creates an autonomous and unsupervised ensemble estimation technique, providing reliable resource demand estimations in dynamic environments. We evaluate SARDE using two realistic datasets. One set of different micro-benchmarks reflecting different possible system states and one dataset consisting of a continuously running application in a changing environment. Our results show that by continuously applying online optimization, selection and estimation, SARDE is able to efficiently adapt to the online trace and reduce the model error using the resulting ensemble technique.

  @article{GrEiBaSpBlHeKo2021-TaaS-SARDE, abstract = {Resource demands are crucial parameters for modeling and predicting the performance of software systems. Currently, resource demand estimators are usually executed once for system analysis. However, the monitored system, as well as the resource demand itself, are subject to constant change in runtime environments. These changes additionally impact the applicability, the required parametrization as well as the resulting accuracy of individual estimation approaches. Over time, this leads to invalid or outdated estimates, which in turn negatively influence the decision-making of adaptive systems. In this article, we present SARDE, a framework for self-adaptive resource demand estimation in continuous environments. SARDE dynamically and continuously tunes, selects, and executes an ensemble of resource demand estimation approaches to adapt to changes in the environment. This creates an autonomous and unsupervised ensemble estimation technique, providing reliable resource demand estimations in dynamic environments. We evaluate SARDE using two realistic datasets. One set of different micro-benchmarks reflecting different possible system states and one dataset consisting of a continuously running application in a changing environment. Our results show that by continuously applying online optimization, selection and estimation, SARDE is able to efficiently adapt to the online trace and reduce the model error using the resulting ensemble technique.}, address = {New York, NY, USA}, author = {Grohmann, Johannes and Eismann, Simon and Bauer, Andr\'{e} and Spinner, Simon and Blum, Johannes and Herbst, Nikolas and Kounev, Samuel}, journal = {ACM Transactions on Autonomous and Adaptive Systems}, keywords = {myown}, month = {jun}, number = 2, publisher = {Association for Computing Machinery}, title = {SARDE: A Framework for Continuous and Self-Adaptive Resource Demand Estimation}, volume = 15, year = 2021 }

  %0 Journal Article %1 GrEiBaSpBlHeKo2021-TaaS-SARDE %A Grohmann, Johannes %A Eismann, Simon %A Bauer, André %A Spinner, Simon %A Blum, Johannes %A Herbst, Nikolas %A Kounev, Samuel %C New York, NY, USA %D 2021 %I Association for Computing Machinery %J ACM Transactions on Autonomous and Adaptive Systems %N 2 %T SARDE: A Framework for Continuous and Self-Adaptive Resource Demand Estimation %U https://doi.org/10.1145/3463369 %V 15 %X Resource demands are crucial parameters for modeling and predicting the performance of software systems. Currently, resource demand estimators are usually executed once for system analysis. However, the monitored system, as well as the resource demand itself, are subject to constant change in runtime environments. These changes additionally impact the applicability, the required parametrization as well as the resulting accuracy of individual estimation approaches. Over time, this leads to invalid or outdated estimates, which in turn negatively influence the decision-making of adaptive systems. In this article, we present SARDE, a framework for self-adaptive resource demand estimation in continuous environments. SARDE dynamically and continuously tunes, selects, and executes an ensemble of resource demand estimation approaches to adapt to changes in the environment. This creates an autonomous and unsupervised ensemble estimation technique, providing reliable resource demand estimations in dynamic environments. We evaluate SARDE using two realistic datasets. One set of different micro-benchmarks reflecting different possible system states and one dataset consisting of a continuously running application in a changing environment. Our results show that by continuously applying online optimization, selection and estimation, SARDE is able to efficiently adapt to the online trace and reduce the model error using the resulting ensemble technique.
• 
  Tackling the Rich Vehicle Routing Problem with Nature-Inspired Algorithms Lesch, Veronika; König, Maximilian; Kounev, Samuel; Stein, Anthony; Krupitzer, Christian; in Applied Intelligence (2021). 52 9476–9500.
  • [ BibTeX ]
  • [ EndNote ]
  • [ DOI ]
  @article{lesch2021tackling, author = {Lesch, Veronika and König, Maximilian and Kounev, Samuel and Stein, Anthony and Krupitzer, Christian}, journal = {Applied Intelligence}, keywords = {myown}, pages = {9476–9500}, title = {Tackling the Rich Vehicle Routing Problem with Nature-Inspired Algorithms}, volume = 52, year = 2021 }

  %0 Journal Article %1 lesch2021tackling %A Lesch, Veronika %A König, Maximilian %A Kounev, Samuel %A Stein, Anthony %A Krupitzer, Christian %D 2021 %J Applied Intelligence %P 9476–9500 %R https://doi.org/10.1007/s10489-021-03035-5 %T Tackling the Rich Vehicle Routing Problem with Nature-Inspired Algorithms %V 52
• 
  An Overview on Approaches for Coordination of Platoons Lesch, Veronika; Breitbach, Martin; Segata, Michele; Becker, Christian; Kounev, Samuel; Krupitzer, Christian; in IEEE Transactions on Intelligent Transportation Systems (2021).

  Early Access on IEEE Xplore
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @article{lesch2021overview, author = {Lesch, Veronika and Breitbach, Martin and Segata, Michele and Becker, Christian and Kounev, Samuel and Krupitzer, Christian}, journal = {IEEE Transactions on Intelligent Transportation Systems}, keywords = {myown}, note = {Early Access on IEEE Xplore}, title = {An Overview on Approaches for Coordination of Platoons}, year = 2021 }

  %0 Journal Article %1 lesch2021overview %A Lesch, Veronika %A Breitbach, Martin %A Segata, Michele %A Becker, Christian %A Kounev, Samuel %A Krupitzer, Christian %D 2021 %J IEEE Transactions on Intelligent Transportation Systems %T An Overview on Approaches for Coordination of Platoons
• 
  People Detection with Depth Silhouettes and Convolutional Neural Networks on a Mobile Robot Spiess, Florian; Reinhart, Lucas; Strobel, Norbert; Kaiser, Dennis; Kounev, Samuel; Kaupp, Tobias; in Journal of Image and Graphics, (J. of Image; Graphics, eds.) (2021). 9(4) 135–139.
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  @article{spiess2021people, author = {Spiess, Florian and Reinhart, Lucas and Strobel, Norbert and Kaiser, Dennis and Kounev, Samuel and Kaupp, Tobias}, editor = {of Image, Journal and Graphics}, journal = {Journal of Image and Graphics}, keywords = {myown}, month = {December 2021}, number = 4, pages = {135-139}, title = {People Detection with Depth Silhouettes and Convolutional Neural Networks on a Mobile Robot}, volume = 9, year = 2021 }

  %0 Journal Article %1 spiess2021people %A Spiess, Florian %A Reinhart, Lucas %A Strobel, Norbert %A Kaiser, Dennis %A Kounev, Samuel %A Kaupp, Tobias %D 2021 %E of Image, Journal %E Graphics, %J Journal of Image and Graphics %N 4 %P 135-139 %T People Detection with Depth Silhouettes and Convolutional Neural Networks on a Mobile Robot %U http://www.joig.net/index.php?m=content&c=index&a=show&catid=72&id=277 %V 9
• 
  A Case Study on Optimization of Warehouses Lesch, Veronika; Müller, Patrick; Krämer, Moritz; Kounev, Samuel; Krupitzer, Christian; in arXiv preprint arXiv:2112.12058 (2021).
  • [ BibTeX ]
  • [ EndNote ]
  @article{lesch2021case, author = {Lesch, Veronika and M{\"u}ller, Patrick and Kr{\"a}mer, Moritz and Kounev, Samuel and Krupitzer, Christian}, journal = {arXiv preprint arXiv:2112.12058}, keywords = {myown}, title = {A Case Study on Optimization of Warehouses}, year = 2021 }

  %0 Journal Article %1 lesch2021case %A Lesch, Veronika %A Müller, Patrick %A Krämer, Moritz %A Kounev, Samuel %A Krupitzer, Christian %D 2021 %J arXiv preprint arXiv:2112.12058 %T A Case Study on Optimization of Warehouses

• 
  Survey and Experimental Comparison of RGB-D Indoor Robot Navigation Methods Supported by ROS and Their Expansion via Fusion with Wheel Odometry and IMU Data Spiess, Florian; Friesslich, Jonas; Tobias, Kaupp; Kounev, Samuel; Strobel, Norbert; in International Journal of Mechanical Engineering and Robotics Research (IJMERR), (P. R. (Chunhui) Yang, ed.) (2020). 9(12) 1532–1540.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  This paper presents an experimental evaluation and comparison of selected Visual Odometry (VO) and Visual-SLAM (V-SLAM) algorithms for indoor mobile robot navigation supported by the Robot Operating System (ROS). The focus is on algorithms involving RGB-D cameras. Since RGB-D cameras integrate color and depth information, they output coherent measurement data and facilitate an efficient processing pipeline. The various underlying methods of vision-based algorithms are described and evaluated on two datasets covering different indoor situations as well as various lighting and movement conditions. In general, V-SLAM algorithms yielded better results. They were superior with respect to handling drift, in particular when loop closures were involved. However, the results confirmed that VO algorithms could outperform V-SLAM methods under certain circumstances. This happened when there was a very good match between an algorithm’s design objectives and the situation at hand. While the experiments showed that there is no single best algorithm for every scenario, ORB-SLAM2 is recommended as a robust stand-alone RGB-D based localization method available under ROS. Furthermore, we observed that the position estimation error could be reduced by around 67% on average when combining vision-based position estimates with sensor data obtained from wheel odometry and an inertial measurement unit (IMU), respectively. This clearly demonstrates the potential of sensor fusion techniques. The best results in case of sensor fusion were obtained with RGB-DSLAMv2.

  @article{yang2020survey, abstract = {This paper presents an experimental evaluation and comparison of selected Visual Odometry (VO) and Visual-SLAM (V-SLAM) algorithms for indoor mobile robot navigation supported by the Robot Operating System (ROS). The focus is on algorithms involving RGB-D cameras. Since RGB-D cameras integrate color and depth information, they output coherent measurement data and facilitate an efficient processing pipeline. The various underlying methods of vision-based algorithms are described and evaluated on two datasets covering different indoor situations as well as various lighting and movement conditions. In general, V-SLAM algorithms yielded better results. They were superior with respect to handling drift, in particular when loop closures were involved. However, the results confirmed that VO algorithms could outperform V-SLAM methods under certain circumstances. This happened when there was a very good match between an algorithm’s design objectives and the situation at hand. While the experiments showed that there is no single best algorithm for every scenario, ORB-SLAM2 is recommended as a robust stand-alone RGB-D based localization method available under ROS. Furthermore, we observed that the position estimation error could be reduced by around 67% on average when combining vision-based position estimates with sensor data obtained from wheel odometry and an inertial measurement unit (IMU), respectively. This clearly demonstrates the potential of sensor fusion techniques. The best results in case of sensor fusion were obtained with RGB-DSLAMv2.}, author = {Spiess, Florian and Friesslich, Jonas and Tobias, Kaupp and Kounev, Samuel and Strobel, Norbert}, editor = {Yang, Prof Richard (Chunhui)}, journal = {International Journal of Mechanical Engineering and Robotics Research (IJMERR)}, keywords = {myown}, month = {dec}, number = 12, pages = {1532-1540}, title = {Survey and Experimental Comparison of RGB-D Indoor Robot Navigation Methods Supported by ROS and Their Expansion via Fusion with Wheel Odometry and IMU Data}, volume = 9, year = 2020 }

  %0 Journal Article %1 yang2020survey %A Spiess, Florian %A Friesslich, Jonas %A Tobias, Kaupp %A Kounev, Samuel %A Strobel, Norbert %D 2020 %E Yang, Prof Richard (Chunhui) %J International Journal of Mechanical Engineering and Robotics Research (IJMERR) %N 12 %P 1532-1540 %T Survey and Experimental Comparison of RGB-D Indoor Robot Navigation Methods Supported by ROS and Their Expansion via Fusion with Wheel Odometry and IMU Data %U http://www.ijmerr.com/index.php?m=content&c=index&a=show&catid=185&id=1529 %V 9 %X This paper presents an experimental evaluation and comparison of selected Visual Odometry (VO) and Visual-SLAM (V-SLAM) algorithms for indoor mobile robot navigation supported by the Robot Operating System (ROS). The focus is on algorithms involving RGB-D cameras. Since RGB-D cameras integrate color and depth information, they output coherent measurement data and facilitate an efficient processing pipeline. The various underlying methods of vision-based algorithms are described and evaluated on two datasets covering different indoor situations as well as various lighting and movement conditions. In general, V-SLAM algorithms yielded better results. They were superior with respect to handling drift, in particular when loop closures were involved. However, the results confirmed that VO algorithms could outperform V-SLAM methods under certain circumstances. This happened when there was a very good match between an algorithm’s design objectives and the situation at hand. While the experiments showed that there is no single best algorithm for every scenario, ORB-SLAM2 is recommended as a robust stand-alone RGB-D based localization method available under ROS. Furthermore, we observed that the position estimation error could be reduced by around 67% on average when combining vision-based position estimates with sensor data obtained from wheel odometry and an inertial measurement unit (IMU), respectively. This clearly demonstrates the potential of sensor fusion techniques. The best results in case of sensor fusion were obtained with RGB-DSLAMv2.
• 
  The impact of the German strategy for containment of Coronavirus SARS-CoV-2 on the training characteristics, physical activity, sleep of highly trained kayakers and canoeists: A retrospective observational study Zinner, Christoph; Matzka, Manuel; Leppich, Robert; Kounev, Samuel; Holmberg, HC; Sperlich, Billy; in Frontiers in Sports and Active Living (2020). 2 127. Frontiers.
  • [ BibTeX ]
  • [ EndNote ]
  @article{zinner2020impact, author = {Zinner, Christoph and Matzka, Manuel and Leppich, Robert and Kounev, Samuel and Holmberg, HC and Sperlich, Billy}, journal = {Frontiers in Sports and Active Living}, keywords = {t_journalmagazine}, pages = 127, publisher = {Frontiers}, title = {The impact of the German strategy for containment of Coronavirus SARS-CoV-2 on the training characteristics, physical activity, sleep of highly trained kayakers and canoeists: A retrospective observational study}, volume = 2, year = 2020 }

  %0 Journal Article %1 zinner2020impact %A Zinner, Christoph %A Matzka, Manuel %A Leppich, Robert %A Kounev, Samuel %A Holmberg, HC %A Sperlich, Billy %D 2020 %I Frontiers %J Frontiers in Sports and Active Living %P 127 %T The impact of the German strategy for containment of Coronavirus SARS-CoV-2 on the training characteristics, physical activity, sleep of highly trained kayakers and canoeists: A retrospective observational study %V 2
• 
  Time Series Forecasting for Self-Aware Systems Bauer, André; Züfle, Marwin; Herbst, Nikolas; Zehe, Albin; Hotho, Andreas; Kounev, Samuel; in Proceedings of the IEEE (2020). 108(7) 1068–1093.
  • [ BibTeX ]
  • [ EndNote ]
  • [ DOI ]
  @article{BaZuHeZeHoKo-PIEEE-Forecasting, author = {Bauer, Andr{\'e} and Z{\"u}fle, Marwin and Herbst, Nikolas and Zehe, Albin and Hotho, Andreas and Kounev, Samuel}, journal = {Proceedings of the IEEE}, keywords = {myown}, month = 7, number = 7, pages = {1068 - 1093}, title = {{Time Series Forecasting for Self-Aware Systems}}, volume = 108, year = 2020 }

  %0 Journal Article %1 BaZuHeZeHoKo-PIEEE-Forecasting %A Bauer, André %A Züfle, Marwin %A Herbst, Nikolas %A Zehe, Albin %A Hotho, Andreas %A Kounev, Samuel %D 2020 %J Proceedings of the IEEE %N 7 %P 1068 - 1093 %R 10.1109/JPROC.2020.2983857 %T Time Series Forecasting for Self-Aware Systems %V 108
• 
  A Taxonomy of Techniques for SLO Failure Prediction in Software Systems Grohmann, Johannes; Herbst, Nikolas; Chalbani, Avi; Arian, Yair; Peretz, Noam; Kounev, Samuel; in Computers (2020). 9(1) 10. Multidisciplinary Digital Publishing Institute (MDPI).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  Failure prediction is an important aspect of self-aware computing systems. Therefore, a multitude of different approaches has been proposed in the literature over the past few years. In this work, we propose a taxonomy for organizing works focusing on the prediction of Service Level Objective (SLO) failures. Our taxonomy classifies related work along the dimensions of the prediction target (e.g., anomaly detection, performance prediction, or failure prediction), the time horizon (e.g., detection or prediction, online or offline application), and the applied modeling type (e.g., time series forecasting, machine learning, or queueing theory). The classification is derived based on a systematic mapping of relevant papers in the area. Additionally, we give an overview of different techniques in each sub-group and address remaining challenges in order to guide future research.

  @article{grohmann2020-MDPI-taxonomy, abstract = {Failure prediction is an important aspect of self-aware computing systems. Therefore, a multitude of different approaches has been proposed in the literature over the past few years. In this work, we propose a taxonomy for organizing works focusing on the prediction of Service Level Objective (SLO) failures. Our taxonomy classifies related work along the dimensions of the prediction target (e.g., anomaly detection, performance prediction, or failure prediction), the time horizon (e.g., detection or prediction, online or offline application), and the applied modeling type (e.g., time series forecasting, machine learning, or queueing theory). The classification is derived based on a systematic mapping of relevant papers in the area. Additionally, we give an overview of different techniques in each sub-group and address remaining challenges in order to guide future research.}, author = {Grohmann, Johannes and Herbst, Nikolas and Chalbani, Avi and Arian, Yair and Peretz, Noam and Kounev, Samuel}, journal = {Computers}, keywords = {myown}, number = 1, pages = 10, publisher = {Multidisciplinary Digital Publishing Institute (MDPI)}, title = {A Taxonomy of Techniques for SLO Failure Prediction in Software Systems}, volume = 9, year = 2020 }

  %0 Journal Article %1 grohmann2020-MDPI-taxonomy %A Grohmann, Johannes %A Herbst, Nikolas %A Chalbani, Avi %A Arian, Yair %A Peretz, Noam %A Kounev, Samuel %D 2020 %I Multidisciplinary Digital Publishing Institute (MDPI) %J Computers %N 1 %P 10 %R https://doi.org/10.3390/computers9010010 %T A Taxonomy of Techniques for SLO Failure Prediction in Software Systems %U https://doi.org/10.3390/computers9010010 %V 9 %X Failure prediction is an important aspect of self-aware computing systems. Therefore, a multitude of different approaches has been proposed in the literature over the past few years. In this work, we propose a taxonomy for organizing works focusing on the prediction of Service Level Objective (SLO) failures. Our taxonomy classifies related work along the dimensions of the prediction target (e.g., anomaly detection, performance prediction, or failure prediction), the time horizon (e.g., detection or prediction, online or offline application), and the applied modeling type (e.g., time series forecasting, machine learning, or queueing theory). The classification is derived based on a systematic mapping of relevant papers in the area. Additionally, we give an overview of different techniques in each sub-group and address remaining challenges in order to guide future research.
• 
  Towards Self-Aware Multirotor Formations Kaiser, Dennis; Lesch, Veronika; Rothe, Julian; Strohmeier, Michael; Spiess, Florian; Krupitzer, Christian; Montenegro, Sergio; Kounev, Samuel; in Computers (2020). 9(7) MDPI.

  Special Issue on Self-Aware Computing
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @article{Kaiser2020Towards, author = {Kaiser, Dennis and Lesch, Veronika and Rothe, Julian and Strohmeier, Michael and Spiess, Florian and Krupitzer, Christian and Montenegro, Sergio and Kounev, Samuel}, journal = {Computers}, keywords = {myown}, month = {Februar}, note = {Special Issue on Self-Aware Computing}, number = 7, publisher = {MDPI}, title = {{Towards Self-Aware Multirotor Formations}}, volume = 9, year = 2020 }

  %0 Journal Article %1 Kaiser2020Towards %A Kaiser, Dennis %A Lesch, Veronika %A Rothe, Julian %A Strohmeier, Michael %A Spiess, Florian %A Krupitzer, Christian %A Montenegro, Sergio %A Kounev, Samuel %D 2020 %I MDPI %J Computers %N 7 %T Towards Self-Aware Multirotor Formations %V 9

• 
  Measuring and Rating the Energy-Efficiency of Servers von Kistowski, Jóakim; Lange, Klaus-Dieter; Arnold, Jeremy A.; Beckett, John; Block, Hansfried; Tricker, Mike; Sharma, Sanjay; Pais, Johann; Kounev, Samuel; in Future Generation Computer Systems (2019). 100 579–589.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  Data centers and servers consume a significant amount of electrical power. As future data centers increase in size, it becomes increasingly important to be able to select servers based on their energy efficiency. Rating the efficiency of servers is challenging, as it depends on the software executed on that server and on its load profile. To account for this, a measurement and rating methodology for servers must make use of both realistic and varying workloads. Existing energy-efficiency benchmarks either run standardized application loads at multiple load levels or multiple micro-benchmarks at full load. This does not enable a full analysis of system behavior as the energy efficiency for different kinds of software at low load levels remains unknown. This article introduces a measurement methodology and metrics for energy-efficiency rating of servers. The methodology defines system setup and instrumentation for reproducible results. It is designed to use multiple, specifically chosen workloads at different load levels for a full system characterization. All partial results are aggregated in a final metric that can be used by regulators to define server-labeling criteria. The methodology and workloads have been implemented in the standardized Server Efficiency Rating Tool. We show the applicability and use of the measurement methodology specifically considering its reproducibility, fairness, and relevance. A reproducibility analysis shows that efficiency scores vary with a maximum coefficient of 1.04% for repeated experiments. In addition, we evaluate the proposed metrics by investigating their energy-efficiency rating on a set of 385 different servers and show the relevance of the selected workloads by analyzing relationships to real-world applications.

  @article{KiLaArBeBlTrShPaKo-FGCS2019-ServerEfficiency, abstract = {Data centers and servers consume a significant amount of electrical power. As future data centers increase in size, it becomes increasingly important to be able to select servers based on their energy efficiency. Rating the efficiency of servers is challenging, as it depends on the software executed on that server and on its load profile. To account for this, a measurement and rating methodology for servers must make use of both realistic and varying workloads. Existing energy-efficiency benchmarks either run standardized application loads at multiple load levels or multiple micro-benchmarks at full load. This does not enable a full analysis of system behavior as the energy efficiency for different kinds of software at low load levels remains unknown. This article introduces a measurement methodology and metrics for energy-efficiency rating of servers. The methodology defines system setup and instrumentation for reproducible results. It is designed to use multiple, specifically chosen workloads at different load levels for a full system characterization. All partial results are aggregated in a final metric that can be used by regulators to define server-labeling criteria. The methodology and workloads have been implemented in the standardized Server Efficiency Rating Tool. We show the applicability and use of the measurement methodology specifically considering its reproducibility, fairness, and relevance. A reproducibility analysis shows that efficiency scores vary with a maximum coefficient of 1.04\% for repeated experiments. In addition, we evaluate the proposed metrics by investigating their energy-efficiency rating on a set of 385 different servers and show the relevance of the selected workloads by analyzing relationships to real-world applications.}, author = {von Kistowski, J{\'o}akim and Lange, Klaus-Dieter and Arnold, Jeremy A. and Beckett, John and Block, Hansfried and Tricker, Mike and Sharma, Sanjay and Pais, Johann and Kounev, Samuel}, journal = {Future Generation Computer Systems}, keywords = {myown}, pages = {579 - 589}, title = {{Measuring and Rating the Energy-Efficiency of Servers}}, volume = 100, year = 2019 }

  %0 Journal Article %1 KiLaArBeBlTrShPaKo-FGCS2019-ServerEfficiency %A von Kistowski, Jóakim %A Lange, Klaus-Dieter %A Arnold, Jeremy A. %A Beckett, John %A Block, Hansfried %A Tricker, Mike %A Sharma, Sanjay %A Pais, Johann %A Kounev, Samuel %D 2019 %J Future Generation Computer Systems %P 579 - 589 %T Measuring and Rating the Energy-Efficiency of Servers %U http://www.sciencedirect.com/science/article/pii/S0167739X18312123 %V 100 %X Data centers and servers consume a significant amount of electrical power. As future data centers increase in size, it becomes increasingly important to be able to select servers based on their energy efficiency. Rating the efficiency of servers is challenging, as it depends on the software executed on that server and on its load profile. To account for this, a measurement and rating methodology for servers must make use of both realistic and varying workloads. Existing energy-efficiency benchmarks either run standardized application loads at multiple load levels or multiple micro-benchmarks at full load. This does not enable a full analysis of system behavior as the energy efficiency for different kinds of software at low load levels remains unknown. This article introduces a measurement methodology and metrics for energy-efficiency rating of servers. The methodology defines system setup and instrumentation for reproducible results. It is designed to use multiple, specifically chosen workloads at different load levels for a full system characterization. All partial results are aggregated in a final metric that can be used by regulators to define server-labeling criteria. The methodology and workloads have been implemented in the standardized Server Efficiency Rating Tool. We show the applicability and use of the measurement methodology specifically considering its reproducibility, fairness, and relevance. A reproducibility analysis shows that efficiency scores vary with a maximum coefficient of 1.04% for repeated experiments. In addition, we evaluate the proposed metrics by investigating their energy-efficiency rating on a set of 385 different servers and show the relevance of the selected workloads by analyzing relationships to real-world applications.
• 
  Modeling of Aggregated IoT Traffic and Its Application to an IoT Cloud Metzger, Florian; Hoßfeld, Tobias; Bauer, André; Kounev, Samuel; Heegaard, Poul. E.; in Proceedings of the IEEE (2019). 107(4) 679–694.
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @article{MeHoBaKoHe2019-PIEEE-IoTTraffic, author = {Metzger, Florian and Ho{\ss}feld, Tobias and Bauer, Andr{\'e} and Kounev, Samuel and Heegaard, Poul. E.}, journal = {Proceedings of the IEEE}, keywords = {myown}, month = {April}, number = 4, pages = {679-694}, title = {{Modeling of Aggregated IoT Traffic and Its Application to an IoT Cloud}}, volume = 107, year = 2019 }

  %0 Journal Article %1 MeHoBaKoHe2019-PIEEE-IoTTraffic %A Metzger, Florian %A Hoßfeld, Tobias %A Bauer, André %A Kounev, Samuel %A Heegaard, Poul. E. %D 2019 %J Proceedings of the IEEE %N 4 %P 679-694 %T Modeling of Aggregated IoT Traffic and Its Application to an IoT Cloud %V 107
• 
  Online model learning for self-aware computing infrastructures Spinner, Simon; Grohmann, Johannes; Eismann, Simon; Kounev, Samuel; in Journal of Systems and Software (2019). 147 1–16.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  • [ Download ]
  Performance models are valuable and powerful tools for performance prediction. However, the creation of performance models usually requires significant manual effort. Furthermore, as the modeled structures are subject to frequent change in modern infrastructures, such performance models need to be adapted as well. We therefore propose a reference architecture for online model learning in virtualized environments, which enables the automatic extraction of the aforementioned performance models. We follow an agent-based approach, which enables us to incorporate the extraction of information about the application structure as well as the virtualization structures present in modern computing centers. Our evaluation shows that our collaborating agents are able to reduce the manual effort of performance model extraction by 85.4%. The resulting performance model is able to predict the system utilization with an absolute error of less than 4% and the end-to-end response time with a relative error of less than 21%.

  @article{SpGrEiKo2019-JSS-ModelLearning, abstract = {Performance models are valuable and powerful tools for performance prediction. However, the creation of performance models usually requires significant manual effort. Furthermore, as the modeled structures are subject to frequent change in modern infrastructures, such performance models need to be adapted as well. We therefore propose a reference architecture for online model learning in virtualized environments, which enables the automatic extraction of the aforementioned performance models. We follow an agent-based approach, which enables us to incorporate the extraction of information about the application structure as well as the virtualization structures present in modern computing centers. Our evaluation shows that our collaborating agents are able to reduce the manual effort of performance model extraction by 85.4%. The resulting performance model is able to predict the system utilization with an absolute error of less than 4% and the end-to-end response time with a relative error of less than 21%.}, author = {Spinner, Simon and Grohmann, Johannes and Eismann, Simon and Kounev, Samuel}, journal = {Journal of Systems and Software}, keywords = {myown}, pages = {1–16}, title = {Online model learning for self-aware computing infrastructures}, volume = 147, year = 2019 }

  %0 Journal Article %1 SpGrEiKo2019-JSS-ModelLearning %A Spinner, Simon %A Grohmann, Johannes %A Eismann, Simon %A Kounev, Samuel %D 2019 %J Journal of Systems and Software %P 1--16 %R 10.1016/j.jss.2018.09.089 %T Online model learning for self-aware computing infrastructures %U https://doi.org/10.1016/j.jss.2018.09.089 %V 147 %X Performance models are valuable and powerful tools for performance prediction. However, the creation of performance models usually requires significant manual effort. Furthermore, as the modeled structures are subject to frequent change in modern infrastructures, such performance models need to be adapted as well. We therefore propose a reference architecture for online model learning in virtualized environments, which enables the automatic extraction of the aforementioned performance models. We follow an agent-based approach, which enables us to incorporate the extraction of information about the application structure as well as the virtualization structures present in modern computing centers. Our evaluation shows that our collaborating agents are able to reduce the manual effort of performance model extraction by 85.4%. The resulting performance model is able to predict the system utilization with an absolute error of less than 4% and the end-to-end response time with a relative error of less than 21%.
• 
  Chameleon: A Hybrid, Proactive Auto-Scaling Mechanism on a Level-Playing Field Bauer, André; Herbst, Nikolas; Spinner, Simon; Ali-Eldin, Ahmed; Kounev, Samuel; in IEEE Transactions on Parallel and Distributed Systems (2019). 30(4) 800–813. IEEE.
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @article{BaHeSpAEKo2019-TPDS-ChameleonAutoscaler, author = {Bauer, Andr{\'e} and Herbst, Nikolas and Spinner, Simon and Ali-Eldin, Ahmed and Kounev, Samuel}, journal = {IEEE Transactions on Parallel and Distributed Systems}, keywords = {myown}, month = {April}, number = 4, pages = {800–813}, publisher = {IEEE}, title = {{Chameleon: A Hybrid, Proactive Auto-Scaling Mechanism on a Level-Playing Field}}, volume = 30, year = 2019 }

  %0 Journal Article %1 BaHeSpAEKo2019-TPDS-ChameleonAutoscaler %A Bauer, André %A Herbst, Nikolas %A Spinner, Simon %A Ali-Eldin, Ahmed %A Kounev, Samuel %D 2019 %I IEEE %J IEEE Transactions on Parallel and Distributed Systems %N 4 %P 800--813 %T Chameleon: A Hybrid, Proactive Auto-Scaling Mechanism on a Level-Playing Field %V 30

• 
  Quantifying Cloud Performance and Dependability: Taxonomy, Metric Design, and Emerging Challenges Herbst, Nikolas; Bauer, André; Kounev, Samuel; Oikonomou, Giorgos; van Eyk, Erwin; Kousiouris, George; Evangelinou, Athanasia; Krebs, Rouven; Brecht, Tim; Abad, Cristina L.; Iosup, Alexandru; in ACM Transactions on Modeling and Performance Evaluation of Computing Systems (ToMPECS) (2018). 3(4) 19:1–19:36. ACM, New York, NY, USA.
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  @article{HeBaKoOiEyKoEvKrBrAbIo2018-TOMPECS-CloudMetrics, address = {New York, NY, USA}, author = {Herbst, Nikolas and Bauer, Andr{\'e} and Kounev, Samuel and Oikonomou, Giorgos and van Eyk, Erwin and Kousiouris, George and Evangelinou, Athanasia and Krebs, Rouven and Brecht, Tim and Abad, Cristina L. and Iosup, Alexandru}, journal = {ACM Transactions on Modeling and Performance Evaluation of Computing Systems (ToMPECS)}, keywords = {t_journalmagazine}, month = 8, number = 4, pages = {19:1–19:36}, publisher = {ACM}, title = {{Quantifying Cloud Performance and Dependability: Taxonomy, Metric Design, and Emerging Challenges}}, volume = 3, year = 2018 }

  %0 Journal Article %1 HeBaKoOiEyKoEvKrBrAbIo2018-TOMPECS-CloudMetrics %A Herbst, Nikolas %A Bauer, André %A Kounev, Samuel %A Oikonomou, Giorgos %A van Eyk, Erwin %A Kousiouris, George %A Evangelinou, Athanasia %A Krebs, Rouven %A Brecht, Tim %A Abad, Cristina L. %A Iosup, Alexandru %C New York, NY, USA %D 2018 %I ACM %J ACM Transactions on Modeling and Performance Evaluation of Computing Systems (ToMPECS) %N 4 %P 19:1--19:36 %T Quantifying Cloud Performance and Dependability: Taxonomy, Metric Design, and Emerging Challenges %U http://doi.acm.org/10.1145/3236332 %V 3
• 
  CUP: A Formalism for Expressing Cloud Usage Patterns for Experts and Non-Experts Milenkoski, Aleksandar; Iosup, Alexandru; Kounev, Samuel; Sachs, Kai; Mularz, Diane E.; Curtiss, Jonathan A.; Ding, Jason J.; Rosenberg, Florian; Rygielski, Piotr; in IEEE Cloud Computing (2018). 5(3) 65–76. IEEE.
  • [ BibTeX ]
  • [ EndNote ]
  @article{MiIoKoSaMuCuDiRoRy2018-IEEECC-cup, author = {Milenkoski, Aleksandar and Iosup, Alexandru and Kounev, Samuel and Sachs, Kai and Mularz, Diane E. and Curtiss, Jonathan A. and Ding, Jason J. and Rosenberg, Florian and Rygielski, Piotr}, journal = {IEEE Cloud Computing}, keywords = {myown}, month = 6, number = 3, pages = {65–76}, publisher = {IEEE}, title = {{CUP: A Formalism for Expressing Cloud Usage Patterns for Experts and Non-Experts}}, volume = 5, year = 2018 }

  %0 Journal Article %1 MiIoKoSaMuCuDiRoRy2018-IEEECC-cup %A Milenkoski, Aleksandar %A Iosup, Alexandru %A Kounev, Samuel %A Sachs, Kai %A Mularz, Diane E. %A Curtiss, Jonathan A. %A Ding, Jason J. %A Rosenberg, Florian %A Rygielski, Piotr %D 2018 %I IEEE %J IEEE Cloud Computing %N 3 %P 65--76 %T CUP: A Formalism for Expressing Cloud Usage Patterns for Experts and Non-Experts %V 5

• 
  Modeling and Extracting Load Intensity Profiles von Kistowski, Jóakim; Herbst, Nikolas; Kounev, Samuel; Groenda, Henning; Stier, Christian; Lehrig, Sebastian; in ACM Transactions on Autonomous and Adaptive Systems (TAAS) (2017). 11(4) 23:1–23:28. ACM, New York, NY, USA.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Today's system developers and operators face the challenge of creating software systems that make efficient use of dynamically allocated resources under highly variable and dynamic load profiles, while at the same time delivering reliable performance. Autonomic controllers, e.g., an advanced auto-scaling mechanism in a cloud computing context, can benefit from an abstracted load model as knowledge to reconfigure on time and precisely. Existing workload characterization approaches have limited support to capture variations the inter-arrival times of incoming work units over time (i.e., a variable load profile). For example, industrial and scientific benchmarks support constant or stepwise increasing load, or inter-arrival times defined by statistical distributions or recorded traces. These options show shortcomings either in representative character of load variation patterns or in abstraction and flexibility of their format. In this article, we present the Descartes Load Intensity Model (DLIM) approach addressing these issues. DLIM provides a modeling formalism for describing load intensity variations over time. A DLIM instance is a compact formal description of a load intensity trace. DLIM-based tools provide features for benchmarking, performance and recorded load intensity trace analysis. As manually obtaining and maintaining DLIM instances becomes time consuming, we contribute three automated extraction methods and devised metrics for comparison and method selection. We discuss how these features are used to enhance system management approaches for adaptations during run-time, and how they are integrated into simulation contexts and enable benchmarking of elastic or adaptive behavior. We show that automatically extracted DLIM instances exhibit an average modeling error of 15.2% over ten different real-world traces that cover between two weeks and seven months. These results underline DLIM model expressiveness. In terms of accuracy and processing speed, our proposed extraction methods for the descriptive models are comparable to existing time series decomposition methods. Additionally, we illustrate DLIM applicability by outlining approaches of workload modeling in systems engineering that employ or rely on our proposed load intensity modeling formalism.

  @article{KiHeKo-TAAS17-ModelExtractLoadProfiles, abstract = {Today's system developers and operators face the challenge of creating software systems that make efficient use of dynamically allocated resources under highly variable and dynamic load profiles, while at the same time delivering reliable performance. Autonomic controllers, e.g., an advanced auto-scaling mechanism in a cloud computing context, can benefit from an abstracted load model as knowledge to reconfigure on time and precisely. Existing workload characterization approaches have limited support to capture variations the inter-arrival times of incoming work units over time (i.e., a variable load profile). For example, industrial and scientific benchmarks support constant or stepwise increasing load, or inter-arrival times defined by statistical distributions or recorded traces. These options show shortcomings either in representative character of load variation patterns or in abstraction and flexibility of their format. In this article, we present the Descartes Load Intensity Model (DLIM) approach addressing these issues. DLIM provides a modeling formalism for describing load intensity variations over time. A DLIM instance is a compact formal description of a load intensity trace. DLIM-based tools provide features for benchmarking, performance and recorded load intensity trace analysis. As manually obtaining and maintaining DLIM instances becomes time consuming, we contribute three automated extraction methods and devised metrics for comparison and method selection. We discuss how these features are used to enhance system management approaches for adaptations during run-time, and how they are integrated into simulation contexts and enable benchmarking of elastic or adaptive behavior. We show that automatically extracted DLIM instances exhibit an average modeling error of 15.2\% over ten different real-world traces that cover between two weeks and seven months. These results underline DLIM model expressiveness. In terms of accuracy and processing speed, our proposed extraction methods for the descriptive models are comparable to existing time series decomposition methods. Additionally, we illustrate DLIM applicability by outlining approaches of workload modeling in systems engineering that employ or rely on our proposed load intensity modeling formalism.}, address = {New York, NY, USA}, author = {von Kistowski, J{\'o}akim and Herbst, Nikolas and Kounev, Samuel and Groenda, Henning and Stier, Christian and Lehrig, Sebastian}, journal = {ACM Transactions on Autonomous and Adaptive Systems (TAAS)}, keywords = {myown}, month = {January}, number = 4, pages = {23:1–23:28}, publisher = {ACM}, title = {{Modeling and Extracting Load Intensity Profiles}}, volume = 11, year = 2017 }

  %0 Journal Article %1 KiHeKo-TAAS17-ModelExtractLoadProfiles %A von Kistowski, Jóakim %A Herbst, Nikolas %A Kounev, Samuel %A Groenda, Henning %A Stier, Christian %A Lehrig, Sebastian %C New York, NY, USA %D 2017 %I ACM %J ACM Transactions on Autonomous and Adaptive Systems (TAAS) %N 4 %P 23:1--23:28 %T Modeling and Extracting Load Intensity Profiles %U http://doi.acm.org/10.1145/3019596 %V 11 %X Today's system developers and operators face the challenge of creating software systems that make efficient use of dynamically allocated resources under highly variable and dynamic load profiles, while at the same time delivering reliable performance. Autonomic controllers, e.g., an advanced auto-scaling mechanism in a cloud computing context, can benefit from an abstracted load model as knowledge to reconfigure on time and precisely. Existing workload characterization approaches have limited support to capture variations the inter-arrival times of incoming work units over time (i.e., a variable load profile). For example, industrial and scientific benchmarks support constant or stepwise increasing load, or inter-arrival times defined by statistical distributions or recorded traces. These options show shortcomings either in representative character of load variation patterns or in abstraction and flexibility of their format. In this article, we present the Descartes Load Intensity Model (DLIM) approach addressing these issues. DLIM provides a modeling formalism for describing load intensity variations over time. A DLIM instance is a compact formal description of a load intensity trace. DLIM-based tools provide features for benchmarking, performance and recorded load intensity trace analysis. As manually obtaining and maintaining DLIM instances becomes time consuming, we contribute three automated extraction methods and devised metrics for comparison and method selection. We discuss how these features are used to enhance system management approaches for adaptations during run-time, and how they are integrated into simulation contexts and enable benchmarking of elastic or adaptive behavior. We show that automatically extracted DLIM instances exhibit an average modeling error of 15.2% over ten different real-world traces that cover between two weeks and seven months. These results underline DLIM model expressiveness. In terms of accuracy and processing speed, our proposed extraction methods for the descriptive models are comparable to existing time series decomposition methods. Additionally, we illustrate DLIM applicability by outlining approaches of workload modeling in systems engineering that employ or rely on our proposed load intensity modeling formalism.
• 
  Model-Based Self-Aware Performance and Resource Management Using the Descartes Modeling Language Huber, Nikolaus; Brosig, Fabian; Spinner, Simon; Kounev, Samuel; Bähr, Manuel; in IEEE Transactions on Software Engineering (TSE) (2017). 43(5) 432–452. IEEE Computer Society.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Modern IT systems have increasingly distributed and dynamic architectures providing flexibility to adapt to changes in the environment and thus enabling higher resource efficiency. However, these benefits come at the cost of higher system complexity and dynamics. Thus, engineering systems that manage their end-to-end application performance and resource efficiency in an autonomic manner is a challenge. In this article, we present a holistic model-based approach for self-aware performance and resource management leveraging the Descartes Modeling Language (DML), an architecture-level modeling language for online performance and resource management. We propose a novel online performance prediction process that dynamically tailors the model solving depending on the requirements regarding accuracy and overhead. Using these prediction capabilities, we implement a generic model-based control loop for proactive system adaptation. We evaluate our model-based approach in the context of two representative case studies showing that with the proposed methods, significant resource efficiency gains can be achieved while maintaining performance requirements. These results represent the first end-to-end validation of our approach, demonstrating its potential for self-aware performance and resource management in the context of modern IT systems and infrastructures.

  @article{HuBrSpKoBa2017-TSE-DML, abstract = {Modern IT systems have increasingly distributed and dynamic architectures providing flexibility to adapt to changes in the environment and thus enabling higher resource efficiency. However, these benefits come at the cost of higher system complexity and dynamics. Thus, engineering systems that manage their end-to-end application performance and resource efficiency in an autonomic manner is a challenge. In this article, we present a holistic model-based approach for self-aware performance and resource management leveraging the Descartes Modeling Language (DML), an architecture-level modeling language for online performance and resource management. We propose a novel online performance prediction process that dynamically tailors the model solving depending on the requirements regarding accuracy and overhead. Using these prediction capabilities, we implement a generic model-based control loop for proactive system adaptation. We evaluate our model-based approach in the context of two representative case studies showing that with the proposed methods, significant resource efficiency gains can be achieved while maintaining performance requirements. These results represent the first end-to-end validation of our approach, demonstrating its potential for self-aware performance and resource management in the context of modern IT systems and infrastructures.}, author = {Huber, Nikolaus and Brosig, Fabian and Spinner, Simon and Kounev, Samuel and B{\"a}hr, Manuel}, journal = {IEEE Transactions on Software Engineering (TSE)}, keywords = {myown}, number = 5, pages = {432–452}, publisher = {IEEE Computer Society}, title = {{Model-Based Self-Aware Performance and Resource Management Using the Descartes Modeling Language}}, volume = 43, year = 2017 }

  %0 Journal Article %1 HuBrSpKoBa2017-TSE-DML %A Huber, Nikolaus %A Brosig, Fabian %A Spinner, Simon %A Kounev, Samuel %A Bähr, Manuel %D 2017 %I IEEE Computer Society %J IEEE Transactions on Software Engineering (TSE) %N 5 %P 432--452 %T Model-Based Self-Aware Performance and Resource Management Using the Descartes Modeling Language %U http://dx.doi.org/10.1109/TSE.2016.2613863 %V 43 %X Modern IT systems have increasingly distributed and dynamic architectures providing flexibility to adapt to changes in the environment and thus enabling higher resource efficiency. However, these benefits come at the cost of higher system complexity and dynamics. Thus, engineering systems that manage their end-to-end application performance and resource efficiency in an autonomic manner is a challenge. In this article, we present a holistic model-based approach for self-aware performance and resource management leveraging the Descartes Modeling Language (DML), an architecture-level modeling language for online performance and resource management. We propose a novel online performance prediction process that dynamically tailors the model solving depending on the requirements regarding accuracy and overhead. Using these prediction capabilities, we implement a generic model-based control loop for proactive system adaptation. We evaluate our model-based approach in the context of two representative case studies showing that with the proposed methods, significant resource efficiency gains can be achieved while maintaining performance requirements. These results represent the first end-to-end validation of our approach, demonstrating its potential for self-aware performance and resource management in the context of modern IT systems and infrastructures.

• 
  Enabling Fluid Analysis for Queueing Petri Nets via Model Transformation Müller, Christoph; Rygielski, Piotr; Spinner, Simon; Kounev, Samuel; in Electronic Notes in Theoretical Computer Science (2016). 327 71–91. Elsevier.

  The 8th International Workshop on Practical Application of Stochastic Modeling, PASM 2016
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Abstract Due to the growing size of modern IT} systems, their performance analysis becomes an even more challenging task. Existing simulators are unable to analyze the behavior of large systems in a reasonable time, whereas analytical methods suffer from the state space explosion problem. Fluid analysis techniques can be used to approximate the solution of high-order Markov chain models enabling time efficient analysis of large performance models. In this paper, we describe a model-to-model transformation from queueing Petri nets (QPN) into layered queueing networks (LQN). Obtained LQN} models can benefit from three existing solvers: LINE, LQNS, LQSIM. LINE} internally utilize fluid limits approximation to speed up the solving process for large models. We present the incentives for developing the automated model-to-model transformation and present a systematic approach that we followed in its design. We demonstrate the transformations using representative examples. Finally, we evaluate and compare the performance predictions of existing analytical, simulation and fluid analysis solvers. We analyze solvers' limitations, solving time, and memory consumption.

  @article{MuRySpKo2016-PASM-QPN-LQN-Transformation, abstract = {Abstract Due to the growing size of modern \{IT\} systems, their performance analysis becomes an even more challenging task. Existing simulators are unable to analyze the behavior of large systems in a reasonable time, whereas analytical methods suffer from the state space explosion problem. Fluid analysis techniques can be used to approximate the solution of high-order Markov chain models enabling time efficient analysis of large performance models. In this paper, we describe a model-to-model transformation from queueing Petri nets (QPN) into layered queueing networks (LQN). Obtained \{LQN\} models can benefit from three existing solvers: LINE, LQNS, LQSIM. \{LINE\} internally utilize fluid limits approximation to speed up the solving process for large models. We present the incentives for developing the automated model-to-model transformation and present a systematic approach that we followed in its design. We demonstrate the transformations using representative examples. Finally, we evaluate and compare the performance predictions of existing analytical, simulation and fluid analysis solvers. We analyze solvers' limitations, solving time, and memory consumption.}, author = {M{\"u}ller, Christoph and Rygielski, Piotr and Spinner, Simon and Kounev, Samuel}, journal = {Electronic Notes in Theoretical Computer Science}, keywords = {myown}, note = {The 8th International Workshop on Practical Application of Stochastic Modeling, PASM 2016}, pages = {71–91}, publisher = {Elsevier}, title = {{Enabling Fluid Analysis for Queueing Petri Nets via Model Transformation}}, volume = 327, year = 2016 }

  %0 Journal Article %1 MuRySpKo2016-PASM-QPN-LQN-Transformation %A Müller, Christoph %A Rygielski, Piotr %A Spinner, Simon %A Kounev, Samuel %D 2016 %I Elsevier %J Electronic Notes in Theoretical Computer Science %P 71--91 %T Enabling Fluid Analysis for Queueing Petri Nets via Model Transformation %U http://www.sciencedirect.com/science/article/pii/S157106611630069X %V 327 %X Abstract Due to the growing size of modern IT} systems, their performance analysis becomes an even more challenging task. Existing simulators are unable to analyze the behavior of large systems in a reasonable time, whereas analytical methods suffer from the state space explosion problem. Fluid analysis techniques can be used to approximate the solution of high-order Markov chain models enabling time efficient analysis of large performance models. In this paper, we describe a model-to-model transformation from queueing Petri nets (QPN) into layered queueing networks (LQN). Obtained LQN} models can benefit from three existing solvers: LINE, LQNS, LQSIM. LINE} internally utilize fluid limits approximation to speed up the solving process for large models. We present the incentives for developing the automated model-to-model transformation and present a systematic approach that we followed in its design. We demonstrate the transformations using representative examples. Finally, we evaluate and compare the performance predictions of existing analytical, simulation and fluid analysis solvers. We analyze solvers' limitations, solving time, and memory consumption.
• 
  A Model-Based Approach to Designing Self-Aware IT Systems and Infrastructures Kounev, Samuel; Huber, Nikolaus; Brosig, Fabian; Zhu, Xiaoyun; in IEEE Computer (2016). 49(7) 53–61. IEEE.
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  @article{KoHuBrZh2016-IEEEComputer-Descartes, author = {Kounev, Samuel and Huber, Nikolaus and Brosig, Fabian and Zhu, Xiaoyun}, journal = {IEEE Computer}, keywords = {myown}, month = {July}, number = 7, pages = {53-61}, publisher = {IEEE}, title = {{A Model-Based Approach to Designing Self-Aware IT Systems and Infrastructures}}, volume = 49, year = 2016 }

  %0 Journal Article %1 KoHuBrZh2016-IEEEComputer-Descartes %A Kounev, Samuel %A Huber, Nikolaus %A Brosig, Fabian %A Zhu, Xiaoyun %D 2016 %I IEEE %J IEEE Computer %N 7 %P 53-61 %T A Model-Based Approach to Designing Self-Aware IT Systems and Infrastructures %U http://dx.doi.org/10.1109/MC.2016.198 %V 49

• 
  Parallel Simulation of Queueing Petri Nets Walter, Jürgen; Spinner, Simon; Kounev, Samuel; in EAI Endorsed Transactions on Industrial Networks and Intelligent Systems (2015). 16(8) ACM.
  • [ BibTeX ]
  • [ EndNote ]
  @article{WaSpKo2016-EAI-parallelSimulation, author = {Walter, J{\"u}rgen and Spinner, Simon and Kounev, Samuel}, journal = {EAI Endorsed Transactions on Industrial Networks and Intelligent Systems}, keywords = {myown}, month = 8, number = 8, publisher = {ACM}, title = {{Parallel Simulation of Queueing Petri Nets}}, volume = 16, year = 2015 }

  %0 Journal Article %1 WaSpKo2016-EAI-parallelSimulation %A Walter, Jürgen %A Spinner, Simon %A Kounev, Samuel %D 2015 %I ACM %J EAI Endorsed Transactions on Industrial Networks and Intelligent Systems %N 8 %T Parallel Simulation of Queueing Petri Nets %V 16
• 
  Evaluating Approaches to Resource Demand Estimation Spinner, Simon; Casale, Giuliano; Brosig, Fabian; Kounev, Samuel; in Performance Evaluation (2015). 92 51–71. Elsevier B.V.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Resource demands are a key parameter of stochastic performance models that needs to be determined when performing a quantitative performance analysis of a system. However, the direct measurement of resource demands is not feasible in most realistic systems. Therefore, statistical approaches that estimate resource demands based on coarse-grained monitoring data (e.g., CPU utilization, and response times) have been proposed in the literature. These approaches have different assumptions and characteristics that need to be considered when estimating resource demands. This paper surveys the state-of-the-art in resource demand estimation and proposes a classification scheme for estimation approaches. Furthermore, it contains an experimental evaluation comparing the impact of different factors (monitoring window size, number of workload classes, load level, collinearity, and model mismatch) on the estimation accuracy of seven different approaches. The classification scheme and the experimental comparison helps performance engineers to select an approach to resource demand estimation that fulfills the requirements of a given analysis scenario.

  @article{SpCaBrKo2015-PEVA-ResDemEstSurvey, abstract = {Resource demands are a key parameter of stochastic performance models that needs to be determined when performing a quantitative performance analysis of a system. However, the direct measurement of resource demands is not feasible in most realistic systems. Therefore, statistical approaches that estimate resource demands based on coarse-grained monitoring data (e.g., CPU utilization, and response times) have been proposed in the literature. These approaches have different assumptions and characteristics that need to be considered when estimating resource demands. This paper surveys the state-of-the-art in resource demand estimation and proposes a classification scheme for estimation approaches. Furthermore, it contains an experimental evaluation comparing the impact of different factors (monitoring window size, number of workload classes, load level, collinearity, and model mismatch) on the estimation accuracy of seven different approaches. The classification scheme and the experimental comparison helps performance engineers to select an approach to resource demand estimation that fulfills the requirements of a given analysis scenario.}, author = {Spinner, Simon and Casale, Giuliano and Brosig, Fabian and Kounev, Samuel}, journal = {Performance Evaluation}, keywords = {myown}, month = {October}, pages = {51–71}, publisher = {Elsevier B.V.}, title = {{Evaluating Approaches to Resource Demand Estimation}}, volume = 92, year = 2015 }

  %0 Journal Article %1 SpCaBrKo2015-PEVA-ResDemEstSurvey %A Spinner, Simon %A Casale, Giuliano %A Brosig, Fabian %A Kounev, Samuel %D 2015 %I Elsevier B.V. %J Performance Evaluation %P 51--71 %T Evaluating Approaches to Resource Demand Estimation %U http://www.sciencedirect.com/science/article/pii/S0166531615000711 %V 92 %X Resource demands are a key parameter of stochastic performance models that needs to be determined when performing a quantitative performance analysis of a system. However, the direct measurement of resource demands is not feasible in most realistic systems. Therefore, statistical approaches that estimate resource demands based on coarse-grained monitoring data (e.g., CPU utilization, and response times) have been proposed in the literature. These approaches have different assumptions and characteristics that need to be considered when estimating resource demands. This paper surveys the state-of-the-art in resource demand estimation and proposes a classification scheme for estimation approaches. Furthermore, it contains an experimental evaluation comparing the impact of different factors (monitoring window size, number of workload classes, load level, collinearity, and model mismatch) on the estimation accuracy of seven different approaches. The classification scheme and the experimental comparison helps performance engineers to select an approach to resource demand estimation that fulfills the requirements of a given analysis scenario.
• 
  Evaluating Computer Intrusion Detection Systems: A Survey of Common Practices Milenkoski, Aleksandar; Vieira, Marco; Kounev, Samuel; Avritzer, Alberto; Payne, Bryan D.; in ACM Computing Surveys (2015). 48(1) 12:1–12:41. ACM, New York, NY, USA.

  5-year Impact Factor (2014): 5.949
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  The evaluation of computer intrusion detection systems (which we refer to as intrusion detection systems) is an active research area. In this paper, we survey and systematize common practices in the area of evaluation of intrusion detection systems. For this purpose, we define a design space structured into three parts: workload, metrics, and measurement methodology. We then provide an overview of the common practices in evaluation of intrusion detection systems by surveying evaluation approaches and methods related to each part of the design space. Finally, we discuss open issues and challenges focusing on evaluation methodologies for novel intrusion detection systems.

  @article{MiViKoAvPa2015-CSUR-IDSEval, abstract = {{The evaluation of computer intrusion detection systems (which we refer to as intrusion detection systems) is an active research area. In this paper, we survey and systematize common practices in the area of evaluation of intrusion detection systems. For this purpose, we define a design space structured into three parts: workload, metrics, and measurement methodology. We then provide an overview of the common practices in evaluation of intrusion detection systems by surveying evaluation approaches and methods related to each part of the design space. Finally, we discuss open issues and challenges focusing on evaluation methodologies for novel intrusion detection systems.}}, address = {New York, NY, USA}, author = {Milenkoski, Aleksandar and Vieira, Marco and Kounev, Samuel and Avritzer, Alberto and Payne, Bryan D.}, journal = {{ACM Computing Surveys}}, keywords = {myown}, month = {{September}}, note = {{<b>5-year Impact Factor (2014): 5.949</b>}}, number = 1, pages = {12:1–12:41}, publisher = {ACM}, title = {{Evaluating Computer Intrusion Detection Systems: A Survey of Common Practices}}, volume = 48, year = 2015 }

  %0 Journal Article %1 MiViKoAvPa2015-CSUR-IDSEval %A Milenkoski, Aleksandar %A Vieira, Marco %A Kounev, Samuel %A Avritzer, Alberto %A Payne, Bryan D. %C New York, NY, USA %D 2015 %I ACM %J ACM Computing Surveys %N 1 %P 12:1--12:41 %T Evaluating Computer Intrusion Detection Systems: A Survey of Common Practices %U http://dl.acm.org/authorize?N06203 %V 48 %X The evaluation of computer intrusion detection systems (which we refer to as intrusion detection systems) is an active research area. In this paper, we survey and systematize common practices in the area of evaluation of intrusion detection systems. For this purpose, we define a design space structured into three parts: workload, metrics, and measurement methodology. We then provide an overview of the common practices in evaluation of intrusion detection systems by surveying evaluation approaches and methods related to each part of the design space. Finally, we discuss open issues and challenges focusing on evaluation methodologies for novel intrusion detection systems.
• 
  Quantitative Evaluation of Model-Driven Performance Analysis and Simulation of Component-based Architectures Brosig, Fabian; Meier, Philipp; Becker, Steffen; Koziolek, Anne; Koziolek, Heiko; Kounev, Samuel; in IEEE Transactions on Software Engineering (TSE) (2015). 41(2) 157–175. IEEE.
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  @article{BrMeBeKoKoKo2014-TSE-ModelSolutionTradeOffs, author = {Brosig, Fabian and Meier, Philipp and Becker, Steffen and Koziolek, Anne and Koziolek, Heiko and Kounev, Samuel}, journal = {IEEE Transactions on Software Engineering (TSE)}, keywords = {myown}, month = {{February}}, number = 2, pages = {157–175}, publisher = {IEEE}, title = {{Quantitative Evaluation of Model-Driven Performance Analysis and Simulation of Component-based Architectures}}, volume = 41, year = 2015 }

  %0 Journal Article %1 BrMeBeKoKoKo2014-TSE-ModelSolutionTradeOffs %A Brosig, Fabian %A Meier, Philipp %A Becker, Steffen %A Koziolek, Anne %A Koziolek, Heiko %A Kounev, Samuel %D 2015 %I IEEE %J IEEE Transactions on Software Engineering (TSE) %N 2 %P 157--175 %T Quantitative Evaluation of Model-Driven Performance Analysis and Simulation of Component-based Architectures %U http://dx.doi.org/10.1109/TSE.2014.2362755 %V 41

• 
  Self-Adaptive Workload Classification and Forecasting for Proactive Resource Provisioning Herbst, Nikolas Roman; Huber, Nikolaus; Kounev, Samuel; Amrehn, Erich; in Concurrency and Computation - Practice and Experience, John Wiley and Sons, Ltd. (2014). 26(12) 2053–2078.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  As modern enterprise software systems become increasingly dynamic, workload forecasting techniques are gaining in importance as a foundation for online capacity planning and resource management. Time series analysis covers a broad spectrum of methods to calculate workload forecasts based on history monitoring data. Related work in the field of workload forecasting mostly concentrates on evaluating specific methods and their individual optimisation potential or on predicting Quality-of-Service (QoS) metrics directly. As a basis, we present a survey on established forecasting methods of the time series analysis concerning their benefits and drawbacks and group them according to their computational overheads. In this paper, we propose a novel self-adaptive approach that selects suitable forecasting methods for a given context based on a decision tree and direct feedback cycles together with a corresponding implementation. The user needs to provide only his general forecasting objectives. In several experiments and case studies based on real world workload traces, we show that our implementation of the approach provides continuous and reliable forecast results at run-time. The results of this extensive evaluation show that the relative error of the individual forecast points is significantly reduced compared to statically applied forecasting methods, e.g. in an exemplary scenario on average by 37%. In a case study, between 55% and 75% of the violations of a given service level agreement can be prevented by applying proactive resource provisioning based on the forecast results of our implementation.

  @article{HuKoAm2013-CCPE-WorkloadClassificationAndForecasting, abstract = {As modern enterprise software systems become increasingly dynamic, workload forecasting techniques are gaining in importance as a foundation for online capacity planning and resource management. Time series analysis covers a broad spectrum of methods to calculate workload forecasts based on history monitoring data. Related work in the field of workload forecasting mostly concentrates on evaluating specific methods and their individual optimisation potential or on predicting Quality-of-Service (QoS) metrics directly. As a basis, we present a survey on established forecasting methods of the time series analysis concerning their benefits and drawbacks and group them according to their computational overheads. In this paper, we propose a novel self-adaptive approach that selects suitable forecasting methods for a given context based on a decision tree and direct feedback cycles together with a corresponding implementation. The user needs to provide only his general forecasting objectives. In several experiments and case studies based on real world workload traces, we show that our implementation of the approach provides continuous and reliable forecast results at run-time. The results of this extensive evaluation show that the relative error of the individual forecast points is significantly reduced compared to statically applied forecasting methods, e.g. in an exemplary scenario on average by 37%. In a case study, between 55% and 75% of the violations of a given service level agreement can be prevented by applying proactive resource provisioning based on the forecast results of our implementation.}, author = {Herbst, Nikolas Roman and Huber, Nikolaus and Kounev, Samuel and Amrehn, Erich}, journal = {Concurrency and Computation - Practice and Experience, John Wiley and Sons, Ltd.}, keywords = {myown}, month = {March}, number = 12, pages = {2053–2078}, title = {{Self-Adaptive Workload Classification and Forecasting for Proactive Resource Provisioning}}, volume = 26, year = 2014 }

  %0 Journal Article %1 HuKoAm2013-CCPE-WorkloadClassificationAndForecasting %A Herbst, Nikolas Roman %A Huber, Nikolaus %A Kounev, Samuel %A Amrehn, Erich %D 2014 %J Concurrency and Computation - Practice and Experience, John Wiley and Sons, Ltd. %N 12 %P 2053--2078 %T Self-Adaptive Workload Classification and Forecasting for Proactive Resource Provisioning %U http://dx.doi.org/10.1002/cpe.3224 %V 26 %X As modern enterprise software systems become increasingly dynamic, workload forecasting techniques are gaining in importance as a foundation for online capacity planning and resource management. Time series analysis covers a broad spectrum of methods to calculate workload forecasts based on history monitoring data. Related work in the field of workload forecasting mostly concentrates on evaluating specific methods and their individual optimisation potential or on predicting Quality-of-Service (QoS) metrics directly. As a basis, we present a survey on established forecasting methods of the time series analysis concerning their benefits and drawbacks and group them according to their computational overheads. In this paper, we propose a novel self-adaptive approach that selects suitable forecasting methods for a given context based on a decision tree and direct feedback cycles together with a corresponding implementation. The user needs to provide only his general forecasting objectives. In several experiments and case studies based on real world workload traces, we show that our implementation of the approach provides continuous and reliable forecast results at run-time. The results of this extensive evaluation show that the relative error of the individual forecast points is significantly reduced compared to statically applied forecasting methods, e.g. in an exemplary scenario on average by 37%. In a case study, between 55% and 75% of the violations of a given service level agreement can be prevented by applying proactive resource provisioning based on the forecast results of our implementation.
• 
  Metrics and Techniques for Quantifying Performance Isolation in Cloud Environments Krebs, Rouven; Momm, Christof; Kounev, Samuel; in Elsevier Science of Computer Programming Journal (SciCo) (2014). Vol. 90, Part B 116–134. Elsevier B.V.
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @article{KrMoKo2013-SciCo-MetricsAndTechniquesForPerformanceIsolation, author = {Krebs, Rouven and Momm, Christof and Kounev, Samuel}, journal = {Elsevier Science of Computer Programming Journal (SciCo)}, keywords = {myown}, month = {February}, pages = {116–134}, publisher = {Elsevier B.V.}, title = {{Metrics and Techniques for Quantifying Performance Isolation in Cloud Environments}}, volume = {Vol. 90, Part B}, year = 2014 }

  %0 Journal Article %1 KrMoKo2013-SciCo-MetricsAndTechniquesForPerformanceIsolation %A Krebs, Rouven %A Momm, Christof %A Kounev, Samuel %D 2014 %I Elsevier B.V. %J Elsevier Science of Computer Programming Journal (SciCo) %P 116--134 %T Metrics and Techniques for Quantifying Performance Isolation in Cloud Environments %V Vol. 90, Part B
• 
  Architecture-Level Software Performance Abstractions for Online Performance Prediction Brosig, Fabian; Huber, Nikolaus; Kounev, Samuel; in Elsevier Science of Computer Programming Journal (SciCo) (2014). Vol. 90, Part B 71–92. Elsevier.
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  @article{BrHuKo2013-SciCo-SoftwarePerformanceAbstractions, author = {Brosig, Fabian and Huber, Nikolaus and Kounev, Samuel}, journal = {Elsevier Science of Computer Programming Journal (SciCo)}, keywords = {myown}, month = {September}, pages = {71–92}, publisher = {Elsevier}, title = {{Architecture-Level Software Performance Abstractions for Online Performance Prediction}}, volume = {Vol. 90, Part B}, year = 2014 }

  %0 Journal Article %1 BrHuKo2013-SciCo-SoftwarePerformanceAbstractions %A Brosig, Fabian %A Huber, Nikolaus %A Kounev, Samuel %D 2014 %I Elsevier %J Elsevier Science of Computer Programming Journal (SciCo) %P 71--92 %T Architecture-Level Software Performance Abstractions for Online Performance Prediction %U http://authors.elsevier.com/sd/article/S0167642313001421 %V Vol. 90, Part B
• 
  Modeling Run-Time Adaptation at the System Architecture Level in Dynamic Service-Oriented Environments Huber, Nikolaus; van Hoorn, André; Koziolek, Anne; Brosig, Fabian; Kounev, Samuel; in Service Oriented Computing and Applications Journal (SOCA) (2014). 8(1) 73–89. Springer-Verlag.
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @article{HuHoKoBrKo2014-SOCA-ModelingRuntimeAdaptation, author = {Huber, Nikolaus and van Hoorn, Andr{\'e} and Koziolek, Anne and Brosig, Fabian and Kounev, Samuel}, journal = {Service Oriented Computing and Applications Journal (SOCA)}, keywords = {myown}, month = {{March}}, number = 1, pages = {73–89}, publisher = {Springer-Verlag}, title = {{Modeling Run-Time Adaptation at the System Architecture Level in Dynamic Service-Oriented Environments}}, volume = 8, year = 2014 }

  %0 Journal Article %1 HuHoKoBrKo2014-SOCA-ModelingRuntimeAdaptation %A Huber, Nikolaus %A van Hoorn, André %A Koziolek, Anne %A Brosig, Fabian %A Kounev, Samuel %D 2014 %I Springer-Verlag %J Service Oriented Computing and Applications Journal (SOCA) %N 1 %P 73--89 %T Modeling Run-Time Adaptation at the System Architecture Level in Dynamic Service-Oriented Environments %V 8

• 
  Performance modeling and analysis of message-oriented event-driven systems Sachs, Kai; Kounev, Samuel; Buchmann, Alejandro; in Journal of Software and Systems Modeling (2013). 12(4) 705–729. Springer-Verlag.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Message-oriented event-driven systems are becoming increasingly ubiquitous in many industry domains including telecommunications, transportation and supply chain management. Applications in these areas typically have stringent requirements for performance and scalability. To guarantee adequate quality-of-service, systems must be subjected to a rigorous performance and scalability analysis before they are put into production. In this paper, we present a comprehensive modeling methodology for message-oriented event-driven systems in the context of a case study of a representative application in the supply chain management domain. The methodology, which is based on queueing Petri nets, provides a basis for performance analysis and capacity planning. We study a deployment of the SPECjms2007 standard benchmark on a leading commercial middleware platform. A detailed system model is built in a step-by-step fashion and then used to predict the system performance under various workload and configuration scenarios. After the case study, we present a set of generic performance modeling patterns that can be used as building blocks when modeling message-oriented event-driven systems. The results demonstrate the effectiveness, practicality and accuracy of the proposed modeling and prediction approach.

  @article{SaKoBu2011-SoSyM-PerfModMoEdSys, abstract = {Message-oriented event-driven systems are becoming increasingly ubiquitous in many industry domains including telecommunications, transportation and supply chain management. Applications in these areas typically have stringent requirements for performance and scalability. To guarantee adequate quality-of-service, systems must be subjected to a rigorous performance and scalability analysis before they are put into production. In this paper, we present a comprehensive modeling methodology for message-oriented event-driven systems in the context of a case study of a representative application in the supply chain management domain. The methodology, which is based on queueing Petri nets, provides a basis for performance analysis and capacity planning. We study a deployment of the SPECjms2007 standard benchmark on a leading commercial middleware platform. A detailed system model is built in a step-by-step fashion and then used to predict the system performance under various workload and configuration scenarios. After the case study, we present a set of generic performance modeling patterns that can be used as building blocks when modeling message-oriented event-driven systems. The results demonstrate the effectiveness, practicality and accuracy of the proposed modeling and prediction approach.}, author = {Sachs, Kai and Kounev, Samuel and Buchmann, Alejandro}, journal = {Journal of Software and Systems Modeling}, keywords = {myown}, number = 4, pages = {705–729}, publisher = {Springer-Verlag}, title = {Performance modeling and analysis of message-oriented event-driven systems}, volume = 12, year = 2013 }

  %0 Journal Article %1 SaKoBu2011-SoSyM-PerfModMoEdSys %A Sachs, Kai %A Kounev, Samuel %A Buchmann, Alejandro %D 2013 %I Springer-Verlag %J Journal of Software and Systems Modeling %N 4 %P 705--729 %T Performance modeling and analysis of message-oriented event-driven systems %V 12 %X Message-oriented event-driven systems are becoming increasingly ubiquitous in many industry domains including telecommunications, transportation and supply chain management. Applications in these areas typically have stringent requirements for performance and scalability. To guarantee adequate quality-of-service, systems must be subjected to a rigorous performance and scalability analysis before they are put into production. In this paper, we present a comprehensive modeling methodology for message-oriented event-driven systems in the context of a case study of a representative application in the supply chain management domain. The methodology, which is based on queueing Petri nets, provides a basis for performance analysis and capacity planning. We study a deployment of the SPECjms2007 standard benchmark on a leading commercial middleware platform. A detailed system model is built in a step-by-step fashion and then used to predict the system performance under various workload and configuration scenarios. After the case study, we present a set of generic performance modeling patterns that can be used as building blocks when modeling message-oriented event-driven systems. The results demonstrate the effectiveness, practicality and accuracy of the proposed modeling and prediction approach.
• 
  Network Virtualization for QoS-Aware Resource Management in Cloud Data Centers: A Survey Rygielski, Piotr; Kounev, Samuel; in PIK --- Praxis der Informationsverarbeitung und Kommunikation (2013). 36(1) 55–64. de Gruyter.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  The increasing popularity of Cloud Computing is leading to the emergence of large virtualized data centers hosting increasingly complex and dynamic IT systems and services. Over the past decade, the efficient sharing of computational resources through virtualization has been subject to intensive research, while network management in cloud data centers has received less attention. A variety of network-intensive applications require QoS (Quality-of-Service) provisioning, performance isolation and support for flexible and efficient migration of virtual machines. In this paper, we survey existing network virtualization approaches and evaluate the extent to which they can be used as a basis for realizing the mentioned requirements in a cloud data center. More specifically, we identify generic network virtualization techniques, characterize them according to their features related to QoS management and performance isolation, and show how they can be composed together and used as building blocks for complex network virtualization solutions. We then present an overview of selected representative cloud platforms and show how they leverage the generic techniques as a basis for network resource management. Finally, we outline open issues and research challenges in the area of performance modeling and proactive resource management of virtualized data center infrastructures.

  @article{RyKo2013, abstract = {The increasing popularity of Cloud Computing is leading to the emergence of large virtualized data centers hosting increasingly complex and dynamic IT systems and services. Over the past decade, the efficient sharing of computational resources through virtualization has been subject to intensive research, while network management in cloud data centers has received less attention. A variety of network-intensive applications require QoS (Quality-of-Service) provisioning, performance isolation and support for flexible and efficient migration of virtual machines. In this paper, we survey existing network virtualization approaches and evaluate the extent to which they can be used as a basis for realizing the mentioned requirements in a cloud data center. More specifically, we identify generic network virtualization techniques, characterize them according to their features related to QoS management and performance isolation, and show how they can be composed together and used as building blocks for complex network virtualization solutions. We then present an overview of selected representative cloud platforms and show how they leverage the generic techniques as a basis for network resource management. Finally, we outline open issues and research challenges in the area of performance modeling and proactive resource management of virtualized data center infrastructures.}, author = {Rygielski, Piotr and Kounev, Samuel}, journal = {PIK — Praxis der Informationsverarbeitung und Kommunikation}, keywords = {myown}, month = {February}, number = 1, pages = {55–64}, publisher = {de Gruyter}, title = {{Network Virtualization for QoS-Aware Resource Management in Cloud Data Centers: A Survey}}, volume = 36, year = 2013 }

  %0 Journal Article %1 RyKo2013 %A Rygielski, Piotr %A Kounev, Samuel %D 2013 %I de Gruyter %J PIK --- Praxis der Informationsverarbeitung und Kommunikation %N 1 %P 55--64 %T Network Virtualization for QoS-Aware Resource Management in Cloud Data Centers: A Survey %U http://www.degruyter.com/view/j/piko-2013-36-issue-1/pik-2012-0136/pik-2012-0136.xml?format=INT %V 36 %X The increasing popularity of Cloud Computing is leading to the emergence of large virtualized data centers hosting increasingly complex and dynamic IT systems and services. Over the past decade, the efficient sharing of computational resources through virtualization has been subject to intensive research, while network management in cloud data centers has received less attention. A variety of network-intensive applications require QoS (Quality-of-Service) provisioning, performance isolation and support for flexible and efficient migration of virtual machines. In this paper, we survey existing network virtualization approaches and evaluate the extent to which they can be used as a basis for realizing the mentioned requirements in a cloud data center. More specifically, we identify generic network virtualization techniques, characterize them according to their features related to QoS management and performance isolation, and show how they can be composed together and used as building blocks for complex network virtualization solutions. We then present an overview of selected representative cloud platforms and show how they leverage the generic techniques as a basis for network resource management. Finally, we outline open issues and research challenges in the area of performance modeling and proactive resource management of virtualized data center infrastructures.
• 
  Modeling Event-based Communication in Component-based Software Architectures for Performance Predictions Rathfelder, Christoph; Klatt, Benjamin; Sachs, Kai; Kounev, Samuel; in Journal of Software and Systems Modeling (SoSyM) (2013). 1–27. Springer Verlag.
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  @article{rathfelder2013a, author = {Rathfelder, Christoph and Klatt, Benjamin and Sachs, Kai and Kounev, Samuel}, journal = {Journal of Software and Systems Modeling (SoSyM)}, keywords = {myown}, month = {March}, pages = {1–27}, publisher = {Springer Verlag}, title = {Modeling Event-based Communication in Component-based Software Architectures for Performance Predictions}, year = 2013 }

  %0 Journal Article %1 rathfelder2013a %A Rathfelder, Christoph %A Klatt, Benjamin %A Sachs, Kai %A Kounev, Samuel %D 2013 %I Springer Verlag %J Journal of Software and Systems Modeling (SoSyM) %P 1--27 %T Modeling Event-based Communication in Component-based Software Architectures for Performance Predictions %U http://dx.doi.org/10.1007/s10270-013-0316-x
• 
  Modeling of Event-based Communication in Component-based Architectures: State-of-the-Art and Future Directions Kounev, Samuel; Rathfelder, Christoph; Klatt, Benjamin; in Electronic Notes in Theoretical Computer Science (ENTCS) (2013). 295 3–9. Elsevier Science Publishers B. V., Amsterdam, The Netherlands.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Event-based communication is used in different domains including telecommunications, transportation, and business information systems to build scalable distributed systems. Such systems typically have stringent requirements for performance and scalability as they provide business and mission critical services. While the use of event-based communication enables loosely-coupled interactions between components and leads to improved system scalability, it makes it much harder for developers to estimate the system's behavior and performance under load due to the decoupling of components and control flow. We present an overview on our approach enabling the modeling and performance prediction of event-based system at the architecture level. Applying a model-to-model transformation, our approach integrates platform-specific performance influences of the underlying middleware while enabling the use of different existing analytical and simulation-based prediction techniques. The results of two real world case studies demonstrate the effectiveness, practicability and accuracy of the proposed modeling and prediction approach.

  @article{KoRaKl2012-FESCA-Keynote, abstract = {Event-based communication is used in different domains including telecommunications, transportation, and business information systems to build scalable distributed systems. Such systems typically have stringent requirements for performance and scalability as they provide business and mission critical services. While the use of event-based communication enables loosely-coupled interactions between components and leads to improved system scalability, it makes it much harder for developers to estimate the system's behavior and performance under load due to the decoupling of components and control flow. We present an overview on our approach enabling the modeling and performance prediction of event-based system at the architecture level. Applying a model-to-model transformation, our approach integrates platform-specific performance influences of the underlying middleware while enabling the use of different existing analytical and simulation-based prediction techniques. The results of two real world case studies demonstrate the effectiveness, practicability and accuracy of the proposed modeling and prediction approach.}, address = {Amsterdam, The Netherlands}, author = {Kounev, Samuel and Rathfelder, Christoph and Klatt, Benjamin}, journal = {{Electronic Notes in Theoretical Computer Science (ENTCS)}}, keywords = {myown}, month = {may}, pages = {3–9}, publisher = {Elsevier Science Publishers B. V.}, title = {{Modeling of Event-based Communication in Component-based Architectures: State-of-the-Art and Future Directions}}, volume = 295, year = 2013 }

  %0 Journal Article %1 KoRaKl2012-FESCA-Keynote %A Kounev, Samuel %A Rathfelder, Christoph %A Klatt, Benjamin %C Amsterdam, The Netherlands %D 2013 %I Elsevier Science Publishers B. V. %J Electronic Notes in Theoretical Computer Science (ENTCS) %P 3--9 %T Modeling of Event-based Communication in Component-based Architectures: State-of-the-Art and Future Directions %U http://www.sciencedirect.com/science/article/pii/S1571066113000248 %V 295 %X Event-based communication is used in different domains including telecommunications, transportation, and business information systems to build scalable distributed systems. Such systems typically have stringent requirements for performance and scalability as they provide business and mission critical services. While the use of event-based communication enables loosely-coupled interactions between components and leads to improved system scalability, it makes it much harder for developers to estimate the system's behavior and performance under load due to the decoupling of components and control flow. We present an overview on our approach enabling the modeling and performance prediction of event-based system at the architecture level. Applying a model-to-model transformation, our approach integrates platform-specific performance influences of the underlying middleware while enabling the use of different existing analytical and simulation-based prediction techniques. The results of two real world case studies demonstrate the effectiveness, practicability and accuracy of the proposed modeling and prediction approach.

• 
  Self-Aware Software and Systems Engineering: A Vision and Research Roadmap Kounev, Samuel; in GI Softwaretechnik-Trends, 31(4), November 2011, ISSN 0720-8928 (2011). Karlsruhe, Germany.
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  @inproceedings{Ko2011-SE-DescartesResearch, address = {Karlsruhe, Germany}, author = {Kounev, Samuel}, booktitle = {{GI Softwaretechnik-Trends, 31(4), November 2011, ISSN 0720-8928}}, keywords = {myown}, title = {{Self-Aware Software and Systems Engineering: A Vision and Research Roadmap}}, year = 2011 }

  %0 Conference Paper %1 Ko2011-SE-DescartesResearch %A Kounev, Samuel %B GI Softwaretechnik-Trends, 31(4), November 2011, ISSN 0720-8928 %C Karlsruhe, Germany %D 2011 %T Self-Aware Software and Systems Engineering: A Vision and Research Roadmap %U http://pi.informatik.uni-siegen.de/stt/31_4/index.html

• 
  Benchmarking and Performance Modeling of Event-Based Systems Kounev, Samuel; Sachs, Kai; in it - Information Technology (2009). 51(5) 262–269. Oldenbourg Wissenschaftsverlag, Munich, Germany.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  Event-based systems are used increasingly often to build loosely-coupled distributed applications. With their growing popularity and gradual adoption in mission critical areas, the need for novel techniques for benchmarking and performance modeling of event-based systems is increasing. In this article, we provide an overview of the state-of-the-art in this area considering both centralized systems based on message-oriented middleware as well as large-scale distributed publish/subscribe systems. We consider a number of specific techniques for benchmarking and performance modeling, discuss their advantages and disadvantages, and provide references for further information. The techniques we review help to ensure that systems are designed and sized to meet their quality-of-service requirements.

  @article{KoSa2009-DEBSPerformance, abstract = {Event-based systems are used increasingly often to build loosely-coupled distributed applications. With their growing popularity and gradual adoption in mission critical areas, the need for novel techniques for benchmarking and performance modeling of event-based systems is increasing. In this article, we provide an overview of the state-of-the-art in this area considering both centralized systems based on message-oriented middleware as well as large-scale distributed publish/subscribe systems. We consider a number of specific techniques for benchmarking and performance modeling, discuss their advantages and disadvantages, and provide references for further information. The techniques we review help to ensure that systems are designed and sized to meet their quality-of-service requirements.}, address = {Munich, Germany}, author = {Kounev, Samuel and Sachs, Kai}, journal = {it - Information Technology}, keywords = {myown}, month = {October}, number = 5, pages = {262–269}, publisher = {Oldenbourg Wissenschaftsverlag}, title = {{Benchmarking and Performance Modeling of Event-Based Systems}}, volume = 51, year = 2009 }

  %0 Journal Article %1 KoSa2009-DEBSPerformance %A Kounev, Samuel %A Sachs, Kai %C Munich, Germany %D 2009 %I Oldenbourg Wissenschaftsverlag %J it - Information Technology %N 5 %P 262--269 %T Benchmarking and Performance Modeling of Event-Based Systems %V 51 %X Event-based systems are used increasingly often to build loosely-coupled distributed applications. With their growing popularity and gradual adoption in mission critical areas, the need for novel techniques for benchmarking and performance modeling of event-based systems is increasing. In this article, we provide an overview of the state-of-the-art in this area considering both centralized systems based on message-oriented middleware as well as large-scale distributed publish/subscribe systems. We consider a number of specific techniques for benchmarking and performance modeling, discuss their advantages and disadvantages, and provide references for further information. The techniques we review help to ensure that systems are designed and sized to meet their quality-of-service requirements.
• 
  Performance evaluation of message-oriented middleware using the SPECjms2007 benchmark Sachs, Kai; Kounev, Samuel; Bacon, Jean; Buchmann, Alejandro; in Performance Evaluation (2009). 66(8) 410–434. Elsevier Science Publishers B. V., Amsterdam, The Netherlands.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Message-oriented middleware (MOM) is at the core of a vast number of financial services and telco applications, and is gaining increasing traction in other industries, such as manufacturing, transportation, health-care and supply chain management. Novel messaging applications, however, pose some serious performance and scalability challenges. In this paper, we present a methodology for performance evaluation of MOM platforms using the SPECjms2007 benchmark which is the world's first industry-standard benchmark specialized for MOM. SPECjms2007 is based on a novel application in the supply chain management domain designed to stress MOM infrastructures in a manner representative of real-world applications. In addition to providing a standard workload and metrics for MOM performance, the benchmark provides a flexible performance analysis framework that allows users to tailor the workload to their requirements. The contributions of this paper are: i) we present a detailed workload characterization of SPECjms2007 with the goal to help users understand the internal components of the workload and the way they are scaled, ii) we show how the workload can be customized to exercise and evaluate selected aspects of MOM performance, iii) we present a case study of a leading JMS platform, the BEA WebLogic server, conducting an in-depth performance analysis of the platform under a number of different workload and configuration scenarios. The methodology we propose is the first one that uses an industry-standard benchmark providing both a representative workload as well as the ability to customize it to evaluate the features of MOM platforms selectively.

  @article{SaKoBaBu2008-PERFEVAL-SPECjms2007, abstract = {Message-oriented middleware (MOM) is at the core of a vast number of financial services and telco applications, and is gaining increasing traction in other industries, such as manufacturing, transportation, health-care and supply chain management. Novel messaging applications, however, pose some serious performance and scalability challenges. In this paper, we present a methodology for performance evaluation of MOM platforms using the SPECjms2007 benchmark which is the world's first industry-standard benchmark specialized for MOM. SPECjms2007 is based on a novel application in the supply chain management domain designed to stress MOM infrastructures in a manner representative of real-world applications. In addition to providing a standard workload and metrics for MOM performance, the benchmark provides a flexible performance analysis framework that allows users to tailor the workload to their requirements. The contributions of this paper are: i) we present a detailed workload characterization of SPECjms2007 with the goal to help users understand the internal components of the workload and the way they are scaled, ii) we show how the workload can be customized to exercise and evaluate selected aspects of MOM performance, iii) we present a case study of a leading JMS platform, the BEA WebLogic server, conducting an in-depth performance analysis of the platform under a number of different workload and configuration scenarios. The methodology we propose is the first one that uses an industry-standard benchmark providing both a representative workload as well as the ability to customize it to evaluate the features of MOM platforms selectively.}, address = {Amsterdam, The Netherlands}, author = {Sachs, Kai and Kounev, Samuel and Bacon, Jean and Buchmann, Alejandro}, journal = {Performance Evaluation}, keywords = {myown}, month = {August}, number = 8, pages = {410–434}, publisher = {Elsevier Science Publishers B. V.}, title = {{Performance evaluation of message-oriented middleware using the SPECjms2007 benchmark}}, volume = 66, year = 2009 }

  %0 Journal Article %1 SaKoBaBu2008-PERFEVAL-SPECjms2007 %A Sachs, Kai %A Kounev, Samuel %A Bacon, Jean %A Buchmann, Alejandro %C Amsterdam, The Netherlands %D 2009 %I Elsevier Science Publishers B. V. %J Performance Evaluation %N 8 %P 410--434 %T Performance evaluation of message-oriented middleware using the SPECjms2007 benchmark %U http://www.elsevier.com/wps/find/journaldescription.cws_home/505618/description %V 66 %X Message-oriented middleware (MOM) is at the core of a vast number of financial services and telco applications, and is gaining increasing traction in other industries, such as manufacturing, transportation, health-care and supply chain management. Novel messaging applications, however, pose some serious performance and scalability challenges. In this paper, we present a methodology for performance evaluation of MOM platforms using the SPECjms2007 benchmark which is the world's first industry-standard benchmark specialized for MOM. SPECjms2007 is based on a novel application in the supply chain management domain designed to stress MOM infrastructures in a manner representative of real-world applications. In addition to providing a standard workload and metrics for MOM performance, the benchmark provides a flexible performance analysis framework that allows users to tailor the workload to their requirements. The contributions of this paper are: i) we present a detailed workload characterization of SPECjms2007 with the goal to help users understand the internal components of the workload and the way they are scaled, ii) we show how the workload can be customized to exercise and evaluate selected aspects of MOM performance, iii) we present a case study of a leading JMS platform, the BEA WebLogic server, conducting an in-depth performance analysis of the platform under a number of different workload and configuration scenarios. The methodology we propose is the first one that uses an industry-standard benchmark providing both a representative workload as well as the ability to customize it to evaluate the features of MOM platforms selectively.
• 
  Autonomic QoS control in enterprise Grid environments using online simulation Nou, Ramon; Kounev, Samuel; Julia, Ferran; Torres, Jordi; in Journal of Systems and Software (2009). 82(3) 486–502. Elsevier Science Publishers B. V., Amsterdam, The Netherlands.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  As Grid Computing increasingly enters the commercial domain, performance and Quality of Service (QoS) issues are becoming a major concern. The inherent complexity, heterogeneity and dynamics of Grid computing environments pose some challenges in managing their capacity to ensure that QoS requirements are continuously met. In this paper, a comprehensive framework for autonomic QoS control in enterprise Grid environments using online simulation is proposed. The paper presents a novel methodology for designing autonomic QoS-aware resource managers that have the capability to predict the performance of the Grid components they manage and allocate resources in such a way that service level agreements are honored. Support for advanced features such as autonomic workload characterization on-the-fly, dynamic deployment of Grid servers on demand, as well as dynamic system reconfiguration after a server failure is provided. The goal is to make the Grid middleware self-configurable and adaptable to changes in the system environment and workload. The approach is subjected to an extensive experimental evaluation in the context of a real-world Grid environment and its effectiveness, practicality and performance are demonstrated.

  @article{NoKoJuTo2008-JSS-GridAutoQoS, abstract = {As Grid Computing increasingly enters the commercial domain, performance and Quality of Service (QoS) issues are becoming a major concern. The inherent complexity, heterogeneity and dynamics of Grid computing environments pose some challenges in managing their capacity to ensure that QoS requirements are continuously met. In this paper, a comprehensive framework for autonomic QoS control in enterprise Grid environments using online simulation is proposed. The paper presents a novel methodology for designing autonomic QoS-aware resource managers that have the capability to predict the performance of the Grid components they manage and allocate resources in such a way that service level agreements are honored. Support for advanced features such as autonomic workload characterization on-the-fly, dynamic deployment of Grid servers on demand, as well as dynamic system reconfiguration after a server failure is provided. The goal is to make the Grid middleware self-configurable and adaptable to changes in the system environment and workload. The approach is subjected to an extensive experimental evaluation in the context of a real-world Grid environment and its effectiveness, practicality and performance are demonstrated.}, address = {Amsterdam, The Netherlands}, author = {Nou, Ramon and Kounev, Samuel and Julia, Ferran and Torres, Jordi}, journal = {Journal of Systems and Software}, keywords = {myown}, month = {March}, number = 3, pages = {486–502}, publisher = {Elsevier Science Publishers B. V.}, title = {{Autonomic QoS control in enterprise Grid environments using online simulation}}, volume = 82, year = 2009 }

  %0 Journal Article %1 NoKoJuTo2008-JSS-GridAutoQoS %A Nou, Ramon %A Kounev, Samuel %A Julia, Ferran %A Torres, Jordi %C Amsterdam, The Netherlands %D 2009 %I Elsevier Science Publishers B. V. %J Journal of Systems and Software %N 3 %P 486--502 %T Autonomic QoS control in enterprise Grid environments using online simulation %V 82 %X As Grid Computing increasingly enters the commercial domain, performance and Quality of Service (QoS) issues are becoming a major concern. The inherent complexity, heterogeneity and dynamics of Grid computing environments pose some challenges in managing their capacity to ensure that QoS requirements are continuously met. In this paper, a comprehensive framework for autonomic QoS control in enterprise Grid environments using online simulation is proposed. The paper presents a novel methodology for designing autonomic QoS-aware resource managers that have the capability to predict the performance of the Grid components they manage and allocate resources in such a way that service level agreements are honored. Support for advanced features such as autonomic workload characterization on-the-fly, dynamic deployment of Grid servers on demand, as well as dynamic system reconfiguration after a server failure is provided. The goal is to make the Grid middleware self-configurable and adaptable to changes in the system environment and workload. The approach is subjected to an extensive experimental evaluation in the context of a real-world Grid environment and its effectiveness, practicality and performance are demonstrated.
• 
  QPME - A Performance Modeling Tool Based on Queueing Petri Nets Kounev, Samuel; Dutz, Christofer; in ACM SIGMETRICS Performance Evaluation Review (PER), Special Issue on Tools for Computer Performance Modeling and Reliability Analysis (2009). 36 46–51. ACM.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Queueing Petri nets are a powerful formalism that can be exploited for modeling distributed systems and analyzing their performance and scalability. By combining the modeling power and expressiveness of queueing networks and stochastic Petri nets, queueing Petri nets provide a number of advantages. In this paper, we present QPME (Queueing Petri net Modeling Environment) - a tool that supports the modeling and analysis of systems using queueing Petri nets. QPME provides an Eclipse-based editor for designing queueing Petri net models and a powerful simulation engine for analyzing the models. After presenting the tool, we discuss the ongoing work on the QPME project and the planned future enhancements of the tool.

  @article{KoDu2009-SIGMETRICS_PER-QPME, abstract = {Queueing Petri nets are a powerful formalism that can be exploited for modeling distributed systems and analyzing their performance and scalability. By combining the modeling power and expressiveness of queueing networks and stochastic Petri nets, queueing Petri nets provide a number of advantages. In this paper, we present QPME (Queueing Petri net Modeling Environment) - a tool that supports the modeling and analysis of systems using queueing Petri nets. QPME provides an Eclipse-based editor for designing queueing Petri net models and a powerful simulation engine for analyzing the models. After presenting the tool, we discuss the ongoing work on the QPME project and the planned future enhancements of the tool.}, author = {Kounev, Samuel and Dutz, Christofer}, journal = {ACM SIGMETRICS Performance Evaluation Review (PER), Special Issue on Tools for Computer Performance Modeling and Reliability Analysis}, keywords = {myown}, month = 3, pages = {46–51}, publisher = {ACM}, title = {{QPME - A Performance Modeling Tool Based on Queueing Petri Nets}}, volume = 36, year = 2009 }

  %0 Journal Article %1 KoDu2009-SIGMETRICS_PER-QPME %A Kounev, Samuel %A Dutz, Christofer %D 2009 %I ACM %J ACM SIGMETRICS Performance Evaluation Review (PER), Special Issue on Tools for Computer Performance Modeling and Reliability Analysis %P 46--51 %T QPME - A Performance Modeling Tool Based on Queueing Petri Nets %V 36 %X Queueing Petri nets are a powerful formalism that can be exploited for modeling distributed systems and analyzing their performance and scalability. By combining the modeling power and expressiveness of queueing networks and stochastic Petri nets, queueing Petri nets provide a number of advantages. In this paper, we present QPME (Queueing Petri net Modeling Environment) - a tool that supports the modeling and analysis of systems using queueing Petri nets. QPME provides an Eclipse-based editor for designing queueing Petri net models and a powerful simulation engine for analyzing the models. After presenting the tool, we discuss the ongoing work on the QPME project and the planned future enhancements of the tool.

• 
  Kaffeekunde - SPECjms misst Message-oriented Middleware Kounev, Samuel; Sachs, Kai; (2008).
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  @misc{sachs2008a, author = {Kounev, Samuel and Sachs, Kai}, howpublished = {iX Magazin, Heft 02/2008, Heise Zeitschriften Verlag}, keywords = {myown}, title = {{Kaffeekunde - SPECjms misst Message-oriented Middleware}}, year = 2008 }

  %0 Generic %1 sachs2008a %A Kounev, Samuel %A Sachs, Kai %D 2008 %T Kaffeekunde - SPECjms misst Message-oriented Middleware %U http://www.heise.de/kiosk/archiv/ix/2008/2/121

• 
  SimQPN - a tool and methodology for analyzing queueing Petri net models by means of simulation Kounev, Samuel; Buchmann, Alejandro; in Performance Evaluation (2006). 63(4-5) 364–394. Elsevier Science Publishers B. V., Amsterdam, The Netherlands.
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @article{KoBu2006-PERFEVAL-SimQPN, address = {Amsterdam, The Netherlands}, author = {Kounev, Samuel and Buchmann, Alejandro}, journal = {Performance Evaluation}, keywords = {myown}, month = 5, number = {4-5}, pages = {364–394}, publisher = {Elsevier Science Publishers B. V.}, title = {{SimQPN - a tool and methodology for analyzing queueing Petri net models by means of simulation}}, volume = 63, year = 2006 }

  %0 Journal Article %1 KoBu2006-PERFEVAL-SimQPN %A Kounev, Samuel %A Buchmann, Alejandro %C Amsterdam, The Netherlands %D 2006 %I Elsevier Science Publishers B. V. %J Performance Evaluation %N 4-5 %P 364--394 %T SimQPN - a tool and methodology for analyzing queueing Petri net models by means of simulation %V 63
• 
  Performance Modeling and Evaluation of Distributed Component-Based Systems using Queueing Petri Nets Kounev, Samuel; in IEEE Transactions on Software Engineering (2006). 32(7) 486–502.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Performance models are used increasingly throughout the phases of the software engineering lifecycle of distributed component-based systems. However, as systems grow in size and complexity, building models that accurately capture the different aspects of their behavior becomes a more and more challenging task. In this paper, we present a novel case study of a realistic distributed component-based system, showing how Queueing Petri Net models can be exploited as a powerful performance prediction tool in the software engineering process. A detailed system model is built in a step-by-step fashion, validated, and then used to evaluate the system performance and scalability. Along with the case study, a practical performance modeling methodology is presented which helps to construct models that accurately reflect the system performance and scalability characteristics. Taking advantage of the modeling power and expressiveness of Queueing Petri Nets, our approach makes it possible to model the system at a higher degree of accuracy, providing a number of important benefits.

  @article{Ko2006-IEEE_TSE-QPN_ModelingMethod, abstract = {Performance models are used increasingly throughout the phases of the software engineering lifecycle of distributed component-based systems. However, as systems grow in size and complexity, building models that accurately capture the different aspects of their behavior becomes a more and more challenging task. In this paper, we present a novel case study of a realistic distributed component-based system, showing how Queueing Petri Net models can be exploited as a powerful performance prediction tool in the software engineering process. A detailed system model is built in a step-by-step fashion, validated, and then used to evaluate the system performance and scalability. Along with the case study, a practical performance modeling methodology is presented which helps to construct models that accurately reflect the system performance and scalability characteristics. Taking advantage of the modeling power and expressiveness of Queueing Petri Nets, our approach makes it possible to model the system at a higher degree of accuracy, providing a number of important benefits.}, author = {Kounev, Samuel}, journal = {IEEE Transactions on Software Engineering}, keywords = {myown}, month = {July}, number = 7, pages = {486–502}, title = {{Performance Modeling and Evaluation of Distributed Component-Based Systems using Queueing Petri Nets}}, volume = 32, year = 2006 }

  %0 Journal Article %1 Ko2006-IEEE_TSE-QPN_ModelingMethod %A Kounev, Samuel %D 2006 %J IEEE Transactions on Software Engineering %N 7 %P 486--502 %T Performance Modeling and Evaluation of Distributed Component-Based Systems using Queueing Petri Nets %V 32 %X Performance models are used increasingly throughout the phases of the software engineering lifecycle of distributed component-based systems. However, as systems grow in size and complexity, building models that accurately capture the different aspects of their behavior becomes a more and more challenging task. In this paper, we present a novel case study of a realistic distributed component-based system, showing how Queueing Petri Net models can be exploited as a powerful performance prediction tool in the software engineering process. A detailed system model is built in a step-by-step fashion, validated, and then used to evaluate the system performance and scalability. Along with the case study, a practical performance modeling methodology is presented which helps to construct models that accurately reflect the system performance and scalability characteristics. Taking advantage of the modeling power and expressiveness of Queueing Petri Nets, our approach makes it possible to model the system at a higher degree of accuracy, providing a number of important benefits.

• 
  Performance Tuning and Optimization of J2EE Applications on the JBoss Platform Kounev, Samuel; Weis, Börn; Buchmann, Alejandro; in Journal of Computer Resource Management (2004). 113 Computer Measurement Group (CMG).
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @article{KoWeBu2004-CMG_JCRM-JBoss, author = {Kounev, Samuel and Weis, B{\"o}rn and Buchmann, Alejandro}, journal = {Journal of Computer Resource Management}, keywords = {myown}, month = {September}, publisher = {Computer Measurement Group (CMG)}, title = {{Performance Tuning and Optimization of J2EE Applications on the JBoss Platform}}, volume = 113, year = 2004 }

  %0 Journal Article %1 KoWeBu2004-CMG_JCRM-JBoss %A Kounev, Samuel %A Weis, Börn %A Buchmann, Alejandro %D 2004 %I Computer Measurement Group (CMG) %J Journal of Computer Resource Management %T Performance Tuning and Optimization of J2EE Applications on the JBoss Platform %V 113

• 
  Distributed Video-Conferencing System Organized for Work on the Internet with the Use of Multimedia Server Nenov, Plamen; Kounev, Samuel; Mihailov, Dimiter; in Journal of Computing and Information (1999).
  • [ BibTeX ]
  • [ EndNote ]
  @article{NeKoMi1999-JCI-VideoConf, author = {Nenov, Plamen and Kounev, Samuel and Mihailov, Dimiter}, journal = {Journal of Computing and Information}, keywords = {myown}, title = {{Distributed Video-Conferencing System Organized for Work on the Internet with the Use of Multimedia Server}}, year = 1999 }

  %0 Journal Article %1 NeKoMi1999-JCI-VideoConf %A Nenov, Plamen %A Kounev, Samuel %A Mihailov, Dimiter %D 1999 %J Journal of Computing and Information %T Distributed Video-Conferencing System Organized for Work on the Internet with the Use of Multimedia Server

• [ 2023 ]
• [ 2022 ]
• [ 2021 ]
• [ 2020 ]
• [ 2019 ]
• [ 2018 ]
• [ 2017 ]
• [ 2016 ]
• [ 2015 ]
• [ 2014 ]
• [ 2013 ]
• [ 2012 ]
• [ 2011 ]
• [ 2010 ]
• [ 2009 ]
• [ 2008 ]
• [ 2007 ]
• [ 2003 ]
• [ 2002 ]
• [ 1999 ]

• 
  An Empirical Study of Container Image Configurations and Their Impact on Start Times Straesser, Martin; Bauer, André; Leppich, Robert; Herbst, Nikolas; Chard, Kyle; Foster, Ian; Kounev, Samuel; in 2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet Computing (CCGrid) (2023). IEEE.

  In print.
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @inproceedings{straesser2023starttime, author = {Straesser, Martin and Bauer, André and Leppich, Robert and Herbst, Nikolas and Chard, Kyle and Foster, Ian and Kounev, Samuel}, booktitle = {2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet Computing (CCGrid)}, keywords = {myown}, note = {In print.}, publisher = {IEEE}, title = {An Empirical Study of Container Image Configurations and Their Impact on Start Times}, year = 2023 }

  %0 Conference Paper %1 straesser2023starttime %A Straesser, Martin %A Bauer, André %A Leppich, Robert %A Herbst, Nikolas %A Chard, Kyle %A Foster, Ian %A Kounev, Samuel %B 2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet Computing (CCGrid) %D 2023 %I IEEE %T An Empirical Study of Container Image Configurations and Their Impact on Start Times
• 
  A Systematic Approach for Benchmarking of Container Orchestration Frameworks Straesser, Martin; Mathiasch, Jonas; Bauer, André; Kounev, Samuel; in Proceedings of the 2023 ACM/SPEC International Conference on Performance Engineering (2023). 187–198. Association for Computing Machinery, New York, NY, USA.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Container orchestration frameworks play a critical role in modern cloud computing paradigms such as cloud-native or serverless computing. They significantly impact the quality and cost of service deployment as they manage many performance-critical tasks such as container provisioning, scheduling, scaling, and networking. Consequently, a comprehensive performance assessment of container orchestration frameworks is essential. However, until now, there is no benchmarking approach that covers the many different tasks implemented in such platforms and supports evaluating different technology stacks. In this paper, we present a systematic approach that enables benchmarking of container orchestrators. Based on a definition of container orchestration, we define the core requirements and benchmarking scope for such platforms. Each requirement is then linked to metrics and measurement methods, and a benchmark architecture is proposed. With COFFEE, we introduce a benchmarking tool supporting the definition of complex test campaigns for container orchestration frameworks. We demonstrate the potential of our approach with case studies of the frameworks Kubernetes and Nomad in a self-hosted environment and on the Google Cloud Platform. The presented case studies focus on container startup times, crash recovery, rolling updates, and more.

  @inproceedings{straesser2023orchestration, abstract = {Container orchestration frameworks play a critical role in modern cloud computing paradigms such as cloud-native or serverless computing. They significantly impact the quality and cost of service deployment as they manage many performance-critical tasks such as container provisioning, scheduling, scaling, and networking. Consequently, a comprehensive performance assessment of container orchestration frameworks is essential. However, until now, there is no benchmarking approach that covers the many different tasks implemented in such platforms and supports evaluating different technology stacks. In this paper, we present a systematic approach that enables benchmarking of container orchestrators. Based on a definition of container orchestration, we define the core requirements and benchmarking scope for such platforms. Each requirement is then linked to metrics and measurement methods, and a benchmark architecture is proposed. With COFFEE, we introduce a benchmarking tool supporting the definition of complex test campaigns for container orchestration frameworks. We demonstrate the potential of our approach with case studies of the frameworks Kubernetes and Nomad in a self-hosted environment and on the Google Cloud Platform. The presented case studies focus on container startup times, crash recovery, rolling updates, and more.}, address = {New York, NY, USA}, author = {Straesser, Martin and Mathiasch, Jonas and Bauer, Andr\'{e} and Kounev, Samuel}, booktitle = {Proceedings of the 2023 ACM/SPEC International Conference on Performance Engineering}, keywords = {myown}, pages = {187-198}, publisher = {Association for Computing Machinery}, series = {ICPE '23}, title = {A Systematic Approach for Benchmarking of Container Orchestration Frameworks}, year = 2023 }

  %0 Conference Paper %1 straesser2023orchestration %A Straesser, Martin %A Mathiasch, Jonas %A Bauer, André %A Kounev, Samuel %B Proceedings of the 2023 ACM/SPEC International Conference on Performance Engineering %C New York, NY, USA %D 2023 %I Association for Computing Machinery %P 187-198 %T A Systematic Approach for Benchmarking of Container Orchestration Frameworks %U https://doi.org/10.1145/3578244.3583726 %X Container orchestration frameworks play a critical role in modern cloud computing paradigms such as cloud-native or serverless computing. They significantly impact the quality and cost of service deployment as they manage many performance-critical tasks such as container provisioning, scheduling, scaling, and networking. Consequently, a comprehensive performance assessment of container orchestration frameworks is essential. However, until now, there is no benchmarking approach that covers the many different tasks implemented in such platforms and supports evaluating different technology stacks. In this paper, we present a systematic approach that enables benchmarking of container orchestrators. Based on a definition of container orchestration, we define the core requirements and benchmarking scope for such platforms. Each requirement is then linked to metrics and measurement methods, and a benchmark architecture is proposed. With COFFEE, we introduce a benchmarking tool supporting the definition of complex test campaigns for container orchestration frameworks. We demonstrate the potential of our approach with case studies of the frameworks Kubernetes and Nomad in a self-hosted environment and on the Google Cloud Platform. The presented case studies focus on container startup times, crash recovery, rolling updates, and more.
• 
  Autoscaler Evaluation and Configuration: A Practitioner’s Guideline Straesser, Martin; Eismann, Simon; von Kistowski, Jóakim; Bauer, André; Kounev, Samuel; in Proceedings of the 2023 ACM/SPEC International Conference on Performance Engineering (2023). Association for Computing Machinery, New York, NY, USA.

  In print.
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @inproceedings{straesser2023guideline, address = {New York, NY, USA}, author = {Straesser, Martin and Eismann, Simon and von Kistowski, J\'{o}akim and Bauer, Andr\'{e} and Kounev, Samuel}, booktitle = {Proceedings of the 2023 ACM/SPEC International Conference on Performance Engineering}, keywords = {myown}, note = {In print.}, publisher = {Association for Computing Machinery}, series = {ICPE '23}, title = {Autoscaler Evaluation and Configuration: A Practitioner's Guideline}, year = 2023 }

  %0 Conference Paper %1 straesser2023guideline %A Straesser, Martin %A Eismann, Simon %A von Kistowski, Jóakim %A Bauer, André %A Kounev, Samuel %B Proceedings of the 2023 ACM/SPEC International Conference on Performance Engineering %C New York, NY, USA %D 2023 %I Association for Computing Machinery %T Autoscaler Evaluation and Configuration: A Practitioner's Guideline

• 
  Investigating the Predictability of QoS Metrics in Cellular Networks Herrnleben, Stefan; Grohmann, Johannes; Lesch, Veronika; Prantl, Thomas; Metzger, Florian; Hoßfeld, Tobias; Kounev, Samuel; in 2022 IEEE/ACM 30th International Symposium on Quality of Service (IWQoS) (2022). 1–10.
  • [ BibTeX ]
  • [ EndNote ]
  @inproceedings{HeGrLePrMeHoKo-2022-IWQoS-QoS-Prediction-Cellular-Networks, author = {Herrnleben, Stefan and Grohmann, Johannes and Lesch, Veronika and Prantl, Thomas and Metzger, Florian and Hoßfeld, Tobias and Kounev, Samuel}, booktitle = {2022 IEEE/ACM 30th International Symposium on Quality of Service (IWQoS)}, keywords = {myown}, pages = {1-10}, title = {{Investigating the Predictability of QoS Metrics in Cellular Networks}}, year = 2022 }

  %0 Conference Paper %1 HeGrLePrMeHoKo-2022-IWQoS-QoS-Prediction-Cellular-Networks %A Herrnleben, Stefan %A Grohmann, Johannes %A Lesch, Veronika %A Prantl, Thomas %A Metzger, Florian %A Hoßfeld, Tobias %A Kounev, Samuel %B 2022 IEEE/ACM 30th International Symposium on Quality of Service (IWQoS) %D 2022 %P 1-10 %T Investigating the Predictability of QoS Metrics in Cellular Networks
• 
  Why Is It Not Solved Yet? Challenges for Production-Ready Autoscaling Straesser, Martin; Grohmann, Johannes; von Kistowski, Jóakim; Eismann, Simon; Bauer, André; Kounev, Samuel; in Proceedings of the 2022 ACM/SPEC on International Conference on Performance Engineering (2022). 105–115. Association for Computing Machinery, New York, NY, USA.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Autoscaling is a task of major importance in the cloud computing domain as it directly affects both operating costs and customer experience. Although there has been active research in this area for over ten years now, there is still a significant gap between the proposed methods in the literature and the deployed autoscalers in practice. Hence, many research autoscalers do not find their way into production deployments. This paper describes six core challenges that arise in production systems that are still not solved by most research autoscalers. We illustrate these problems through experiments in a realistic cloud environment with a real-world multi-service business application and show that commonly used autoscalers have various shortcomings. In addition, we analyze the behavior of overloaded services and show that these can be problematic for existing autoscalers. Generally, we analyze that these challenges are only insufficiently addressed in the literature and conclude that future scaling approaches should focus on the needs of production systems.

  @inproceedings{10.1145/3489525.3511680, abstract = {Autoscaling is a task of major importance in the cloud computing domain as it directly affects both operating costs and customer experience. Although there has been active research in this area for over ten years now, there is still a significant gap between the proposed methods in the literature and the deployed autoscalers in practice. Hence, many research autoscalers do not find their way into production deployments. This paper describes six core challenges that arise in production systems that are still not solved by most research autoscalers. We illustrate these problems through experiments in a realistic cloud environment with a real-world multi-service business application and show that commonly used autoscalers have various shortcomings. In addition, we analyze the behavior of overloaded services and show that these can be problematic for existing autoscalers. Generally, we analyze that these challenges are only insufficiently addressed in the literature and conclude that future scaling approaches should focus on the needs of production systems.}, address = {New York, NY, USA}, author = {Straesser, Martin and Grohmann, Johannes and von Kistowski, J\'{o}akim and Eismann, Simon and Bauer, Andr\'{e} and Kounev, Samuel}, booktitle = {Proceedings of the 2022 ACM/SPEC on International Conference on Performance Engineering}, keywords = {myown}, pages = {105–115}, publisher = {Association for Computing Machinery}, series = {ICPE '22}, title = {Why Is It Not Solved Yet? Challenges for Production-Ready Autoscaling}, year = 2022 }

  %0 Conference Paper %1 10.1145/3489525.3511680 %A Straesser, Martin %A Grohmann, Johannes %A von Kistowski, Jóakim %A Eismann, Simon %A Bauer, André %A Kounev, Samuel %B Proceedings of the 2022 ACM/SPEC on International Conference on Performance Engineering %C New York, NY, USA %D 2022 %I Association for Computing Machinery %P 105–115 %T Why Is It Not Solved Yet? Challenges for Production-Ready Autoscaling %U https://doi.org/10.1145/3489525.3511680 %X Autoscaling is a task of major importance in the cloud computing domain as it directly affects both operating costs and customer experience. Although there has been active research in this area for over ten years now, there is still a significant gap between the proposed methods in the literature and the deployed autoscalers in practice. Hence, many research autoscalers do not find their way into production deployments. This paper describes six core challenges that arise in production systems that are still not solved by most research autoscalers. We illustrate these problems through experiments in a realistic cloud environment with a real-world multi-service business application and show that commonly used autoscalers have various shortcomings. In addition, we analyze the behavior of overloaded services and show that these can be problematic for existing autoscalers. Generally, we analyze that these challenges are only insufficiently addressed in the literature and conclude that future scaling approaches should focus on the needs of production systems.
• 
  Towards a Cryptography Benchmark: A View on Attribute Based Encryption Schemes Prantl, Thomas; Zeck, Timo; Iffländer, Lukas; Beierlieb, Lukas; Dmitrenko, Alexandra; Krupitzer, Christian; Kounev, Samuel; in 2022 5th Conference on Cloud and Internet of Things (CIoT) (2022). IEEE.
  • [ BibTeX ]
  • [ EndNote ]
  @inproceedings{PrZeIfBeDmKrKo2022-CIoT-ABEBenchmark, author = {Prantl, Thomas and Zeck, Timo and Iffländer, Lukas and Beierlieb, Lukas and Dmitrenko, Alexandra and Krupitzer, Christian and Kounev, Samuel}, booktitle = {2022 5th Conference on Cloud and Internet of Things (CIoT)}, keywords = {myown}, organization = {IEEE}, title = {Towards a Cryptography Benchmark: A View on Attribute Based Encryption Schemes}, year = 2022 }

  %0 Conference Paper %1 PrZeIfBeDmKrKo2022-CIoT-ABEBenchmark %A Prantl, Thomas %A Zeck, Timo %A Iffländer, Lukas %A Beierlieb, Lukas %A Dmitrenko, Alexandra %A Krupitzer, Christian %A Kounev, Samuel %B 2022 5th Conference on Cloud and Internet of Things (CIoT) %D 2022 %T Towards a Cryptography Benchmark: A View on Attribute Based Encryption Schemes

• 
  Towards Situation-Aware Meta-Optimization of Adaptation Planning Strategies Lesch, Veronika; Noack, Tanja; Hefter, Johannes; Kounev, Samuel; Krupitzer, Christian; in Proceedings of the 2nd IEEE International Conference on Autonomic Computing and Self-Organizing Systems - ACSOS 2021 (2021). IEEE.

  Best Paper Award Runner Up, Full Paper Acceptance Ratio: 25%
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @inproceedings{lesch2021towards, author = {Lesch, Veronika and Noack, Tanja and Hefter, Johannes and Kounev, Samuel and Krupitzer, Christian}, booktitle = {Proceedings of the 2nd IEEE International Conference on Autonomic Computing and Self-Organizing Systems - ACSOS 2021}, keywords = {myown}, note = {Best Paper Award Runner Up, Full Paper Acceptance Ratio: 25%}, publisher = {IEEE}, title = {Towards Situation-Aware Meta-Optimization of Adaptation Planning Strategies}, year = 2021 }

  %0 Conference Paper %1 lesch2021towards %A Lesch, Veronika %A Noack, Tanja %A Hefter, Johannes %A Kounev, Samuel %A Krupitzer, Christian %B Proceedings of the 2nd IEEE International Conference on Autonomic Computing and Self-Organizing Systems - ACSOS 2021 %D 2021 %I IEEE %T Towards Situation-Aware Meta-Optimization of Adaptation Planning Strategies
• 
  Libra: A Benchmark for Time Series Forecasting Methods Bauer, André; Züfle, Marwin; Eismann, Simon; Grohmann, Johannes; Herbst, Nikolas; Kounev, Samuel; in Proceedings of the 12th ACM/SPEC International Conference on Performance Engineering (ICPE) (2021). ACM, New York, NY, USA.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  • [ Download ]
  In many areas of decision making, forecasting is an essential pillar. Consequently, there are many different forecasting methods. According to the "No-Free-Lunch Theorem", there is no single forecasting method that performs best for all time series. In other words, each method has its advantages and disadvantages depending on the specific use case. Therefore, the choice of the forecasting method remains a mandatory expert task. However, expert knowledge cannot be fully automated. To establish a level playing field for evaluating the performance of time series forecasting methods in a broad setting, we propose Libra, a forecasting benchmark that automatically evaluates and ranks forecasting methods based on their performance in a diverse set of evaluation scenarios. The benchmark comprises four different use cases, each covering 100 heterogeneous time series taken from different domains. The data set was assembled from publicly available time series and was designed to exhibit much higher diversity than existing forecasting competitions. Based on this benchmark, we perform a comprehensive evaluation to compare different existing time series forecasting methods.

  @inproceedings{bauer2021benchmark, abstract = {In many areas of decision making, forecasting is an essential pillar. Consequently, there are many different forecasting methods. According to the "No-Free-Lunch Theorem", there is no single forecasting method that performs best for all time series. In other words, each method has its advantages and disadvantages depending on the specific use case. Therefore, the choice of the forecasting method remains a mandatory expert task. However, expert knowledge cannot be fully automated. To establish a level playing field for evaluating the performance of time series forecasting methods in a broad setting, we propose Libra, a forecasting benchmark that automatically evaluates and ranks forecasting methods based on their performance in a diverse set of evaluation scenarios. The benchmark comprises four different use cases, each covering 100 heterogeneous time series taken from different domains. The data set was assembled from publicly available time series and was designed to exhibit much higher diversity than existing forecasting competitions. Based on this benchmark, we perform a comprehensive evaluation to compare different existing time series forecasting methods.}, address = {New York, NY, USA}, author = {Bauer, Andr{\'e} and Z{\"u}fle, Marwin and Eismann, Simon and Grohmann, Johannes and Herbst, Nikolas and Kounev, Samuel}, booktitle = {Proceedings of the 12th ACM/SPEC International Conference on Performance Engineering (ICPE)}, keywords = {myown}, month = {April}, organization = {ACM}, title = {Libra: A Benchmark for Time Series Forecasting Methods}, year = 2021 }

  %0 Conference Paper %1 bauer2021benchmark %A Bauer, André %A Züfle, Marwin %A Eismann, Simon %A Grohmann, Johannes %A Herbst, Nikolas %A Kounev, Samuel %B Proceedings of the 12th ACM/SPEC International Conference on Performance Engineering (ICPE) %C New York, NY, USA %D 2021 %R 10.1145/3427921.3450241 %T Libra: A Benchmark for Time Series Forecasting Methods %U https://dl.acm.org/doi/10.1145/3427921.3450241 %X In many areas of decision making, forecasting is an essential pillar. Consequently, there are many different forecasting methods. According to the "No-Free-Lunch Theorem", there is no single forecasting method that performs best for all time series. In other words, each method has its advantages and disadvantages depending on the specific use case. Therefore, the choice of the forecasting method remains a mandatory expert task. However, expert knowledge cannot be fully automated. To establish a level playing field for evaluating the performance of time series forecasting methods in a broad setting, we propose Libra, a forecasting benchmark that automatically evaluates and ranks forecasting methods based on their performance in a diverse set of evaluation scenarios. The benchmark comprises four different use cases, each covering 100 heterogeneous time series taken from different domains. The data set was assembled from publicly available time series and was designed to exhibit much higher diversity than existing forecasting competitions. Based on this benchmark, we perform a comprehensive evaluation to compare different existing time series forecasting methods.
• 
  SuanMing: Explainable Prediction of Performance Degradations in Microservice Applications Grohmann, Johannes; Straesser, Martin; Chalbani, Avi; Eismann, Simon; Arian, Yair; Herbst, Nikolas; Peretz, Noam; Kounev, Samuel; in Proceedings of the 12th ACM/SPEC International Conference on Performance Engineering (ICPE) (2021). ACM, New York, NY, USA.

  Acceptance Rate: 29%
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  • [ Download ]
  Application performance management (APM) tools are useful to observe the performance properties of an application during production. However, APM is normally purely reactive, that is, it can only report about current or past performance degradation. Although some approaches capable of predictive application monitoring have been proposed, they can only report a predicted degradation but cannot explain its root-cause, making it hard to prevent the expected degradation. In this paper, we present SuanMing---a framework for predicting performance degradation of microservice applications running in cloud environments. SuanMing is able to predict future root causes for anticipated performance degradations and therefore aims at preventing performance degradations before they actually occur. We evaluate SuanMing on two realistic microservice applications, TeaStore and TrainTicket, and we show that our approach is able to predict and pinpoint performance degradations with an accuracy of over 90%.

  @inproceedings{GrStChEiArHePeKo2021-ICPE, abstract = {Application performance management (APM) tools are useful to observe the performance properties of an application during production. However, APM is normally purely reactive, that is, it can only report about current or past performance degradation. Although some approaches capable of predictive application monitoring have been proposed, they can only report a predicted degradation but cannot explain its root-cause, making it hard to prevent the expected degradation. In this paper, we present SuanMing—a framework for predicting performance degradation of microservice applications running in cloud environments. SuanMing is able to predict future root causes for anticipated performance degradations and therefore aims at preventing performance degradations before they actually occur. We evaluate SuanMing on two realistic microservice applications, TeaStore and TrainTicket, and we show that our approach is able to predict and pinpoint performance degradations with an accuracy of over 90\%.}, address = {New York, NY, USA}, author = {Grohmann, Johannes and Straesser, Martin and Chalbani, Avi and Eismann, Simon and Arian, Yair and Herbst, Nikolas and Peretz, Noam and Kounev, Samuel}, booktitle = {Proceedings of the 12th ACM/SPEC International Conference on Performance Engineering (ICPE)}, keywords = {myown}, month = {April}, note = {Acceptance Rate: 29%}, organization = {ACM}, title = {SuanMing: Explainable Prediction of Performance Degradations in Microservice Applications}, year = 2021 }

  %0 Conference Paper %1 GrStChEiArHePeKo2021-ICPE %A Grohmann, Johannes %A Straesser, Martin %A Chalbani, Avi %A Eismann, Simon %A Arian, Yair %A Herbst, Nikolas %A Peretz, Noam %A Kounev, Samuel %B Proceedings of the 12th ACM/SPEC International Conference on Performance Engineering (ICPE) %C New York, NY, USA %D 2021 %R https://doi.org/10.1145/3427921.3450248 %T SuanMing: Explainable Prediction of Performance Degradations in Microservice Applications %U https://doi.org/10.1145/3427921.3450248 %X Application performance management (APM) tools are useful to observe the performance properties of an application during production. However, APM is normally purely reactive, that is, it can only report about current or past performance degradation. Although some approaches capable of predictive application monitoring have been proposed, they can only report a predicted degradation but cannot explain its root-cause, making it hard to prevent the expected degradation. In this paper, we present SuanMing---a framework for predicting performance degradation of microservice applications running in cloud environments. SuanMing is able to predict future root causes for anticipated performance degradations and therefore aims at preventing performance degradations before they actually occur. We evaluate SuanMing on two realistic microservice applications, TeaStore and TrainTicket, and we show that our approach is able to predict and pinpoint performance degradations with an accuracy of over 90%.
• 
  A Predictive Maintenance Methodology: Predicting the Time-to-Failure of Machines in Industry 4.0 Züfle, Marwin; Agne, Joachim; Grohmann, Johannes; Dörtoluk, Ibrahim; Kounev, Samuel; in Proceedings of the 21st IEEE IES International Conference on Industrial Informatics (2021). IEEE.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  • [ Download ]
  Predictive maintenance is an essential aspect of the concept of Industry 4.0. In contrast to previous maintenance strategies, which plan repairs based on periodic schedules or threshold values, predictive maintenance is normally based on estimating the time-to-failure of machines. Thus, predictive maintenance enables a more efficient and effective maintenance approach. Although much research has already been done on time-to-failure prediction, most existing works provide only specialized approaches for specific machines. In most cases, these are either rotary machines (i.e., bearings) or lithium-ion batteries. To bridge the gap to a more general time-to-failure prediction, we propose a generic end-to-end predictive maintenance methodology for the time-to-failure prediction of industrial machines. Our methodology exhibits a number of novel aspects including a universally applicable method for feature extraction based on different types of sensor data, well-known feature transformation and selection techniques, adjustable target class assignment based on fault records with three different labeling strategies, and the training of multiple state-of-the-art machine learning classification models including hyperparameter optimization. We evaluated our time-to-failure prediction methodology in a real-world case study consisting of monitoring data gathered over several years from a large industrial press. The results demonstrated the effectiveness of the proposed methodology for six different time-to-failure pre-diction windows, as well as for the downscaled binary prediction of impending failures. In this case study, the multi-class feed-forward neural network model achieved the overall best results.

  @inproceedings{ZuAgGrDoKo-INDIN2021-PdM, abstract = {Predictive maintenance is an essential aspect of the concept of Industry 4.0. In contrast to previous maintenance strategies, which plan repairs based on periodic schedules or threshold values, predictive maintenance is normally based on estimating the time-to-failure of machines. Thus, predictive maintenance enables a more efficient and effective maintenance approach. Although much research has already been done on time-to-failure prediction, most existing works provide only specialized approaches for specific machines. In most cases, these are either rotary machines (i.e., bearings) or lithium-ion batteries. To bridge the gap to a more general time-to-failure prediction, we propose a generic end-to-end predictive maintenance methodology for the time-to-failure prediction of industrial machines. Our methodology exhibits a number of novel aspects including a universally applicable method for feature extraction based on different types of sensor data, well-known feature transformation and selection techniques, adjustable target class assignment based on fault records with three different labeling strategies, and the training of multiple state-of-the-art machine learning classification models including hyperparameter optimization. We evaluated our time-to-failure prediction methodology in a real-world case study consisting of monitoring data gathered over several years from a large industrial press. The results demonstrated the effectiveness of the proposed methodology for six different time-to-failure pre-diction windows, as well as for the downscaled binary prediction of impending failures. In this case study, the multi-class feed-forward neural network model achieved the overall best results.}, author = {Züfle, Marwin and Agne, Joachim and Grohmann, Johannes and Dörtoluk, Ibrahim and Kounev, Samuel}, booktitle = {Proceedings of the 21st IEEE IES International Conference on Industrial Informatics}, keywords = {myown}, month = {July}, publisher = {IEEE}, series = {INDIN'21}, title = {A Predictive Maintenance Methodology: Predicting the Time-to-Failure of Machines in Industry 4.0}, year = 2021 }

  %0 Conference Paper %1 ZuAgGrDoKo-INDIN2021-PdM %A Züfle, Marwin %A Agne, Joachim %A Grohmann, Johannes %A Dörtoluk, Ibrahim %A Kounev, Samuel %B Proceedings of the 21st IEEE IES International Conference on Industrial Informatics %D 2021 %I IEEE %R 10.1109/INDIN45523.2021.9557387 %T A Predictive Maintenance Methodology: Predicting the Time-to-Failure of Machines in Industry 4.0 %U https://ieeexplore.ieee.org/abstract/document/9557387 %X Predictive maintenance is an essential aspect of the concept of Industry 4.0. In contrast to previous maintenance strategies, which plan repairs based on periodic schedules or threshold values, predictive maintenance is normally based on estimating the time-to-failure of machines. Thus, predictive maintenance enables a more efficient and effective maintenance approach. Although much research has already been done on time-to-failure prediction, most existing works provide only specialized approaches for specific machines. In most cases, these are either rotary machines (i.e., bearings) or lithium-ion batteries. To bridge the gap to a more general time-to-failure prediction, we propose a generic end-to-end predictive maintenance methodology for the time-to-failure prediction of industrial machines. Our methodology exhibits a number of novel aspects including a universally applicable method for feature extraction based on different types of sensor data, well-known feature transformation and selection techniques, adjustable target class assignment based on fault records with three different labeling strategies, and the training of multiple state-of-the-art machine learning classification models including hyperparameter optimization. We evaluated our time-to-failure prediction methodology in a real-world case study consisting of monitoring data gathered over several years from a large industrial press. The results demonstrated the effectiveness of the proposed methodology for six different time-to-failure pre-diction windows, as well as for the downscaled binary prediction of impending failures. In this case study, the multi-class feed-forward neural network model achieved the overall best results.
• 
  Recommendations for Data-Driven Degradation Estimation with Case Studies from Manufacturing and Dry-Bulk Shipping Finke, Nils; Mohr, Marisa; Lontke, Alexander; Züfle, Marwin; Kounev, Samuel; Möller, Ralf; in Research Challenges in Information Science (RCIS) (2021). 189–204. Springer.
  • [ BibTeX ]
  • [ EndNote ]
  @inproceedings{finke2021recommendations, author = {Finke, Nils and Mohr, Marisa and Lontke, Alexander and Z{\"u}fle, Marwin and Kounev, Samuel and M{\"o}ller, Ralf}, booktitle = {Research Challenges in Information Science (RCIS)}, keywords = {myown}, organization = {Springer}, pages = {189–204}, title = {Recommendations for Data-Driven Degradation Estimation with Case Studies from Manufacturing and Dry-Bulk Shipping}, year = 2021 }

  %0 Conference Paper %1 finke2021recommendations %A Finke, Nils %A Mohr, Marisa %A Lontke, Alexander %A Züfle, Marwin %A Kounev, Samuel %A Möller, Ralf %B Research Challenges in Information Science (RCIS) %D 2021 %P 189--204 %T Recommendations for Data-Driven Degradation Estimation with Case Studies from Manufacturing and Dry-Bulk Shipping
• 
  A Simulation-based Optimization Framework for Online Adaptation of Networks Herrnleben, Stefan; Grohmann, Johannes; Rygielski, Pitor; Lesch, Veronika; Krupitzer, Christian; Kounev, Samuel; in Proceedings of the 12th EAI International Conference on Simulation Tools and Techniques (SIMUtools), H. Song, D. Jiang (eds.) (2021). 513–532. Springer International Publishing, Cham.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  • [ Download ]
  Today's data centers face a rapid change of deployed services, growing complexity, and increasing performance requirements. Customers expect not only round-the-clock availability of the hosted services but also high responsiveness. Besides optimizing software architectures and deployments, networks have to be adapted to handle the changing and volatile demands. Approaches from self-adaptive systems can be used for optimizing data center networks to continuously meet Service Level Agreements (SLAs) between data center operators and customers. However, existing approaches focus only on specific objectives like topology design, power optimization, or traffic engineering. In this paper, we present an extensible framework that analyzes networks using different types of simulation and adapts them subject to multiple objectives using various adaptation techniques. Analyzing each suggested adaptation ensures that performance requirements and SLAs are continuously met. We evaluate our framework w.r.t. (i) general requirements and assessments of languages and frameworks for adaptation models, (ii) finding Pareto-optimal solutions considering a multi-dimensional cost model, and (iii) scalability. The evaluation shows that our approach detects the bottlenecks and the violated SLAs correctly, outputs valid and cost-optimal adaptations, and keeps the runtime for the adaptation process constant even with increasing network size and an increasing number of alternative configurations.

  @inproceedings{HeGrRyLeKrKo2020-SIMUtools-Network-Online-Adaptation, abstract = {Today's data centers face a rapid change of deployed services, growing complexity, and increasing performance requirements. Customers expect not only round-the-clock availability of the hosted services but also high responsiveness. Besides optimizing software architectures and deployments, networks have to be adapted to handle the changing and volatile demands. Approaches from self-adaptive systems can be used for optimizing data center networks to continuously meet Service Level Agreements (SLAs) between data center operators and customers. However, existing approaches focus only on specific objectives like topology design, power optimization, or traffic engineering. In this paper, we present an extensible framework that analyzes networks using different types of simulation and adapts them subject to multiple objectives using various adaptation techniques. Analyzing each suggested adaptation ensures that performance requirements and SLAs are continuously met. We evaluate our framework w.r.t. (i) general requirements and assessments of languages and frameworks for adaptation models, (ii) finding Pareto-optimal solutions considering a multi-dimensional cost model, and (iii) scalability. The evaluation shows that our approach detects the bottlenecks and the violated SLAs correctly, outputs valid and cost-optimal adaptations, and keeps the runtime for the adaptation process constant even with increasing network size and an increasing number of alternative configurations.}, address = {Cham}, author = {Herrnleben, Stefan and Grohmann, Johannes and Rygielski, Pitor and Lesch, Veronika and Krupitzer, Christian and Kounev, Samuel}, booktitle = {Proceedings of the 12th EAI International Conference on Simulation Tools and Techniques (SIMUtools)}, editor = {Song, Houbing and Jiang, Dingde}, keywords = {t_full}, month = {August}, pages = {513–532}, publisher = {Springer International Publishing}, series = {SIMUtools 2020}, title = {A Simulation-based Optimization Framework for Online Adaptation of Networks}, year = 2021 }

  %0 Conference Paper %1 HeGrRyLeKrKo2020-SIMUtools-Network-Online-Adaptation %A Herrnleben, Stefan %A Grohmann, Johannes %A Rygielski, Pitor %A Lesch, Veronika %A Krupitzer, Christian %A Kounev, Samuel %B Proceedings of the 12th EAI International Conference on Simulation Tools and Techniques (SIMUtools) %C Cham %D 2021 %E Song, Houbing %E Jiang, Dingde %I Springer International Publishing %P 513--532 %R 10.1007/978-3-030-72792-5_41 %T A Simulation-based Optimization Framework for Online Adaptation of Networks %U https://doi.org/10.1007/978-3-030-72792-5_41 %X Today's data centers face a rapid change of deployed services, growing complexity, and increasing performance requirements. Customers expect not only round-the-clock availability of the hosted services but also high responsiveness. Besides optimizing software architectures and deployments, networks have to be adapted to handle the changing and volatile demands. Approaches from self-adaptive systems can be used for optimizing data center networks to continuously meet Service Level Agreements (SLAs) between data center operators and customers. However, existing approaches focus only on specific objectives like topology design, power optimization, or traffic engineering. In this paper, we present an extensible framework that analyzes networks using different types of simulation and adapts them subject to multiple objectives using various adaptation techniques. Analyzing each suggested adaptation ensures that performance requirements and SLAs are continuously met. We evaluate our framework w.r.t. (i) general requirements and assessments of languages and frameworks for adaptation models, (ii) finding Pareto-optimal solutions considering a multi-dimensional cost model, and (iii) scalability. The evaluation shows that our approach detects the bottlenecks and the violated SLAs correctly, outputs valid and cost-optimal adaptations, and keeps the runtime for the adaptation process constant even with increasing network size and an increasing number of alternative configurations. %@ 978-3-030-72792-5
• 
  Performance Evaluation of a Post-Quantum Public-Key Cryptosystem Prantl, Thomas; Prantl, Dominik; Beierlieb, Lukas; Iffländer, Lukas; Dmitrienko, Alexandra; Krupitzer, Christian; Kounev, Samuel; in 2021 IEEE 40th International Performance Computing and Communications Conference (IPCCC) (2021). IEEE.
  • [ BibTeX ]
  • [ EndNote ]
  @inproceedings{PrPrBeIfDnKrKo2021-IPCCC-QuantumPublicKeyCryposystem, author = {Prantl, Thomas and Prantl, Dominik and Beierlieb, Lukas and Iffländer, Lukas and Dmitrienko, Alexandra and Krupitzer, Christian and Kounev, Samuel}, booktitle = {2021 IEEE 40th International Performance Computing and Communications Conference (IPCCC)}, keywords = {myown}, organization = {IEEE}, title = {Performance Evaluation of a Post-Quantum Public-Key Cryptosystem}, year = 2021 }

  %0 Conference Paper %1 PrPrBeIfDnKrKo2021-IPCCC-QuantumPublicKeyCryposystem %A Prantl, Thomas %A Prantl, Dominik %A Beierlieb, Lukas %A Iffländer, Lukas %A Dmitrienko, Alexandra %A Krupitzer, Christian %A Kounev, Samuel %B 2021 IEEE 40th International Performance Computing and Communications Conference (IPCCC) %D 2021 %T Performance Evaluation of a Post-Quantum Public-Key Cryptosystem
• 
  Machine Learning Model Update Strategies for Hard Disk Drive Failure Prediction Züfle, Marwin; Erhard, Florian; Kounev, Samuel; in 20th IEEE International Conference on Machine Learning and Applications (ICMLA) (2021).

  (to appear)
  • [ BibTeX ]
  • [ EndNote ]
  @inproceedings{zufle2021machine, author = {Z{\"u}fle, Marwin and Erhard, Florian and Kounev, Samuel}, booktitle = {20th IEEE International Conference on Machine Learning and Applications (ICMLA)}, keywords = {myown}, note = {(to appear)}, title = {Machine Learning Model Update Strategies for Hard Disk Drive Failure Prediction}, year = 2021 }

  %0 Conference Paper %1 zufle2021machine %A Züfle, Marwin %A Erhard, Florian %A Kounev, Samuel %B 20th IEEE International Conference on Machine Learning and Applications (ICMLA) %D 2021 %T Machine Learning Model Update Strategies for Hard Disk Drive Failure Prediction
• 
  Sizeless: Predicting the Optimal Size of Serverless Functions Eismann, Simon; Bui, Long; Grohmann, Johannes; Abad, Cristina; Herbst, Nikolas; Kounev, Samuel; in Proceedings of the 22nd International MIDDLEWARE Conference (2021). 248–259.

  Best Student Paper Award, ACM Artifacts Evaluated — Functional
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  • [ Download ]
  Serverless functions are an emerging cloud computing paradigm that is being rapidly adopted by both industry and academia. In this cloud computing model, the provider opaquely handles resource management tasks such as resource provisioning, deployment, and auto-scaling. The only resource management task that developers are still in charge of is selecting how much resources are allocated to each worker instance. However, selecting the optimal size of serverless functions is quite challenging, so developers often neglect it despite its significant cost and performance benefits. Existing approaches aiming to automate serverless functions resource sizing require dedicated performance tests, which are time-consuming to implement and maintain. In this paper, we introduce an approach to predict the optimal resource size of a serverless function using monitoring data from a single resource size. As our approach does not require dedicated performance tests, it enables cloud providers to implement resource sizing on a platform level and automate the last resource management task associated with serverless functions. We evaluate our approach on four different serverless applications on AWS, where it predicts the execution time of the other memory sizes based on monitoring data for a single memory size with an average prediction error of 15.3%. Based on these predictions, it selects the optimal memory size for 79.0% of the serverless functions and the secondbest memory size for 12.3% of the serverless functions, which results in an average speedup of 39.7% while also decreasing average costs by 2.6%.

  @inproceedings{eismann2021sizeless, abstract = {Serverless functions are an emerging cloud computing paradigm that is being rapidly adopted by both industry and academia. In this cloud computing model, the provider opaquely handles resource management tasks such as resource provisioning, deployment, and auto-scaling. The only resource management task that developers are still in charge of is selecting how much resources are allocated to each worker instance. However, selecting the optimal size of serverless functions is quite challenging, so developers often neglect it despite its significant cost and performance benefits. Existing approaches aiming to automate serverless functions resource sizing require dedicated performance tests, which are time-consuming to implement and maintain. In this paper, we introduce an approach to predict the optimal resource size of a serverless function using monitoring data from a single resource size. As our approach does not require dedicated performance tests, it enables cloud providers to implement resource sizing on a platform level and automate the last resource management task associated with serverless functions. We evaluate our approach on four different serverless applications on AWS, where it predicts the execution time of the other memory sizes based on monitoring data for a single memory size with an average prediction error of 15.3%. Based on these predictions, it selects the optimal memory size for 79.0% of the serverless functions and the secondbest memory size for 12.3% of the serverless functions, which results in an average speedup of 39.7% while also decreasing average costs by 2.6%.}, author = {Eismann, Simon and Bui, Long and Grohmann, Johannes and Abad, Cristina and Herbst, Nikolas and Kounev, Samuel}, booktitle = {Proceedings of the 22nd International MIDDLEWARE Conference}, keywords = {myown}, note = {Best Student Paper Award, ACM Artifacts Evaluated — Functional}, pages = {248–259}, title = {Sizeless: Predicting the Optimal Size of Serverless Functions}, year = 2021 }

  %0 Conference Paper %1 eismann2021sizeless %A Eismann, Simon %A Bui, Long %A Grohmann, Johannes %A Abad, Cristina %A Herbst, Nikolas %A Kounev, Samuel %B Proceedings of the 22nd International MIDDLEWARE Conference %D 2021 %P 248–259 %R 10.1145/3464298.3493398 %T Sizeless: Predicting the Optimal Size of Serverless Functions %U https://dl.acm.org/doi/abs/10.1145/3464298.3493398 %X Serverless functions are an emerging cloud computing paradigm that is being rapidly adopted by both industry and academia. In this cloud computing model, the provider opaquely handles resource management tasks such as resource provisioning, deployment, and auto-scaling. The only resource management task that developers are still in charge of is selecting how much resources are allocated to each worker instance. However, selecting the optimal size of serverless functions is quite challenging, so developers often neglect it despite its significant cost and performance benefits. Existing approaches aiming to automate serverless functions resource sizing require dedicated performance tests, which are time-consuming to implement and maintain. In this paper, we introduce an approach to predict the optimal resource size of a serverless function using monitoring data from a single resource size. As our approach does not require dedicated performance tests, it enables cloud providers to implement resource sizing on a platform level and automate the last resource management task associated with serverless functions. We evaluate our approach on four different serverless applications on AWS, where it predicts the execution time of the other memory sizes based on monitoring data for a single memory size with an average prediction error of 15.3%. Based on these predictions, it selects the optimal memory size for 79.0% of the serverless functions and the secondbest memory size for 12.3% of the serverless functions, which results in an average speedup of 39.7% while also decreasing average costs by 2.6%.
• 
  ComBench: A Benchmarking Framework for Publish/Subscribe Communication Protocols Under Network Limitations Herrnleben, Stefan; Leidinger, Maximilian; Lesch, Veronika; Prantl, Thomas; Grohmann, Johannes; Krupitzer, Christian; Kounev, Samuel; in Performance Evaluation Methodologies and Tools, Q. Zhao, L. Xia (eds.) (2021). 72–92. Springer International Publishing, Cham.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  • [ Download ]
  Efficient and dependable communication is a highly relevant aspect for Internet of Things (IoT) systems in which tiny sensors, actuators, wearables, or other smart devices exchange messages. Various publish/subscribe protocols address the challenges of communication in IoT systems. The selection process of a suitable protocol should consider the communication behavior of the application, the protocol's performance, the resource requirements on the end device, and the network connection quality, as IoT environments often rely on wireless networks. Benchmarking is a common approach to evaluate and compare systems, considering the performance and aspects like dependability or security. In this paper, we present our IoT communication benchmarking framework ComBench for publish/subscribe protocols focusing on constrained networks with varying quality conditions. The benchmarking framework supports system designers, software engineers, and application developers to select and investigate the behavior of communication protocols. Our benchmarking framework contributes to (i) show the impact of fluctuating network quality on communication, (ii) compare multiple protocols, protocol features, and protocol implementations, and (iii) analyze scalability, robustness, and dependability of clients, networks, and brokers in different scenarios. Our case study demonstrates the applicability of our framework to support the decision for the best-suited protocol in various scenarios.

  @inproceedings{10.1007/978-3-030-92511-6_5, abstract = {Efficient and dependable communication is a highly relevant aspect for Internet of Things (IoT) systems in which tiny sensors, actuators, wearables, or other smart devices exchange messages. Various publish/subscribe protocols address the challenges of communication in IoT systems. The selection process of a suitable protocol should consider the communication behavior of the application, the protocol's performance, the resource requirements on the end device, and the network connection quality, as IoT environments often rely on wireless networks. Benchmarking is a common approach to evaluate and compare systems, considering the performance and aspects like dependability or security. In this paper, we present our IoT communication benchmarking framework ComBench for publish/subscribe protocols focusing on constrained networks with varying quality conditions. The benchmarking framework supports system designers, software engineers, and application developers to select and investigate the behavior of communication protocols. Our benchmarking framework contributes to (i) show the impact of fluctuating network quality on communication, (ii) compare multiple protocols, protocol features, and protocol implementations, and (iii) analyze scalability, robustness, and dependability of clients, networks, and brokers in different scenarios. Our case study demonstrates the applicability of our framework to support the decision for the best-suited protocol in various scenarios.}, address = {Cham}, author = {Herrnleben, Stefan and Leidinger, Maximilian and Lesch, Veronika and Prantl, Thomas and Grohmann, Johannes and Krupitzer, Christian and Kounev, Samuel}, booktitle = {Performance Evaluation Methodologies and Tools}, editor = {Zhao, Qianchuan and Xia, Li}, keywords = {myown}, pages = {72–92}, publisher = {Springer International Publishing}, title = {ComBench: A Benchmarking Framework for Publish/Subscribe Communication Protocols Under Network Limitations}, year = 2021 }

  %0 Conference Paper %1 10.1007/978-3-030-92511-6_5 %A Herrnleben, Stefan %A Leidinger, Maximilian %A Lesch, Veronika %A Prantl, Thomas %A Grohmann, Johannes %A Krupitzer, Christian %A Kounev, Samuel %B Performance Evaluation Methodologies and Tools %C Cham %D 2021 %E Zhao, Qianchuan %E Xia, Li %I Springer International Publishing %P 72--92 %R https://doi.org/10.1007/978-3-030-92511-6_5 %T ComBench: A Benchmarking Framework for Publish/Subscribe Communication Protocols Under Network Limitations %U https://link.springer.com/chapter/10.1007%2F978-3-030-92511-6_5 %X Efficient and dependable communication is a highly relevant aspect for Internet of Things (IoT) systems in which tiny sensors, actuators, wearables, or other smart devices exchange messages. Various publish/subscribe protocols address the challenges of communication in IoT systems. The selection process of a suitable protocol should consider the communication behavior of the application, the protocol's performance, the resource requirements on the end device, and the network connection quality, as IoT environments often rely on wireless networks. Benchmarking is a common approach to evaluate and compare systems, considering the performance and aspects like dependability or security. In this paper, we present our IoT communication benchmarking framework ComBench for publish/subscribe protocols focusing on constrained networks with varying quality conditions. The benchmarking framework supports system designers, software engineers, and application developers to select and investigate the behavior of communication protocols. Our benchmarking framework contributes to (i) show the impact of fluctuating network quality on communication, (ii) compare multiple protocols, protocol features, and protocol implementations, and (iii) analyze scalability, robustness, and dependability of clients, networks, and brokers in different scenarios. Our case study demonstrates the applicability of our framework to support the decision for the best-suited protocol in various scenarios.
• 
  Towards a Mobile Robot Localization Benchmark with Challenging Sensordata in an Industrial Environment Spiess, Florian; Friesslich, Jonas; Bluemm, Daniel; Mast, Fabio; Vinokour, Dmitrij; Kounev, Samuel; Kaupp, Tobias; Strobel, Norbert; in 2021 20th International Conference on Advanced Robotics (ICAR) (2021). 857–864.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  To arrive at a realistic assessment of localization methods in terms of their performance in an industrial environment under various challenging conditions, we provide a benchmark to evaluate algorithms both for individual components as well as multi-sensor systems. For several sensor types, including wheel odometry, RGB cameras, RGB-D cameras, and LIDAR, potential issues were identified. The accuracy of wheel odometry, for example, when there are bumps on the track. For each sensor type, we explicitly chose a track for the benchmark dataset containing situations where the sensor fails to provide adequate measurements. Based on the acquired sensor data, localization can be achieved either using a single sensor information or sensor fusion. To help evaluate the output of associated localization algorithms, we provide a software to evaluate a set of metrics as part of the paper. An example application of the benchmark with state-of-the-art algorithms for each sensor is also provided.

  @inproceedings{9659355, abstract = {To arrive at a realistic assessment of localization methods in terms of their performance in an industrial environment under various challenging conditions, we provide a benchmark to evaluate algorithms both for individual components as well as multi-sensor systems. For several sensor types, including wheel odometry, RGB cameras, RGB-D cameras, and LIDAR, potential issues were identified. The accuracy of wheel odometry, for example, when there are bumps on the track. For each sensor type, we explicitly chose a track for the benchmark dataset containing situations where the sensor fails to provide adequate measurements. Based on the acquired sensor data, localization can be achieved either using a single sensor information or sensor fusion. To help evaluate the output of associated localization algorithms, we provide a software to evaluate a set of metrics as part of the paper. An example application of the benchmark with state-of-the-art algorithms for each sensor is also provided.}, author = {Spiess, Florian and Friesslich, Jonas and Bluemm, Daniel and Mast, Fabio and Vinokour, Dmitrij and Kounev, Samuel and Kaupp, Tobias and Strobel, Norbert}, booktitle = {2021 20th International Conference on Advanced Robotics (ICAR)}, keywords = {myown}, month = {dec}, pages = {857-864}, title = {Towards a Mobile Robot Localization Benchmark with Challenging Sensordata in an Industrial Environment}, year = 2021 }

  %0 Conference Paper %1 9659355 %A Spiess, Florian %A Friesslich, Jonas %A Bluemm, Daniel %A Mast, Fabio %A Vinokour, Dmitrij %A Kounev, Samuel %A Kaupp, Tobias %A Strobel, Norbert %B 2021 20th International Conference on Advanced Robotics (ICAR) %D 2021 %P 857-864 %T Towards a Mobile Robot Localization Benchmark with Challenging Sensordata in an Industrial Environment %U https://ieeexplore.ieee.org/document/9659355/ %X To arrive at a realistic assessment of localization methods in terms of their performance in an industrial environment under various challenging conditions, we provide a benchmark to evaluate algorithms both for individual components as well as multi-sensor systems. For several sensor types, including wheel odometry, RGB cameras, RGB-D cameras, and LIDAR, potential issues were identified. The accuracy of wheel odometry, for example, when there are bumps on the track. For each sensor type, we explicitly chose a track for the benchmark dataset containing situations where the sensor fails to provide adequate measurements. Based on the acquired sensor data, localization can be achieved either using a single sensor information or sensor fusion. To help evaluate the output of associated localization algorithms, we provide a software to evaluate a set of metrics as part of the paper. An example application of the benchmark with state-of-the-art algorithms for each sensor is also provided.
• 
  Benchmarking of Pre- and Post-Quantum Group Encryption Schemes with Focus on IoT Prantl, Thomas; Prantl, Dominik; Bauer, André; Iffländer, Lukas; Dmitrenko, Alexandra; Krupitzer, Christian; Kounev, Samuel; in 2021 IEEE 40th International Performance Computing and Communications Conference (IPCCC) (2021).
  • [ BibTeX ]
  • [ EndNote ]
  @inproceedings{PrPrBaIfDmKrKo2021-IPCCC-QuantumGroupEncryption, author = {Prantl, Thomas and Prantl, Dominik and Bauer, André and Iffländer, Lukas and Dmitrenko, Alexandra and Krupitzer, Christian and Kounev, Samuel}, booktitle = {2021 IEEE 40th International Performance Computing and Communications Conference (IPCCC)}, keywords = {myown}, title = {Benchmarking of Pre- and Post-Quantum Group Encryption Schemes with Focus on IoT}, year = 2021 }

  %0 Conference Paper %1 PrPrBaIfDmKrKo2021-IPCCC-QuantumGroupEncryption %A Prantl, Thomas %A Prantl, Dominik %A Bauer, André %A Iffländer, Lukas %A Dmitrenko, Alexandra %A Krupitzer, Christian %A Kounev, Samuel %B 2021 IEEE 40th International Performance Computing and Communications Conference (IPCCC) %D 2021 %T Benchmarking of Pre- and Post-Quantum Group Encryption Schemes with Focus on IoT

• 
  Evaluating the Performance of a State-of-the-Art Group-oriented Encryption Scheme for Dynamic Groups in an IoT Scenario Prantl, Thomas; Ten, Peter; Iffländer, Lukas; Dmitrenko, Alexandra; Kounev, Samuel; Krupitzer, Christian; in 2020 IEEE 28th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS) (2020).

  Acceptance Rate: 27%
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  New emerging technologies, such as autonomous driving, intelligent buildings, and smart cities, are promising to revolutionize user experience and offer new services. The world has to undergo large scale deployment of billions of things — cost-efficient intelligent sensors that will be interconnected into extensive networks and will collect and supply data to intelligent algorithms — to make it happen. To date, however, it is challenging to secure such an infrastructure for many-fold reasons, such as resource constraints of things, large scale deployment, manyto-many communication patterns, and dynamically changing communication groups. All these factors rule out most of the state-of-the-art encryption and key-management techniques. Group encryption algorithms are well-suitable for many-tomany communication patterns typical for IoT networks, and many of them can deal with dynamic groups. There are, however, very few constructions that could potentially fulfill the computational and storage constraints of IoT devices while providing sufficient scalability for large networks. The promising candidates, such as construction by Nishat et al. [1], were not evaluated using IoT platforms and under constraints typical for IoT networks. In this paper, we aim to fill this gap and present the evaluation of a state-of-the-art group-oriented encryption scheme by Nishat et al. to identify its applicability to IoT systems. In detail, we provide a measurement workflow, a revised version of the approach, and describe a reproducible hardware testbed. Using this evaluation environment, we analyze the performance of the encryption scheme in a typical IoT scenario from a group member perspective. The results show that all calculation times can be assumed to be constant and are always below 2 seconds. The memory requirement for permanent parameters can also be considered to be constant and are below 8.5 kbit in each case. However, the information that has to be stored temporarily for group updates has turned out to be the bottleneck of the scheme, since their memory requirements increase linearly with the group size.

  @inproceedings{PrTeIfDmKo2020-MASCOTS-GroupEncryption, abstract = {New emerging technologies, such as autonomous driving, intelligent buildings, and smart cities, are promising to revolutionize user experience and offer new services. The world has to undergo large scale deployment of billions of things — cost-efficient intelligent sensors that will be interconnected into extensive networks and will collect and supply data to intelligent algorithms — to make it happen. To date, however, it is challenging to secure such an infrastructure for many-fold reasons, such as resource constraints of things, large scale deployment, manyto-many communication patterns, and dynamically changing communication groups. All these factors rule out most of the state-of-the-art encryption and key-management techniques. Group encryption algorithms are well-suitable for many-tomany communication patterns typical for IoT networks, and many of them can deal with dynamic groups. There are, however, very few constructions that could potentially fulfill the computational and storage constraints of IoT devices while providing sufficient scalability for large networks. The promising candidates, such as construction by Nishat et al. [1], were not evaluated using IoT platforms and under constraints typical for IoT networks. In this paper, we aim to fill this gap and present the evaluation of a state-of-the-art group-oriented encryption scheme by Nishat et al. to identify its applicability to IoT systems. In detail, we provide a measurement workflow, a revised version of the approach, and describe a reproducible hardware testbed. Using this evaluation environment, we analyze the performance of the encryption scheme in a typical IoT scenario from a group member perspective. The results show that all calculation times can be assumed to be constant and are always below 2 seconds. The memory requirement for permanent parameters can also be considered to be constant and are below 8.5 kbit in each case. However, the information that has to be stored temporarily for group updates has turned out to be the bottleneck of the scheme, since their memory requirements increase linearly with the group size.}, author = {Prantl, Thomas and Ten, Peter and Iffländer, Lukas and Dmitrenko, Alexandra and Kounev, Samuel and Krupitzer, Christian}, booktitle = {2020 IEEE 28th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS)}, keywords = {sss-group}, month = {November}, note = {Acceptance Rate: 27\%}, series = {MASCOTS '20}, title = {Evaluating the Performance of a State-of-the-Art Group-oriented Encryption Scheme for Dynamic Groups in an IoT Scenario}, year = 2020 }

  %0 Conference Paper %1 PrTeIfDmKo2020-MASCOTS-GroupEncryption %A Prantl, Thomas %A Ten, Peter %A Iffländer, Lukas %A Dmitrenko, Alexandra %A Kounev, Samuel %A Krupitzer, Christian %B 2020 IEEE 28th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS) %D 2020 %T Evaluating the Performance of a State-of-the-Art Group-oriented Encryption Scheme for Dynamic Groups in an IoT Scenario %X New emerging technologies, such as autonomous driving, intelligent buildings, and smart cities, are promising to revolutionize user experience and offer new services. The world has to undergo large scale deployment of billions of things — cost-efficient intelligent sensors that will be interconnected into extensive networks and will collect and supply data to intelligent algorithms — to make it happen. To date, however, it is challenging to secure such an infrastructure for many-fold reasons, such as resource constraints of things, large scale deployment, manyto-many communication patterns, and dynamically changing communication groups. All these factors rule out most of the state-of-the-art encryption and key-management techniques. Group encryption algorithms are well-suitable for many-tomany communication patterns typical for IoT networks, and many of them can deal with dynamic groups. There are, however, very few constructions that could potentially fulfill the computational and storage constraints of IoT devices while providing sufficient scalability for large networks. The promising candidates, such as construction by Nishat et al. [1], were not evaluated using IoT platforms and under constraints typical for IoT networks. In this paper, we aim to fill this gap and present the evaluation of a state-of-the-art group-oriented encryption scheme by Nishat et al. to identify its applicability to IoT systems. In detail, we provide a measurement workflow, a revised version of the approach, and describe a reproducible hardware testbed. Using this evaluation environment, we analyze the performance of the encryption scheme in a typical IoT scenario from a group member perspective. The results show that all calculation times can be assumed to be constant and are always below 2 seconds. The memory requirement for permanent parameters can also be considered to be constant and are below 8.5 kbit in each case. However, the information that has to be stored temporarily for group updates has turned out to be the bottleneck of the scheme, since their memory requirements increase linearly with the group size.
• 
  Baloo: Measuring and Modeling the Performance Configurations of Distributed DBMS Grohmann, Johannes; Seybold, Daniel; Eismann, Simon; Leznik, Mark; Kounev, Samuel; Domaschka, Jörg; in 2020 28th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS) (2020). 1–8. IEEE.

  Acceptance Rate: 27%
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  • [ Download ]
  Correctly configuring a distributed database management system (DBMS) deployed in a cloud environment for maximizing performance poses many challenges to operators. Even if the entire configuration spectrum could be measured directly, which is often infeasible due to the multitude of parameters, single measurements are subject to random variations and need to be repeated multiple times. In this work, we propose Baloo, a framework for systematically measuring and modeling different performance-relevant configurations of distributed DBMS in cloud environments. Baloo dynamically estimates the required number of configurations, as well as the number of required measurement repetitions per configuration based on a desired target accuracy. We evaluate Baloo based on a data set consisting of 900 DBMS configuration measurements conducted in our private cloud setup. Our evaluation shows that the highly configurable framework is able to achieve a prediction error of up to 12% while saving 80% of the measurement effort. We also publish all code and the acquired data set to foster future research.

  @inproceedings{GrSeEiLeKoDo2020-MASCOTS-DBMSPerformance, abstract = {Correctly configuring a distributed database management system (DBMS) deployed in a cloud environment for maximizing performance poses many challenges to operators. Even if the entire configuration spectrum could be measured directly, which is often infeasible due to the multitude of parameters, single measurements are subject to random variations and need to be repeated multiple times. In this work, we propose Baloo, a framework for systematically measuring and modeling different performance-relevant configurations of distributed DBMS in cloud environments. Baloo dynamically estimates the required number of configurations, as well as the number of required measurement repetitions per configuration based on a desired target accuracy. We evaluate Baloo based on a data set consisting of 900 DBMS configuration measurements conducted in our private cloud setup. Our evaluation shows that the highly configurable framework is able to achieve a prediction error of up to 12% while saving 80% of the measurement effort. We also publish all code and the acquired data set to foster future research.}, author = {Grohmann, Johannes and Seybold, Daniel and Eismann, Simon and Leznik, Mark and Kounev, Samuel and Domaschka, Jörg}, booktitle = {2020 28th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS)}, keywords = {t_full}, month = {November}, note = {Acceptance Rate: 27%}, pages = {1–8}, publisher = {IEEE}, series = {MASCOTS '20}, title = {Baloo: Measuring and Modeling the Performance Configurations of Distributed DBMS}, year = 2020 }

  %0 Conference Paper %1 GrSeEiLeKoDo2020-MASCOTS-DBMSPerformance %A Grohmann, Johannes %A Seybold, Daniel %A Eismann, Simon %A Leznik, Mark %A Kounev, Samuel %A Domaschka, Jörg %B 2020 28th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS) %D 2020 %I IEEE %P 1--8 %R 10.1109/MASCOTS50786.2020.9285960 %T Baloo: Measuring and Modeling the Performance Configurations of Distributed DBMS %U https://doi.org/10.1109/MASCOTS50786.2020.9285960 %X Correctly configuring a distributed database management system (DBMS) deployed in a cloud environment for maximizing performance poses many challenges to operators. Even if the entire configuration spectrum could be measured directly, which is often infeasible due to the multitude of parameters, single measurements are subject to random variations and need to be repeated multiple times. In this work, we propose Baloo, a framework for systematically measuring and modeling different performance-relevant configurations of distributed DBMS in cloud environments. Baloo dynamically estimates the required number of configurations, as well as the number of required measurement repetitions per configuration based on a desired target accuracy. We evaluate Baloo based on a data set consisting of 900 DBMS configuration measurements conducted in our private cloud setup. Our evaluation shows that the highly configurable framework is able to achieve a prediction error of up to 12% while saving 80% of the measurement effort. We also publish all code and the acquired data set to foster future research.
• 
  Quantifying measurement quality and load distribution in Tor Greubel, Andre; Pohl, Steffen; Kounev, Samuel; in Proceedings of the 36th Annual Computer Security Applications Conference (ACSAC 2020) (2020).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Tor is a widely used anonymization network. Traffic is routed over different relay nodes to conceal the communication partners. However, if a single relay handles too much traffic, de-anonymization attacks are possible. The Tor Load Balancing Mechanism (TLBM) is responsible for balanced and secure load distribution. It must verify that relays cannot attract more traffic than they should by lying about their self-reported bandwidth. This work shows that the current bandwidth measurement method used for bandwidth verification is not suitable to verify the bandwidth of many relays. Most importantly, multiple measurements of high-bandwidth relays are uncorrelated to each other. Furthermore, we analyze the current load distribution in Tor. We show that the current load distribution reduces the resources necessary for several large-scale de-anonymization attacks by more than 80%. Additionally, as Tor favors fast relays during path selection, verifiable relays only handle a small fraction of Tor’s traffic. More precisely, we show that only 7.21% of all paths consist of entry and exit relays verifiable by measurements. We discuss these results’ security implications and argue that future TLBM research should focus at least as much on secure load distribution as on high traffic performance.

  @article{greubel2020quantifying, abstract = {Tor is a widely used anonymization network. Traffic is routed over different relay nodes to conceal the communication partners. However, if a single relay handles too much traffic, de-anonymization attacks are possible. The Tor Load Balancing Mechanism (TLBM) is responsible for balanced and secure load distribution. It must verify that relays cannot attract more traffic than they should by lying about their self-reported bandwidth. This work shows that the current bandwidth measurement method used for bandwidth verification is not suitable to verify the bandwidth of many relays. Most importantly, multiple measurements of high-bandwidth relays are uncorrelated to each other. Furthermore, we analyze the current load distribution in Tor. We show that the current load distribution reduces the resources necessary for several large-scale de-anonymization attacks by more than 80\%. Additionally, as Tor favors fast relays during path selection, verifiable relays only handle a small fraction of Tor’s traffic. More precisely, we show that only 7.21\% of all paths consist of entry and exit relays verifiable by measurements. We discuss these results’ security implications and argue that future TLBM research should focus at least as much on secure load distribution as on high traffic performance.}, author = {Greubel, Andre and Pohl, Steffen and Kounev, Samuel}, journal = {Proceedings of the 36th Annual Computer Security Applications Conference (ACSAC 2020)}, keywords = {myown}, title = {Quantifying measurement quality and load distribution in Tor}, year = 2020 }

  %0 Journal Article %1 greubel2020quantifying %A Greubel, Andre %A Pohl, Steffen %A Kounev, Samuel %D 2020 %J Proceedings of the 36th Annual Computer Security Applications Conference (ACSAC 2020) %T Quantifying measurement quality and load distribution in Tor %X Tor is a widely used anonymization network. Traffic is routed over different relay nodes to conceal the communication partners. However, if a single relay handles too much traffic, de-anonymization attacks are possible. The Tor Load Balancing Mechanism (TLBM) is responsible for balanced and secure load distribution. It must verify that relays cannot attract more traffic than they should by lying about their self-reported bandwidth. This work shows that the current bandwidth measurement method used for bandwidth verification is not suitable to verify the bandwidth of many relays. Most importantly, multiple measurements of high-bandwidth relays are uncorrelated to each other. Furthermore, we analyze the current load distribution in Tor. We show that the current load distribution reduces the resources necessary for several large-scale de-anonymization attacks by more than 80%. Additionally, as Tor favors fast relays during path selection, verifiable relays only handle a small fraction of Tor’s traffic. More precisely, we show that only 7.21% of all paths consist of entry and exit relays verifiable by measurements. We discuss these results’ security implications and argue that future TLBM research should focus at least as much on secure load distribution as on high traffic performance.
• 
  An IoT Network Emulator for Analyzing the Influence of Varying Network Quality Herrnleben, Stefan; Ailabouni, Rudy; Grohmann, Johannes; Prantl, Thomas; Krupitzer, Christian; Kounev, Samuel; in Proceedings of the 12th EAI International Conference on Simulation Tools and Techniques (SIMUtools) (2020).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  • [ Download ]
  IoT devices often communicate over wireless or cellular networks with varying connection quality. These fluctuations are caused, among others, by the free-space path loss (FSPL), buildings, topological obstacles, weather, and mobility of the receiver. Varying signal quality affects bandwidth, transmission delays, packet loss, and jitter. Mobile IoT applications exposed to varying connection characteristics have to handle such variations and take them into account during development and testing. However, tests in real mobile networks are complex and challenging to reproduce. Therefore, network emulators can be used to simulate the behavior of real-world networks by adding artificial disturbance. However, existing network emulators often require a lot of technical knowledge and complex setup. Integrating such emulators into automated software testing pipelines could be a challenging task. In this paper, we propose a framework for emulating IoT networks with varying quality characteristics. An existing base emulator is used and integrated into our framework enabling the user to utilize it without extensive network expertise and configuration effort. The evaluation proves that our framework can simulate a variety of different network quality characteristics as well as emulating real-world network traces.

  @inproceedings{HeAiGrPrKrKo2020-SIMUtools-IoT-Network-Emulator, abstract = {IoT devices often communicate over wireless or cellular networks with varying connection quality. These fluctuations are caused, among others, by the free-space path loss (FSPL), buildings, topological obstacles, weather, and mobility of the receiver. Varying signal quality affects bandwidth, transmission delays, packet loss, and jitter. Mobile IoT applications exposed to varying connection characteristics have to handle such variations and take them into account during development and testing. However, tests in real mobile networks are complex and challenging to reproduce. Therefore, network emulators can be used to simulate the behavior of real-world networks by adding artificial disturbance. However, existing network emulators often require a lot of technical knowledge and complex setup. Integrating such emulators into automated software testing pipelines could be a challenging task. In this paper, we propose a framework for emulating IoT networks with varying quality characteristics. An existing base emulator is used and integrated into our framework enabling the user to utilize it without extensive network expertise and configuration effort. The evaluation proves that our framework can simulate a variety of different network quality characteristics as well as emulating real-world network traces.}, author = {Herrnleben, Stefan and Ailabouni, Rudy and Grohmann, Johannes and Prantl, Thomas and Krupitzer, Christian and Kounev, Samuel}, booktitle = {Proceedings of the 12th EAI International Conference on Simulation Tools and Techniques (SIMUtools)}, keywords = {t_full}, month = {August}, series = {SIMUtools 2020}, title = {An IoT Network Emulator for Analyzing the Influence of Varying Network Quality}, year = 2020 }

  %0 Conference Paper %1 HeAiGrPrKrKo2020-SIMUtools-IoT-Network-Emulator %A Herrnleben, Stefan %A Ailabouni, Rudy %A Grohmann, Johannes %A Prantl, Thomas %A Krupitzer, Christian %A Kounev, Samuel %B Proceedings of the 12th EAI International Conference on Simulation Tools and Techniques (SIMUtools) %D 2020 %R https://doi.org/10.1007/978-3-030-72795-6_47 %T An IoT Network Emulator for Analyzing the Influence of Varying Network Quality %U https://doi.org/10.1007/978-3-030-72795-6_47 %X IoT devices often communicate over wireless or cellular networks with varying connection quality. These fluctuations are caused, among others, by the free-space path loss (FSPL), buildings, topological obstacles, weather, and mobility of the receiver. Varying signal quality affects bandwidth, transmission delays, packet loss, and jitter. Mobile IoT applications exposed to varying connection characteristics have to handle such variations and take them into account during development and testing. However, tests in real mobile networks are complex and challenging to reproduce. Therefore, network emulators can be used to simulate the behavior of real-world networks by adding artificial disturbance. However, existing network emulators often require a lot of technical knowledge and complex setup. Integrating such emulators into automated software testing pipelines could be a challenging task. In this paper, we propose a framework for emulating IoT networks with varying quality characteristics. An existing base emulator is used and integrated into our framework enabling the user to utilize it without extensive network expertise and configuration effort. The evaluation proves that our framework can simulate a variety of different network quality characteristics as well as emulating real-world network traces.
• 
  To Fail Or Not To Fail: Predicting Hard Disk Drive Failure Time Windows Züfle, Marwin; Krupitzer, Christian; Erhard, Florian; Grohmann, Johannes; Kounev, Samuel; in Proceedings of the 20th International GI/ITG Conference on Measurement, Modelling and Evaluation of Computing Systems (2020). 19–36. Springer, Cham.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  • [ Download ]
  Due to the increasing size of today's data centers as well as the expectation of 24/7 availability, the complexity in the administration of hardware continuously icreases. Techniques as the Self-Monitoring, Analysis, and Reporting Technology (S.M.A.R.T.) support the monitoring of the hardware. However, those techniques often lack algorithms for intelligent data analytics. Especially, the integration of machine learning to identify potential failures in advance seems to be promising to reduce administration overhead. In this work, we present three machine learning approaches to (i) identify imminent failures, (ii) predict time windows for failures, as well as (iii) predict the exact time-to-failure. In a case study with real data from 369 hard disks, we achieve an F1-score of up to 98.0% and 97.6% for predicting potential failures with two or multiple time windows, respectively, and a hit rate of 84.9% (with a mean absolute error of 4.5 hours) for predicting the time-to-failure.

  @inproceedings{ZuKrErGrKo-MMB2020-ToFailOrNotToFail, abstract = {Due to the increasing size of today's data centers as well as the expectation of 24/7 availability, the complexity in the administration of hardware continuously icreases. Techniques as the Self-Monitoring, Analysis, and Reporting Technology (S.M.A.R.T.) support the monitoring of the hardware. However, those techniques often lack algorithms for intelligent data analytics. Especially, the integration of machine learning to identify potential failures in advance seems to be promising to reduce administration overhead. In this work, we present three machine learning approaches to (i) identify imminent failures, (ii) predict time windows for failures, as well as (iii) predict the exact time-to-failure. In a case study with real data from 369 hard disks, we achieve an F1-score of up to 98.0% and 97.6% for predicting potential failures with two or multiple time windows, respectively, and a hit rate of 84.9% (with a mean absolute error of 4.5 hours) for predicting the time-to-failure.}, address = {Cham}, author = {Z{\"u}fle, Marwin and Krupitzer, Christian and Erhard, Florian and Grohmann, Johannes and Kounev, Samuel}, booktitle = {Proceedings of the 20th International GI/ITG Conference on Measurement, Modelling and Evaluation of Computing Systems}, keywords = {myown}, month = {March}, pages = {19–36}, publisher = {Springer}, series = {MMB 2020}, title = {{To Fail Or Not To Fail: Predicting Hard Disk Drive Failure Time Windows}}, year = 2020 }

  %0 Conference Paper %1 ZuKrErGrKo-MMB2020-ToFailOrNotToFail %A Züfle, Marwin %A Krupitzer, Christian %A Erhard, Florian %A Grohmann, Johannes %A Kounev, Samuel %B Proceedings of the 20th International GI/ITG Conference on Measurement, Modelling and Evaluation of Computing Systems %C Cham %D 2020 %I Springer %P 19--36 %R 10.1007/978-3-030-43024-5_2 %T To Fail Or Not To Fail: Predicting Hard Disk Drive Failure Time Windows %U https://doi.org/10.1007/978-3-030-43024-5_2 %X Due to the increasing size of today's data centers as well as the expectation of 24/7 availability, the complexity in the administration of hardware continuously icreases. Techniques as the Self-Monitoring, Analysis, and Reporting Technology (S.M.A.R.T.) support the monitoring of the hardware. However, those techniques often lack algorithms for intelligent data analytics. Especially, the integration of machine learning to identify potential failures in advance seems to be promising to reduce administration overhead. In this work, we present three machine learning approaches to (i) identify imminent failures, (ii) predict time windows for failures, as well as (iii) predict the exact time-to-failure. In a case study with real data from 369 hard disks, we achieve an F1-score of up to 98.0% and 97.6% for predicting potential failures with two or multiple time windows, respectively, and a hit rate of 84.9% (with a mean absolute error of 4.5 hours) for predicting the time-to-failure. %@ 978-3-030-43024-5
• 
  Model-based Performance Predictions for SDN-based Networks: A Case Study Herrnleben, Stefan; Rygielski, Piotr; Grohmann, Johannes; Eismann, Simon; Hossfeld, Tobias; Kounev, Samuel; in Proceedings of the 20th International GI/ITG Conference on Measurement, Modelling and Evaluation of Computing Systems (2020). Springer, Cham.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  • [ Download ]
  Emerging paradigms for network virtualization like Software-Defined Networking (SDN) and Network Functions Virtualization (NFV) form new challenges for accurate performance modeling and analysis tools. Therefore, performance modeling and prediction approaches that support SDN or NFV technologies help system operators to analyze the performance of a data center and its corresponding network. The Descartes Network Infrastructures (DNI) offers a high-level descriptive language to model SDN-based networks, which can be transformed into various predictive modeling formalisms. However, these modeling concepts have not yet been evaluated in a realistic scenario. In this paper, we present an extensive case study evaluating the DNI modeling capabilities, the transformations to predictive models, and the performance prediction using the OMNeT++ and SimQPN simulation frameworks. We present five realistic scenarios of a content distribution network (CDN), compare the performance predictions with real-world measurements, and discuss modeling gaps and calibration issues causing mispredictions in some scenarios.

  @inproceedings{HeRyGrEiHoKo-MMB2020-Model-based-SDN-Performance, abstract = {Emerging paradigms for network virtualization like Software-Defined Networking (SDN) and Network Functions Virtualization (NFV) form new challenges for accurate performance modeling and analysis tools. Therefore, performance modeling and prediction approaches that support SDN or NFV technologies help system operators to analyze the performance of a data center and its corresponding network. The Descartes Network Infrastructures (DNI) offers a high-level descriptive language to model SDN-based networks, which can be transformed into various predictive modeling formalisms. However, these modeling concepts have not yet been evaluated in a realistic scenario. In this paper, we present an extensive case study evaluating the DNI modeling capabilities, the transformations to predictive models, and the performance prediction using the OMNeT++ and SimQPN simulation frameworks. We present five realistic scenarios of a content distribution network (CDN), compare the performance predictions with real-world measurements, and discuss modeling gaps and calibration issues causing mispredictions in some scenarios.}, address = {Cham}, author = {Herrnleben, Stefan and Rygielski, Piotr and Grohmann, Johannes and Eismann, Simon and Hossfeld, Tobias and Kounev, Samuel}, booktitle = {Proceedings of the 20th International GI/ITG Conference on Measurement, Modelling and Evaluation of Computing Systems}, keywords = {myown}, month = {March}, publisher = {Springer}, series = {MMB 2020}, title = {{Model-based Performance Predictions for SDN-based Networks: A Case Study}}, year = 2020 }

  %0 Conference Paper %1 HeRyGrEiHoKo-MMB2020-Model-based-SDN-Performance %A Herrnleben, Stefan %A Rygielski, Piotr %A Grohmann, Johannes %A Eismann, Simon %A Hossfeld, Tobias %A Kounev, Samuel %B Proceedings of the 20th International GI/ITG Conference on Measurement, Modelling and Evaluation of Computing Systems %C Cham %D 2020 %I Springer %R 10.1007/978-3-030-43024-5_6 %T Model-based Performance Predictions for SDN-based Networks: A Case Study %U https://doi.org/10.1007/978-3-030-43024-5_6 %X Emerging paradigms for network virtualization like Software-Defined Networking (SDN) and Network Functions Virtualization (NFV) form new challenges for accurate performance modeling and analysis tools. Therefore, performance modeling and prediction approaches that support SDN or NFV technologies help system operators to analyze the performance of a data center and its corresponding network. The Descartes Network Infrastructures (DNI) offers a high-level descriptive language to model SDN-based networks, which can be transformed into various predictive modeling formalisms. However, these modeling concepts have not yet been evaluated in a realistic scenario. In this paper, we present an extensive case study evaluating the DNI modeling capabilities, the transformations to predictive models, and the performance prediction using the OMNeT++ and SimQPN simulation frameworks. We present five realistic scenarios of a content distribution network (CDN), compare the performance predictions with real-world measurements, and discuss modeling gaps and calibration issues causing mispredictions in some scenarios. %@ 978-3-030-43024-5
• 
  Predicting the Costs of Serverless Workflows Eismann, Simon; Grohmann, Johannes; van Eyk, Erwin; Herbst, Nikolas; Kounev, Samuel; in Proceedings of the 2020 ACM/SPEC International Conference on Performance Engineering (ICPE) (2020). 265–276. Association for Computing Machinery (ACM), New York, NY, USA.

  Acceptance Rate: 23.4% (15/64)
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  • [ Download ]
  Function-as-a-Service (FaaS) platforms enable users to run arbitrary functions without being concerned about operational issues, while only paying for the consumed resources. Individual functions are often composed into workflows for complex tasks. However, the pay-per-use model and nontransparent reporting by cloud providers make it challenging to estimate the expected cost of a workflow, which prevents informed business decisions. Existing cost-estimation approaches assume a static response time for the serverless functions, without taking input parameters into account. In this paper, we propose a methodology for the cost prediction of serverless workflows consisting of input-parameter sensitive function models and a monte-carlo simulation of an abstract workflow model. Our approach enables workflow designers to predict, compare, and optimize the expected costs and performance of a planned workflow, which currently requires time-intensive experimentation. In our evaluation, we show that our approach can predict the response time and output parameters of a function based on its input parameters with an accuracy of 96.1%. In a case study with two audio-processing workflows, our approach predicts the costs of the two workflows with an accuracy of 96.2%.

  @inproceedings{EiGrEyHeKo2020-ICPE-ServerlessWorkflows, abstract = {Function-as-a-Service (FaaS) platforms enable users to run arbitrary functions without being concerned about operational issues, while only paying for the consumed resources. Individual functions are often composed into workflows for complex tasks. However, the pay-per-use model and nontransparent reporting by cloud providers make it challenging to estimate the expected cost of a workflow, which prevents informed business decisions. Existing cost-estimation approaches assume a static response time for the serverless functions, without taking input parameters into account. In this paper, we propose a methodology for the cost prediction of serverless workflows consisting of input-parameter sensitive function models and a monte-carlo simulation of an abstract workflow model. Our approach enables workflow designers to predict, compare, and optimize the expected costs and performance of a planned workflow, which currently requires time-intensive experimentation. In our evaluation, we show that our approach can predict the response time and output parameters of a function based on its input parameters with an accuracy of 96.1%. In a case study with two audio-processing workflows, our approach predicts the costs of the two workflows with an accuracy of 96.2%.}, address = {New York, NY, USA}, author = {Eismann, Simon and Grohmann, Johannes and van Eyk, Erwin and Herbst, Nikolas and Kounev, Samuel}, booktitle = {Proceedings of the 2020 ACM/SPEC International Conference on Performance Engineering (ICPE)}, keywords = {myown}, month = {April}, note = {{Acceptance Rate: 23.4% (15/64)}}, pages = {265–276}, publisher = {Association for Computing Machinery (ACM)}, series = {ICPE '20}, title = {Predicting the Costs of Serverless Workflows}, year = 2020 }

  %0 Conference Paper %1 EiGrEyHeKo2020-ICPE-ServerlessWorkflows %A Eismann, Simon %A Grohmann, Johannes %A van Eyk, Erwin %A Herbst, Nikolas %A Kounev, Samuel %B Proceedings of the 2020 ACM/SPEC International Conference on Performance Engineering (ICPE) %C New York, NY, USA %D 2020 %I Association for Computing Machinery (ACM) %P 265--276 %R 10.1145/3358960.3379133 %T Predicting the Costs of Serverless Workflows %U https://doi.org/10.1145/3358960.3379133 %X Function-as-a-Service (FaaS) platforms enable users to run arbitrary functions without being concerned about operational issues, while only paying for the consumed resources. Individual functions are often composed into workflows for complex tasks. However, the pay-per-use model and nontransparent reporting by cloud providers make it challenging to estimate the expected cost of a workflow, which prevents informed business decisions. Existing cost-estimation approaches assume a static response time for the serverless functions, without taking input parameters into account. In this paper, we propose a methodology for the cost prediction of serverless workflows consisting of input-parameter sensitive function models and a monte-carlo simulation of an abstract workflow model. Our approach enables workflow designers to predict, compare, and optimize the expected costs and performance of a planned workflow, which currently requires time-intensive experimentation. In our evaluation, we show that our approach can predict the response time and output parameters of a function based on its input parameters with an accuracy of 96.1%. In a case study with two audio-processing workflows, our approach predicts the costs of the two workflows with an accuracy of 96.2%. %@ 9781450369916

• 
  Integrating Statistical Response Time Models in Architectural Performance Models Eismann, Simon; Grohmann, Johannes; Walter, Jürgen; von Kistowski, Jóakim; Kounev, Samuel; in Proceedings of the 2019 IEEE International Conference on Software Architecture (ICSA) (2019). 71–80. IEEE.

  Acceptance Rate: 21,9% (21/96)
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  • [ Download ]
  Performance predictions enable software architects to optimize the performance of a software system early in the development cycle. Architectural performance models and statistical response time models are commonly used to derive these performance predictions. However, both methods have significant downsides: Statistical response time models can only predict scenarios for which training data is available, making the prediction of previously unseen system configurations infeasible. In contrast, the time required to simulate an architectural performance model increases exponentially with both system size and level of modeling detail, making the analysis of large, detailed models challenging. Existing approaches use statistical response time models in architectural performance models to avoid modeling subsystems that are difficult or time-consuming to model, yet they do not consider simulation time. In this paper, we propose to model software systems using classical queuing theory and statistical response time models in parallel. This approach allows users to tailor the model for each analysis run, based on the performed adaptations and the requested performance metrics. Our approach enables faster model solution compared to traditional performance models while retaining their ability to predict previously unseen scenarios. In our experiments we observed speedups of up to 94.8%, making the analysis of much larger and more detailed systems feasible.

  @inproceedings{EiGrWaKiKo2019-ICSA-Integrating, abstract = {Performance predictions enable software architects to optimize the performance of a software system early in the development cycle. Architectural performance models and statistical response time models are commonly used to derive these performance predictions. However, both methods have significant downsides: Statistical response time models can only predict scenarios for which training data is available, making the prediction of previously unseen system configurations infeasible. In contrast, the time required to simulate an architectural performance model increases exponentially with both system size and level of modeling detail, making the analysis of large, detailed models challenging. Existing approaches use statistical response time models in architectural performance models to avoid modeling subsystems that are difficult or time-consuming to model, yet they do not consider simulation time. In this paper, we propose to model software systems using classical queuing theory and statistical response time models in parallel. This approach allows users to tailor the model for each analysis run, based on the performed adaptations and the requested performance metrics. Our approach enables faster model solution compared to traditional performance models while retaining their ability to predict previously unseen scenarios. In our experiments we observed speedups of up to 94.8%, making the analysis of much larger and more detailed systems feasible.}, author = {Eismann, Simon and Grohmann, Johannes and Walter, J{\"u}rgen and von Kistowski, J{\'o}akim and Kounev, Samuel}, booktitle = {Proceedings of the 2019 IEEE International Conference on Software Architecture (ICSA)}, keywords = {myown}, month = {March}, note = {Acceptance Rate: 21,9\% (21/96)}, pages = {71–80}, publisher = {IEEE}, title = {{Integrating Statistical Response Time Models in Architectural Performance Models}}, year = 2019 }

  %0 Conference Paper %1 EiGrWaKiKo2019-ICSA-Integrating %A Eismann, Simon %A Grohmann, Johannes %A Walter, Jürgen %A von Kistowski, Jóakim %A Kounev, Samuel %B Proceedings of the 2019 IEEE International Conference on Software Architecture (ICSA) %D 2019 %I IEEE %P 71--80 %R 10.1109/ICSA.2019.00016 %T Integrating Statistical Response Time Models in Architectural Performance Models %U https://doi.org/10.1109/ICSA.2019.00016 %X Performance predictions enable software architects to optimize the performance of a software system early in the development cycle. Architectural performance models and statistical response time models are commonly used to derive these performance predictions. However, both methods have significant downsides: Statistical response time models can only predict scenarios for which training data is available, making the prediction of previously unseen system configurations infeasible. In contrast, the time required to simulate an architectural performance model increases exponentially with both system size and level of modeling detail, making the analysis of large, detailed models challenging. Existing approaches use statistical response time models in architectural performance models to avoid modeling subsystems that are difficult or time-consuming to model, yet they do not consider simulation time. In this paper, we propose to model software systems using classical queuing theory and statistical response time models in parallel. This approach allows users to tailor the model for each analysis run, based on the performed adaptations and the requested performance metrics. Our approach enables faster model solution compared to traditional performance models while retaining their ability to predict previously unseen scenarios. In our experiments we observed speedups of up to 94.8%, making the analysis of much larger and more detailed systems feasible.
• 
  Measuring the Energy Efficiency of Transactional Loads on GPGPU von Kistowski, Jóakim; Pais, Johann; Wahl, Tobias; Lange, Klaus-Dieter; Block, Hansfried; Beckett, John; Kounev, Samuel; in Proceedings of the 19th ACM/SPEC International Conference on Performance Engineering (2019). ACM, New York, NY, USA.

  Best Industry Paper Award
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  General Purpose Graphics Processing Units (GPGPUs) are becoming more and more common in current servers and data centers, which in turn consume a significant amount of electrical power. Measuring and benchmarking this power consumption is important as it helps with optimization and selection of these servers. However, benchmarking and comparing the energy efficiency of GPGPU workloads is challenging as standardized workloads are rare and standardized power and efficiency measurement methods and metrics do not exist. In addition, not all GPGPU systems run at maximum load all the time. Systems that are utilized in transactional, request driven workloads, for example, can run at lower utilization levels. Existing benchmarks for GPGPU systems primarily consider performance and are intended only to run at maximum load. They do not measure performance or energy efficiency at other loads. In turn, server energy-efficiency benchmarks that consider multiple load levels do not address GPGPUs. This paper introduces a measurement methodology for servers with GPGPU accelerators that considers multiple load levels for transactional workloads. The methodology also addresses verifiability of results in order to achieve comparability of different device solutions. We analyze our methodology on three different systems with solutions from two different accelerator vendors. We investigate the efficacy of different methods of load levels scaling and our methodology's reproducibility. We show that the methodology is able to produce consistent and reproducible results with a maximum coefficient of variation of 1.4% regarding power consumption.

  @inproceedings{KiPaWaLaBlBeKo2019-ICPE-GPGPUPower, abstract = {{General Purpose Graphics Processing Units (GPGPUs) are becoming more and more common in current servers and data centers, which in turn consume a significant amount of electrical power. Measuring and benchmarking this power consumption is important as it helps with optimization and selection of these servers. However, benchmarking and comparing the energy efficiency of GPGPU workloads is challenging as standardized workloads are rare and standardized power and efficiency measurement methods and metrics do not exist. In addition, not all GPGPU systems run at maximum load all the time. Systems that are utilized in transactional, request driven workloads, for example, can run at lower utilization levels. Existing benchmarks for GPGPU systems primarily consider performance and are intended only to run at maximum load. They do not measure performance or energy efficiency at other loads. In turn, server energy-efficiency benchmarks that consider multiple load levels do not address GPGPUs. This paper introduces a measurement methodology for servers with GPGPU accelerators that considers multiple load levels for transactional workloads. The methodology also addresses verifiability of results in order to achieve comparability of different device solutions. We analyze our methodology on three different systems with solutions from two different accelerator vendors. We investigate the efficacy of different methods of load levels scaling and our methodology's reproducibility. We show that the methodology is able to produce consistent and reproducible results with a maximum coefficient of variation of 1.4\% regarding power consumption.}}, address = {New York, NY, USA}, author = {von Kistowski, J{\'o}akim and Pais, Johann and Wahl, Tobias and Lange, Klaus-Dieter and Block, Hansfried and Beckett, John and Kounev, Samuel}, booktitle = {Proceedings of the 19th ACM/SPEC International Conference on Performance Engineering}, keywords = {myown}, note = {{<b>Best Industry Paper Award</b>}}, publisher = {ACM}, series = {ICPE '19}, title = {{Measuring the Energy Efficiency of Transactional Loads on GPGPU}}, year = 2019 }

  %0 Conference Paper %1 KiPaWaLaBlBeKo2019-ICPE-GPGPUPower %A von Kistowski, Jóakim %A Pais, Johann %A Wahl, Tobias %A Lange, Klaus-Dieter %A Block, Hansfried %A Beckett, John %A Kounev, Samuel %B Proceedings of the 19th ACM/SPEC International Conference on Performance Engineering %C New York, NY, USA %D 2019 %I ACM %T Measuring the Energy Efficiency of Transactional Loads on GPGPU %X General Purpose Graphics Processing Units (GPGPUs) are becoming more and more common in current servers and data centers, which in turn consume a significant amount of electrical power. Measuring and benchmarking this power consumption is important as it helps with optimization and selection of these servers. However, benchmarking and comparing the energy efficiency of GPGPU workloads is challenging as standardized workloads are rare and standardized power and efficiency measurement methods and metrics do not exist. In addition, not all GPGPU systems run at maximum load all the time. Systems that are utilized in transactional, request driven workloads, for example, can run at lower utilization levels. Existing benchmarks for GPGPU systems primarily consider performance and are intended only to run at maximum load. They do not measure performance or energy efficiency at other loads. In turn, server energy-efficiency benchmarks that consider multiple load levels do not address GPGPUs. This paper introduces a measurement methodology for servers with GPGPU accelerators that considers multiple load levels for transactional workloads. The methodology also addresses verifiability of results in order to achieve comparability of different device solutions. We analyze our methodology on three different systems with solutions from two different accelerator vendors. We investigate the efficacy of different methods of load levels scaling and our methodology's reproducibility. We show that the methodology is able to produce consistent and reproducible results with a maximum coefficient of variation of 1.4% regarding power consumption.
• 
  Predicting Server Power Consumption from Standard Rating Results von Kistowski, Jóakim; Grohmann, Johannes; Schmitt, Norbert; Kounev, Samuel; in Proceedings of the 19th ACM/SPEC International Conference on Performance Engineering (2019). 301–312. Association for Computing Machinery (ACM), New York, NY, USA.

  Full Paper Acceptance Rate: 18.6% (13/70)
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  • [ Download ]
  Data center providers and server operators try to reduce the power consumption of their servers. Finding an energy efficient server for a specific target application is a first step in this regard. Estimating the power consumption of an application on an unavailable server is difficult, as nameplate power values are generally overestimations. Offline power models are able to predict the consumption accurately, but are usually intended for system design, requiring very specific and detailed knowledge about the system under consideration. In this paper, we introduce an offline power prediction method that uses the results of standard power rating tools. The method predicts the power consumption of a specific application for multiple load levels on a target server that is otherwise unavailable for testing. We evaluate our approach by predicting the power consumption of three applications on different physical servers. Our method is able to achieve an average prediction error of 9.49% for three workloads running on real-world, physical servers.

  @inproceedings{KiGrScKo2019-ICPE-PowerPrediction, abstract = {Data center providers and server operators try to reduce the power consumption of their servers. Finding an energy efficient server for a specific target application is a first step in this regard. Estimating the power consumption of an application on an unavailable server is difficult, as nameplate power values are generally overestimations. Offline power models are able to predict the consumption accurately, but are usually intended for system design, requiring very specific and detailed knowledge about the system under consideration. In this paper, we introduce an offline power prediction method that uses the results of standard power rating tools. The method predicts the power consumption of a specific application for multiple load levels on a target server that is otherwise unavailable for testing. We evaluate our approach by predicting the power consumption of three applications on different physical servers. Our method is able to achieve an average prediction error of 9.49% for three workloads running on real-world, physical servers.}, address = {New York, NY, USA}, author = {von Kistowski, J{\'o}akim and Grohmann, Johannes and Schmitt, Norbert and Kounev, Samuel}, booktitle = {Proceedings of the 19th ACM/SPEC International Conference on Performance Engineering}, keywords = {myown}, note = {{Full Paper Acceptance Rate: 18.6\% (13/70)}}, pages = {301–312}, publisher = {Association for Computing Machinery (ACM)}, series = {ICPE '19}, title = {{Predicting Server Power Consumption from Standard Rating Results}}, year = 2019 }

  %0 Conference Paper %1 KiGrScKo2019-ICPE-PowerPrediction %A von Kistowski, Jóakim %A Grohmann, Johannes %A Schmitt, Norbert %A Kounev, Samuel %B Proceedings of the 19th ACM/SPEC International Conference on Performance Engineering %C New York, NY, USA %D 2019 %I Association for Computing Machinery (ACM) %P 301--312 %R 10.1145/3297663.3310298 %T Predicting Server Power Consumption from Standard Rating Results %U https://doi.org/10.1145/3297663.3310298 %X Data center providers and server operators try to reduce the power consumption of their servers. Finding an energy efficient server for a specific target application is a first step in this regard. Estimating the power consumption of an application on an unavailable server is difficult, as nameplate power values are generally overestimations. Offline power models are able to predict the consumption accurately, but are usually intended for system design, requiring very specific and detailed knowledge about the system under consideration. In this paper, we introduce an offline power prediction method that uses the results of standard power rating tools. The method predicts the power consumption of a specific application for multiple load levels on a target server that is otherwise unavailable for testing. We evaluate our approach by predicting the power consumption of three applications on different physical servers. Our method is able to achieve an average prediction error of 9.49% for three workloads running on real-world, physical servers. %@ 9781450362399
• 
  Online Power Consumption Estimation for Functions in Cloud Applications Schmitt, Norbert; Iffländer, Lukas; Bauer, André; Kounev, Samuel; in Proceedings of the 16th IEEE International Conference on Autonomic Computing (ICAC) (2019). IEEE.
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @inproceedings{ScIfBaKo2019-ICAC-OnlinePowerEstimation, author = {Schmitt, Norbert and Iffländer, Lukas and Bauer, André and Kounev, Samuel}, booktitle = {Proceedings of the 16th IEEE International Conference on Autonomic Computing (ICAC)}, keywords = {t_full}, month = {June}, publisher = {IEEE}, title = {Online Power Consumption Estimation for Functions in Cloud Applications}, year = 2019 }

  %0 Conference Paper %1 ScIfBaKo2019-ICAC-OnlinePowerEstimation %A Schmitt, Norbert %A Iffländer, Lukas %A Bauer, André %A Kounev, Samuel %B Proceedings of the 16th IEEE International Conference on Autonomic Computing (ICAC) %D 2019 %I IEEE %T Online Power Consumption Estimation for Functions in Cloud Applications
• 
  Chamulteon: Coordinated Auto-Scaling of Micro-Services Bauer, André; Lesch, Veronika; Versluis, Laurens; Ilyushkin, Alexey; Herbst, Nikolas; Kounev, Samuel; in Proceedings of the 39th IEEE International Conference on Distributed Computing Systems (ICDCS) (2019).
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @inproceedings{BaLeVeIlHeKo2019-ICDCS-Chamulteon, author = {Bauer, Andr{\'e} and Lesch, Veronika and Versluis, Laurens and Ilyushkin, Alexey and Herbst, Nikolas and Kounev, Samuel}, booktitle = {Proceedings of the 39th IEEE International Conference on Distributed Computing Systems (ICDCS)}, keywords = {myown}, month = {July}, title = {Chamulteon: Coordinated Auto-Scaling of Micro-Services}, year = 2019 }

  %0 Conference Paper %1 BaLeVeIlHeKo2019-ICDCS-Chamulteon %A Bauer, André %A Lesch, Veronika %A Versluis, Laurens %A Ilyushkin, Alexey %A Herbst, Nikolas %A Kounev, Samuel %B Proceedings of the 39th IEEE International Conference on Distributed Computing Systems (ICDCS) %D 2019 %T Chamulteon: Coordinated Auto-Scaling of Micro-Services
• 
  Autonomic Forecasting Method Selection: Examination and Ways Ahead Züfle, Marwin; Bauer, André; Lesch, Veronika; Krupitzer, Christian; Herbst, Nikolas; Kounev, Samuel; Curtef, Valentin; in Proceedings of the 16th IEEE International Conference on Autonomic Computing (ICAC) (2019). IEEE.
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @inproceedings{ZuBaLeKrHeKoCu2019-ICAC-Autonomic, author = {Z{\"u}fle, Marwin and Bauer, Andr{\'e} and Lesch, Veronika and Krupitzer, Christian and Herbst, Nikolas and Kounev, Samuel and Curtef, Valentin}, booktitle = {Proceedings of the 16th {IEEE} International Conference on Autonomic Computing (ICAC)}, keywords = {myown}, month = {June}, publisher = {IEEE}, title = {{Autonomic Forecasting Method Selection: Examination and Ways Ahead}}, year = 2019 }

  %0 Conference Paper %1 ZuBaLeKrHeKoCu2019-ICAC-Autonomic %A Züfle, Marwin %A Bauer, André %A Lesch, Veronika %A Krupitzer, Christian %A Herbst, Nikolas %A Kounev, Samuel %A Curtef, Valentin %B Proceedings of the 16th IEEE International Conference on Autonomic Computing (ICAC) %D 2019 %I IEEE %T Autonomic Forecasting Method Selection: Examination and Ways Ahead
• 
  Detecting Parametric Dependencies for Performance Models Using Feature Selection Techniques Grohmann, Johannes; Eismann, Simon; Elflein, Sven; Mazkatli, Manar; von Kistowski, Jóakim; Kounev, Samuel; in 2019 IEEE 27th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS) (2019). 309–322. IEEE Computer Society.

  Acceptance Rate: 23.8% (29/122)
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  • [ Download ]
  Architectural performance models are a common approach to predict the performance properties of a software system. Parametric dependencies, which describe the relation between the input parameters of a component and its performance properties, significantly increase the prediction accuracy of architectural performance models. However, manually modeling parametric dependencies is time-intensive and requires expert knowledge. Existing automated extraction approaches require dedicated performance tests, which are often infeasible. In this paper, we introduce an approach to automatically identify parametric dependencies from monitoring data using feature selection techniques from the area of machine learning. We evaluate the applicability of three techniques selected from each of the three groups of feature selection methods: a filter method, an embedded method, and a wrapper method. Our evaluation shows that the filter technique outperforms the other approaches. Based on these results, we apply this technique to a distributed micro-service web-shop, where it correctly identifies 11 performance-relevant dependencies, achieving a precision of 91.7% based on a manually labeled gold-standard.

  @inproceedings{GrEiElMaKiKo2019-MASCOTS-DependencyIdentification, abstract = {Architectural performance models are a common approach to predict the performance properties of a software system. Parametric dependencies, which describe the relation between the input parameters of a component and its performance properties, significantly increase the prediction accuracy of architectural performance models. However, manually modeling parametric dependencies is time-intensive and requires expert knowledge. Existing automated extraction approaches require dedicated performance tests, which are often infeasible. In this paper, we introduce an approach to automatically identify parametric dependencies from monitoring data using feature selection techniques from the area of machine learning. We evaluate the applicability of three techniques selected from each of the three groups of feature selection methods: a filter method, an embedded method, and a wrapper method. Our evaluation shows that the filter technique outperforms the other approaches. Based on these results, we apply this technique to a distributed micro-service web-shop, where it correctly identifies 11 performance-relevant dependencies, achieving a precision of 91.7% based on a manually labeled gold-standard.}, author = {Grohmann, Johannes and Eismann, Simon and Elflein, Sven and Mazkatli, Manar and von Kistowski, J{\'o}akim and Kounev, Samuel}, booktitle = {2019 IEEE 27th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS)}, keywords = {myown}, month = {October}, note = {{Acceptance Rate: 23.8% (29/122)}}, pages = {309–322}, publisher = {IEEE Computer Society}, series = {MASCOTS '19}, title = {{Detecting Parametric Dependencies for Performance Models Using Feature Selection Techniques}}, year = 2019 }

  %0 Conference Paper %1 GrEiElMaKiKo2019-MASCOTS-DependencyIdentification %A Grohmann, Johannes %A Eismann, Simon %A Elflein, Sven %A Mazkatli, Manar %A von Kistowski, Jóakim %A Kounev, Samuel %B 2019 IEEE 27th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS) %D 2019 %I IEEE Computer Society %P 309--322 %R 10.1109/MASCOTS.2019.00042 %T Detecting Parametric Dependencies for Performance Models Using Feature Selection Techniques %U https://doi.org/10.1109/MASCOTS.2019.00042 %X Architectural performance models are a common approach to predict the performance properties of a software system. Parametric dependencies, which describe the relation between the input parameters of a component and its performance properties, significantly increase the prediction accuracy of architectural performance models. However, manually modeling parametric dependencies is time-intensive and requires expert knowledge. Existing automated extraction approaches require dedicated performance tests, which are often infeasible. In this paper, we introduce an approach to automatically identify parametric dependencies from monitoring data using feature selection techniques from the area of machine learning. We evaluate the applicability of three techniques selected from each of the three groups of feature selection methods: a filter method, an embedded method, and a wrapper method. Our evaluation shows that the filter technique outperforms the other approaches. Based on these results, we apply this technique to a distributed micro-service web-shop, where it correctly identifies 11 performance-relevant dependencies, achieving a precision of 91.7% based on a manually labeled gold-standard. %@ 978-1-7281-4950-9
• 
  Monitorless: Predicting Performance Degradation in Cloud Applications with Machine Learning Grohmann, Johannes; Nicholson, Patrick K.; Iglesias, Jesus Omana; Kounev, Samuel; Lugones, Diego; in Proceedings of the 20th International Middleware Conference (2019). 149–162. Association for Computing Machinery (ACM), New York, NY, USA.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  • [ Download ]
  Today, software operation engineers rely on application key performance indicators (KPIs) for sizing and orchestrating cloud resources dynamically. KPIs are monitored to assess the achievable performance and to configure various cloud-specific parameters such as flavors of instances and autoscaling rules, among others. Usually, keeping KPIs within acceptable levels requires application expertise which is expensive and can slow down the continuous delivery of software. Expertise is required because KPIs are normally based on application-specific quality-of-service metrics, like service response time and processing rate, instead of generic platform metrics, like those typical across various environments (e.g., CPU and memory utilization, I/O rate, etc.)In this paper, we investigate the feasibility of outsourcing the management of application performance from developers to cloud operators. In the same way that the serverless paradigm allows the execution environment to be fully managed by a third party, we discuss a monitorless model to streamline application deployment by delegating performance management. We show that training a machine learning model with platform-level data, collected from the execution of representative containerized services, allows inferring application KPI degradation. This is an opportunity to simplify operations as engineers can rely solely on platform metrics -- while still fulfilling application KPIs -- to configure portable and application agnostic rules and other cloud-specific parameters to automatically trigger actions such as autoscaling, instance migration, network slicing, etc.Results show that monitorless infers KPI degradation with an accuracy of 97% and, notably, it performs similarly to typical autoscaling solutions, even when autoscaling rules are optimally tuned with knowledge of the expected workload.

  @inproceedings{GrNiIgKoLu-MIDDLEWARE2019-Monitorless, abstract = {Today, software operation engineers rely on application key performance indicators (KPIs) for sizing and orchestrating cloud resources dynamically. KPIs are monitored to assess the achievable performance and to configure various cloud-specific parameters such as flavors of instances and autoscaling rules, among others. Usually, keeping KPIs within acceptable levels requires application expertise which is expensive and can slow down the continuous delivery of software. Expertise is required because KPIs are normally based on application-specific quality-of-service metrics, like service response time and processing rate, instead of generic platform metrics, like those typical across various environments (e.g., CPU and memory utilization, I/O rate, etc.)In this paper, we investigate the feasibility of outsourcing the management of application performance from developers to cloud operators. In the same way that the serverless paradigm allows the execution environment to be fully managed by a third party, we discuss a monitorless model to streamline application deployment by delegating performance management. We show that training a machine learning model with platform-level data, collected from the execution of representative containerized services, allows inferring application KPI degradation. This is an opportunity to simplify operations as engineers can rely solely on platform metrics – while still fulfilling application KPIs – to configure portable and application agnostic rules and other cloud-specific parameters to automatically trigger actions such as autoscaling, instance migration, network slicing, etc.Results show that monitorless infers KPI degradation with an accuracy of 97% and, notably, it performs similarly to typical autoscaling solutions, even when autoscaling rules are optimally tuned with knowledge of the expected workload.}, address = {New York, NY, USA}, author = {Grohmann, Johannes and Nicholson, Patrick K. and Iglesias, Jesus Omana and Kounev, Samuel and Lugones, Diego}, booktitle = {Proceedings of the 20th International Middleware Conference}, keywords = {myown}, pages = {149–162}, publisher = {Association for Computing Machinery (ACM)}, series = {Middleware '19}, title = {Monitorless: Predicting Performance Degradation in Cloud Applications with Machine Learning}, year = 2019 }

  %0 Conference Paper %1 GrNiIgKoLu-MIDDLEWARE2019-Monitorless %A Grohmann, Johannes %A Nicholson, Patrick K. %A Iglesias, Jesus Omana %A Kounev, Samuel %A Lugones, Diego %B Proceedings of the 20th International Middleware Conference %C New York, NY, USA %D 2019 %I Association for Computing Machinery (ACM) %P 149--162 %R 10.1145/3361525.3361543 %T Monitorless: Predicting Performance Degradation in Cloud Applications with Machine Learning %U https://doi.org/10.1145/3361525.3361543 %X Today, software operation engineers rely on application key performance indicators (KPIs) for sizing and orchestrating cloud resources dynamically. KPIs are monitored to assess the achievable performance and to configure various cloud-specific parameters such as flavors of instances and autoscaling rules, among others. Usually, keeping KPIs within acceptable levels requires application expertise which is expensive and can slow down the continuous delivery of software. Expertise is required because KPIs are normally based on application-specific quality-of-service metrics, like service response time and processing rate, instead of generic platform metrics, like those typical across various environments (e.g., CPU and memory utilization, I/O rate, etc.)In this paper, we investigate the feasibility of outsourcing the management of application performance from developers to cloud operators. In the same way that the serverless paradigm allows the execution environment to be fully managed by a third party, we discuss a monitorless model to streamline application deployment by delegating performance management. We show that training a machine learning model with platform-level data, collected from the execution of representative containerized services, allows inferring application KPI degradation. This is an opportunity to simplify operations as engineers can rely solely on platform metrics -- while still fulfilling application KPIs -- to configure portable and application agnostic rules and other cloud-specific parameters to automatically trigger actions such as autoscaling, instance migration, network slicing, etc.Results show that monitorless infers KPI degradation with an accuracy of 97% and, notably, it performs similarly to typical autoscaling solutions, even when autoscaling rules are optimally tuned with knowledge of the expected workload. %@ 9781450370097

• 
  Modeling of Parametric Dependencies for Performance Prediction of Component-based Software Systems at Run-time Eismann, Simon; Walter, Jürgen; von Kistowski, Jóakim; Kounev, Samuel; in 2018 IEEE International Conference on Software Architecture (ICSA) (2018). 135–144.

  Acceptance Rate: 25,6% (22/86)
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Model-based performance analysis can be leveraged to explore performance properties of software systems. To capture the behavior of varying workload mixes, configurations, and deployments of a software system requires formal modeling of the impact of configuration parameters and user input on the system behavior. Such influences are represented as parametric dependencies in software performance models. Existing modeling approaches focus on modeling parametric dependencies at design-time. This paper identifies runtime specific parametric dependency features, which are not supported by existing work. Therefore, this paper proposes a novel modeling methodology for parametric dependencies and a corresponding graph-based resolution algorithm. This algorithm enables the solution of models containing component instance-level dependencies, variables with multiple descriptions in parallel, and correlations modeled as parametric dependencies. We integrate our work into the Descartes Modeling Language (DML), allowing for accurate and efficient modeling and analysis of parametric dependencies. These performance predictions are valuable for various purposes such as capacity planning, bottleneck analysis, configuration optimization and proactive auto-scaling. Our evaluation analyzes a video store application. The prediction for varying language mixes and video sizes shows a mean error below 5% for utilization and below 10% for response time.

  @inproceedings{EiWaKiKo2018-ParametricDependencies, abstract = {Model-based performance analysis can be leveraged to explore performance properties of software systems. To capture the behavior of varying workload mixes, configurations, and deployments of a software system requires formal modeling of the impact of configuration parameters and user input on the system behavior. Such influences are represented as parametric dependencies in software performance models. Existing modeling approaches focus on modeling parametric dependencies at design-time. This paper identifies runtime specific parametric dependency features, which are not supported by existing work. Therefore, this paper proposes a novel modeling methodology for parametric dependencies and a corresponding graph-based resolution algorithm. This algorithm enables the solution of models containing component instance-level dependencies, variables with multiple descriptions in parallel, and correlations modeled as parametric dependencies. We integrate our work into the Descartes Modeling Language (DML), allowing for accurate and efficient modeling and analysis of parametric dependencies. These performance predictions are valuable for various purposes such as capacity planning, bottleneck analysis, configuration optimization and proactive auto-scaling. Our evaluation analyzes a video store application. The prediction for varying language mixes and video sizes shows a mean error below 5% for utilization and below 10% for response time.}, author = {Eismann, Simon and Walter, J{\"u}rgen and von Kistowski, J{\'o}akim and Kounev, Samuel}, booktitle = {2018 IEEE International Conference on Software Architecture (ICSA)}, keywords = {myown}, month = {April}, note = {Acceptance Rate: 25,6\% (22/86)}, pages = {135-144}, title = {{Modeling of Parametric Dependencies for Performance Prediction of Component-based Software Systems at Run-time}}, year = 2018 }

  %0 Conference Paper %1 EiWaKiKo2018-ParametricDependencies %A Eismann, Simon %A Walter, Jürgen %A von Kistowski, Jóakim %A Kounev, Samuel %B 2018 IEEE International Conference on Software Architecture (ICSA) %D 2018 %P 135-144 %T Modeling of Parametric Dependencies for Performance Prediction of Component-based Software Systems at Run-time %U https://ieeexplore.ieee.org/document/8417146 %X Model-based performance analysis can be leveraged to explore performance properties of software systems. To capture the behavior of varying workload mixes, configurations, and deployments of a software system requires formal modeling of the impact of configuration parameters and user input on the system behavior. Such influences are represented as parametric dependencies in software performance models. Existing modeling approaches focus on modeling parametric dependencies at design-time. This paper identifies runtime specific parametric dependency features, which are not supported by existing work. Therefore, this paper proposes a novel modeling methodology for parametric dependencies and a corresponding graph-based resolution algorithm. This algorithm enables the solution of models containing component instance-level dependencies, variables with multiple descriptions in parallel, and correlations modeled as parametric dependencies. We integrate our work into the Descartes Modeling Language (DML), allowing for accurate and efficient modeling and analysis of parametric dependencies. These performance predictions are valuable for various purposes such as capacity planning, bottleneck analysis, configuration optimization and proactive auto-scaling. Our evaluation analyzes a video store application. The prediction for varying language mixes and video sizes shows a mean error below 5% for utilization and below 10% for response time.
• 
  FOX: Cost-Awareness for Autonomic Resource Management in Public Clouds Lesch, Veronika; Bauer, André; Herbst, Nikolas; Kounev, Samuel; in Proceedings of the 9th ACM/SPEC International Conference on Performance Engineering (ICPE 2018) (2018). ACM, New York, NY, USA.

  Full Paper Acceptance Ratio: 23,7%
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  @inproceedings{LeBaHeKo-ICPE18-FOX-AS-COST, address = {New York, NY, USA}, author = {Lesch, Veronika and Bauer, Andr{\'e} and Herbst, Nikolas and Kounev, Samuel}, booktitle = {Proceedings of the 9th ACM/SPEC International Conference on Performance Engineering (ICPE 2018)}, keywords = {Prediction}, month = {April}, note = {Full Paper Acceptance Ratio: 23,7%}, publisher = {ACM}, title = {{FOX: Cost-Awareness for Autonomic Resource Management in Public Clouds}}, year = 2018 }

  %0 Conference Paper %1 LeBaHeKo-ICPE18-FOX-AS-COST %A Lesch, Veronika %A Bauer, André %A Herbst, Nikolas %A Kounev, Samuel %B Proceedings of the 9th ACM/SPEC International Conference on Performance Engineering (ICPE 2018) %C New York, NY, USA %D 2018 %I ACM %T FOX: Cost-Awareness for Autonomic Resource Management in Public Clouds %U https://doi.org/10.1145/3184407.3184415
• 
  On the Value of Service Demand Estimation for Auto-Scaling Bauer, André; Grohmann, Johannes; Herbst, Nikolas; Kounev, Samuel; in Proceedings of 19th International GI/ITG Conference on Measurement, Modelling and Evaluation of Computing Systems (MMB 2018) (2018). (Vol. 10740) 142–156. Springer, Cham.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  • [ Download ]
  In the context of performance models, service demands are key model parameters capturing the average time individual requests of different workload classes are actively processed. In a system under load, due to measurement interference, service demands normally cannot be measured directly, however, a number of estimation approaches exist based on high-level performance metrics. In this paper, we show that service demands provide significant benefits for implementing modern auto-scalers. Auto-scaling describes the process of dynamically adjusting the number of allocated virtual resources (e.g., virtual machines) in a data center according to the incoming workload. We demonstrate that even a simple auto-scaler that leverages information about service demands significantly outperforms auto-scalers solely based on CPU utilization measurements. This is shown by testing two approaches in three different scenarios. Our results show that the service demand-based auto-scaler outperforms the CPU utilization-based one in all scenarios. Our results encourage further research on the application of service demand estimates for resource management in data centers.

  @inproceedings{BaGrHeKo2018-MMB-ServiceDemand, abstract = {In the context of performance models, service demands are key model parameters capturing the average time individual requests of different workload classes are actively processed. In a system under load, due to measurement interference, service demands normally cannot be measured directly, however, a number of estimation approaches exist based on high-level performance metrics. In this paper, we show that service demands provide significant benefits for implementing modern auto-scalers. Auto-scaling describes the process of dynamically adjusting the number of allocated virtual resources (e.g., virtual machines) in a data center according to the incoming workload. We demonstrate that even a simple auto-scaler that leverages information about service demands significantly outperforms auto-scalers solely based on CPU utilization measurements. This is shown by testing two approaches in three different scenarios. Our results show that the service demand-based auto-scaler outperforms the CPU utilization-based one in all scenarios. Our results encourage further research on the application of service demand estimates for resource management in data centers.}, address = {Cham}, author = {Bauer, Andr{\'e} and Grohmann, Johannes and Herbst, Nikolas and Kounev, Samuel}, booktitle = {Proceedings of 19th International GI/ITG Conference on Measurement, Modelling and Evaluation of Computing Systems (MMB 2018)}, keywords = {myown}, month = {February}, pages = {142–156}, publisher = {Springer}, series = {Lecture Notes in Computer Science}, title = {{On the Value of Service Demand Estimation for Auto-Scaling}}, volume = 10740, year = 2018 }

  %0 Conference Paper %1 BaGrHeKo2018-MMB-ServiceDemand %A Bauer, André %A Grohmann, Johannes %A Herbst, Nikolas %A Kounev, Samuel %B Proceedings of 19th International GI/ITG Conference on Measurement, Modelling and Evaluation of Computing Systems (MMB 2018) %C Cham %D 2018 %I Springer %P 142--156 %R 10.1007/978-3-319-74947-1_10 %T On the Value of Service Demand Estimation for Auto-Scaling %U https://doi.org/10.1007/978-3-319-74947-1_10 %V 10740 %X In the context of performance models, service demands are key model parameters capturing the average time individual requests of different workload classes are actively processed. In a system under load, due to measurement interference, service demands normally cannot be measured directly, however, a number of estimation approaches exist based on high-level performance metrics. In this paper, we show that service demands provide significant benefits for implementing modern auto-scalers. Auto-scaling describes the process of dynamically adjusting the number of allocated virtual resources (e.g., virtual machines) in a data center according to the incoming workload. We demonstrate that even a simple auto-scaler that leverages information about service demands significantly outperforms auto-scalers solely based on CPU utilization measurements. This is shown by testing two approaches in three different scenarios. Our results show that the service demand-based auto-scaler outperforms the CPU utilization-based one in all scenarios. Our results encourage further research on the application of service demand estimates for resource management in data centers. %@ 978-3-319-74947-1
• 
  SmarTor: Smarter Tor with Smart Contracts: Improving resilience of topology distribution in the Tor network Greubel, Andre; Dmitrienko, Alexandra; Kounev, Samuel; in Proceedings of the 34th Annual Computer Security Applications Conference (2018). ACM, New York, NY, USA.
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @inproceedings{GrDmKo2018-ACSAC-SmarTor, address = {New York, NY, USA}, author = {Greubel, Andre and Dmitrienko, Alexandra and Kounev, Samuel}, booktitle = {Proceedings of the 34th Annual Computer Security Applications Conference}, keywords = {myown}, month = {December}, organization = {ACM}, series = {ACSAC '18}, title = {{SmarTor: Smarter Tor with Smart Contracts: Improving resilience of topology distribution in the Tor network}}, year = 2018 }

  %0 Conference Paper %1 GrDmKo2018-ACSAC-SmarTor %A Greubel, Andre %A Dmitrienko, Alexandra %A Kounev, Samuel %B Proceedings of the 34th Annual Computer Security Applications Conference %C New York, NY, USA %D 2018 %T SmarTor: Smarter Tor with Smart Contracts: Improving resilience of topology distribution in the Tor network
• 
  TeaStore: A Micro-Service Reference Application for Benchmarking, Modeling and Resource Management Research von Kistowski, Jóakim; Eismann, Simon; Schmitt, Norbert; Bauer, André; Grohmann, Johannes; Kounev, Samuel; in Proceedings of the 26th IEEE International Symposium on the Modelling, Analysis, and Simulation of Computer and Telecommunication Systems (2018). 223–236. IEEE Computer Society.

  Acceptance Rate: 29.5% (23/78)
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ DOI ]
  • [ Download ]
  Modern distributed applications offer complex performance behavior and many degrees of freedom regarding deployment and configuration. Researchers employ various methods of analysis, modeling, and management that leverage these degrees of freedom to predict or improve non-functional properties of the software under consideration. In order to demonstrate and evaluate their applicability in the real world, methods resulting from such research areas require test and reference applications that offer a range of different behaviors, as well as the necessary degrees of freedom. Existing production software is often inaccessible for researchers or closed off to instrumentation. Existing testing and benchmarking frameworks, on the other hand, are either designed for specific testing scenarios, or they do not offer the necessary degrees of freedom. Further, most test applications are difficult to deploy and run, or are outdated. In this paper, we introduce the TeaStore, a state-of-the-art micro-service-based test and reference application. TeaStore offers services with different performance characteristics and many degrees of freedom regarding deployment and configuration to be used as a benchmarking framework for researchers. The TeaStore allows evaluating performance modeling and resource management techniques; it also offers instrumented variants to enable extensive run-time analysis. We demonstrate TeaStore's use in three contexts: performance modeling, cloud resource management, and energy efficiency analysis. Our experiments show that TeaStore can be used for evaluating novel approaches in these contexts and also motivates further research in the areas of performance modeling and resource management.

  @inproceedings{KiEiScBaGrKo2018-MASCOTS-TeaStore, abstract = {Modern distributed applications offer complex performance behavior and many degrees of freedom regarding deployment and configuration. Researchers employ various methods of analysis, modeling, and management that leverage these degrees of freedom to predict or improve non-functional properties of the software under consideration. In order to demonstrate and evaluate their applicability in the real world, methods resulting from such research areas require test and reference applications that offer a range of different behaviors, as well as the necessary degrees of freedom. Existing production software is often inaccessible for researchers or closed off to instrumentation. Existing testing and benchmarking frameworks, on the other hand, are either designed for specific testing scenarios, or they do not offer the necessary degrees of freedom. Further, most test applications are difficult to deploy and run, or are outdated. In this paper, we introduce the TeaStore, a state-of-the-art micro-service-based test and reference application. TeaStore offers services with different performance characteristics and many degrees of freedom regarding deployment and configuration to be used as a benchmarking framework for researchers. The TeaStore allows evaluating performance modeling and resource management techniques; it also offers instrumented variants to enable extensive run-time analysis. We demonstrate TeaStore's use in three contexts: performance modeling, cloud resource management, and energy efficiency analysis. Our experiments show that TeaStore can be used for evaluating novel approaches in these contexts and also motivates further research in the areas of performance modeling and resource management.}, author = {von Kistowski, J{\'o}akim and Eismann, Simon and Schmitt, Norbert and Bauer, Andr{\'e} and Grohmann, Johannes and Kounev, Samuel}, booktitle = {Proceedings of the 26th IEEE International Symposium on the Modelling, Analysis, and Simulation of Computer and Telecommunication Systems}, keywords = {myown}, month = {September}, note = {{Acceptance Rate: 29.5\% (23/78)}}, pages = {223–236}, publisher = {IEEE Computer Society}, series = {MASCOTS '18}, title = {{TeaStore: A Micro-Service Reference Application for Benchmarking, Modeling and Resource Management Research}}, year = 2018 }

  %0 Conference Paper %1 KiEiScBaGrKo2018-MASCOTS-TeaStore %A von Kistowski, Jóakim %A Eismann, Simon %A Schmitt, Norbert %A Bauer, André %A Grohmann, Johannes %A Kounev, Samuel %B Proceedings of the 26th IEEE International Symposium on the Modelling, Analysis, and Simulation of Computer and Telecommunication Systems %D 2018 %I IEEE Computer Society %P 223--236 %R 10.1109/MASCOTS.2018.00030 %T TeaStore: A Micro-Service Reference Application for Benchmarking, Modeling and Resource Management Research %U https://doi.org/10.1109/MASCOTS.2018.00030 %X Modern distributed applications offer complex performance behavior and many degrees of freedom regarding deployment and configuration. Researchers employ various methods of analysis, modeling, and management that leverage these degrees of freedom to predict or improve non-functional properties of the software under consideration. In order to demonstrate and evaluate their applicability in the real world, methods resulting from such research areas require test and reference applications that offer a range of different behaviors, as well as the necessary degrees of freedom. Existing production software is often inaccessible for researchers or closed off to instrumentation. Existing testing and benchmarking frameworks, on the other hand, are either designed for specific testing scenarios, or they do not offer the necessary degrees of freedom. Further, most test applications are difficult to deploy and run, or are outdated. In this paper, we introduce the TeaStore, a state-of-the-art micro-service-based test and reference application. TeaStore offers services with different performance characteristics and many degrees of freedom regarding deployment and configuration to be used as a benchmarking framework for researchers. The TeaStore allows evaluating performance modeling and resource management techniques; it also offers instrumented variants to enable extensive run-time analysis. We demonstrate TeaStore's use in three contexts: performance modeling, cloud resource management, and energy efficiency analysis. Our experiments show that TeaStore can be used for evaluating novel approaches in these contexts and also motivates further research in the areas of performance modeling and resource management.

• 
  Performance Assessment of Cloud Migrations from Network and Application Point of View Iffländer, Lukas; Wamser, Florian; Metter, Christopher; Tran-Gia, Phuoc; Kounev, Samuel; in Proceedings of 9th EAI International Conference on Mobile Networks and Management (MONAMI 2018) (2017).
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @inproceedings{IfWaMeTrKo-Monami-Migration-Performance, author = {Iffländer, Lukas and Wamser, Florian and Metter, Christopher and Tran-Gia, Phuoc and Kounev, Samuel}, booktitle = {{Proceedings of 9th EAI International Conference on Mobile Networks and Management (MONAMI 2018)}}, keywords = {t_full}, month = {December}, title = {Performance Assessment of Cloud Migrations from Network and Application Point of View}, year = 2017 }

  %0 Conference Paper %1 IfWaMeTrKo-Monami-Migration-Performance %A Iffländer, Lukas %A Wamser, Florian %A Metter, Christopher %A Tran-Gia, Phuoc %A Kounev, Samuel %B Proceedings of 9th EAI International Conference on Mobile Networks and Management (MONAMI 2018) %D 2017 %T Performance Assessment of Cloud Migrations from Network and Application Point of View
• 
  Automated and Adaptable Decision Support for Software Performance Engineering Walter, Jürgen; van Hoorn, Andre; Kounev, Samuel; in Proceedings of the 11th EAI International Conference on Performance Evaluation Methodologies and Tools (2017).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Software performance engineering (SPE) provides a plethora of methods and tooling for measuring, modeling, and evaluating performance properties of software systems. The solution approaches come with different strengths and limitations concerning, for example, accuracy, time-to-result, or system overhead. While approaches allow for interchangeability, the choice of an appropriate approach and tooling to solve a given performance concern still relies on expert knowledge. Currently, there is no automated and extensible approach for decision support. In this paper, we present a methodology for the automated selection of performance engineering approaches tailored to user concerns. We decouple the complexity of selecting an SPE approach for a given scenario providing a decision engine and solution approach capability models. This separation allows to easily append additional solution approaches and rating criteria. We demonstrate the applicability by presenting decision engines that compare measurement- and model-based analysis approaches.

  @inproceedings{WalterHoKo2017-DecisionSupport, abstract = {Software performance engineering (SPE) provides a plethora of methods and tooling for measuring, modeling, and evaluating performance properties of software systems. The solution approaches come with different strengths and limitations concerning, for example, accuracy, time-to-result, or system overhead. While approaches allow for interchangeability, the choice of an appropriate approach and tooling to solve a given performance concern still relies on expert knowledge. Currently, there is no automated and extensible approach for decision support. In this paper, we present a methodology for the automated selection of performance engineering approaches tailored to user concerns. We decouple the complexity of selecting an SPE approach for a given scenario providing a decision engine and solution approach capability models. This separation allows to easily append additional solution approaches and rating criteria. We demonstrate the applicability by presenting decision engines that compare measurement- and model-based analysis approaches.}, author = {Walter, J{\"u}rgen and van Hoorn, Andre and Kounev, Samuel}, booktitle = {Proceedings of the 11th EAI International Conference on Performance Evaluation Methodologies and Tools}, keywords = {myown}, month = 12, title = {{Automated and Adaptable Decision Support for Software Performance Engineering}}, year = 2017 }

  %0 Conference Paper %1 WalterHoKo2017-DecisionSupport %A Walter, Jürgen %A van Hoorn, Andre %A Kounev, Samuel %B Proceedings of the 11th EAI International Conference on Performance Evaluation Methodologies and Tools %D 2017 %T Automated and Adaptable Decision Support for Software Performance Engineering %X Software performance engineering (SPE) provides a plethora of methods and tooling for measuring, modeling, and evaluating performance properties of software systems. The solution approaches come with different strengths and limitations concerning, for example, accuracy, time-to-result, or system overhead. While approaches allow for interchangeability, the choice of an appropriate approach and tooling to solve a given performance concern still relies on expert knowledge. Currently, there is no automated and extensible approach for decision support. In this paper, we present a methodology for the automated selection of performance engineering approaches tailored to user concerns. We decouple the complexity of selecting an SPE approach for a given scenario providing a decision engine and solution approach capability models. This separation allows to easily append additional solution approaches and rating criteria. We demonstrate the applicability by presenting decision engines that compare measurement- and model-based analysis approaches.

• 
  Variations in CPU Power Consumption von Kistowski, Jóakim; Block, Hansfried; Beckett, John; Spradling, Cloyce; Lange, Klaus-Dieter; Kounev, Samuel; in Proceedings of the 7th ACM/SPEC International Conference on Performance Engineering (ICPE 2016) (2016). ACM, New York, NY, USA.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Experimental analysis of computer systems' power consumption has become an integral part of system performance evaluation, efficiency management, and model-based analysis. As with all measurements, repeatability and reproducibility of power measurements is a major challenge. Nominally identical systems can have different power consumption running the same workload under otherwise identical conditions. This behavior can also be observed for individual system components. Specifically, CPU power consumption can vary amongst different samples of nominally identical CPUs. This in turn has a significant impact on the overall system power, considering that a system's processor is the largest and most dynamic power consumer of the overall system. The concrete impact of CPU sample power variations is unknown, as comprehensive studies about differences in power consumption for nominally identical systems are currently missing. We address this lack of studies by conducting measurements on four different processor types from two different architectures. For each of these types, we compare up to 30 physical processor samples with a total sum of 90 samples over all processor types. We analyze the variations in power consumption for the different samples using six different workloads over five load levels. Additionally, we analyze how these variations change for different processor core counts and architectures. The results of this paper show that selection of a processor sample can have a statistically significant impact on power consumption. With no correlation to performance, power consumption for nominally identical processors can differ as much as 29.6% in idle and 19.5% at full load. We also show that these variations change over different architectures and processor types.

  @inproceedings{KiBlBeSpLaKo2016-ICPE-PowerVariation, abstract = {{Experimental analysis of computer systems' power consumption has become an integral part of system performance evaluation, efficiency management, and model-based analysis. As with all measurements, repeatability and reproducibility of power measurements is a major challenge. Nominally identical systems can have different power consumption running the same workload under otherwise identical conditions. This behavior can also be observed for individual system components. Specifically, CPU power consumption can vary amongst different samples of nominally identical CPUs. This in turn has a significant impact on the overall system power, considering that a system's processor is the largest and most dynamic power consumer of the overall system. The concrete impact of CPU sample power variations is unknown, as comprehensive studies about differences in power consumption for nominally identical systems are currently missing. We address this lack of studies by conducting measurements on four different processor types from two different architectures. For each of these types, we compare up to 30 physical processor samples with a total sum of 90 samples over all processor types. We analyze the variations in power consumption for the different samples using six different workloads over five load levels. Additionally, we analyze how these variations change for different processor core counts and architectures. The results of this paper show that selection of a processor sample can have a statistically significant impact on power consumption. With no correlation to performance, power consumption for nominally identical processors can differ as much as 29.6\% in idle and 19.5\% at full load. We also show that these variations change over different architectures and processor types.}}, address = {New York, NY, USA}, author = {von Kistowski, J{\'o}akim and Block, Hansfried and Beckett, John and Spradling, Cloyce and Lange, Klaus-Dieter and Kounev, Samuel}, booktitle = {Proceedings of the 7th ACM/SPEC International Conference on Performance Engineering (ICPE 2016)}, keywords = {myown}, month = {March}, publisher = {ACM}, title = {{Variations in CPU Power Consumption}}, year = 2016 }

  %0 Conference Paper %1 KiBlBeSpLaKo2016-ICPE-PowerVariation %A von Kistowski, Jóakim %A Block, Hansfried %A Beckett, John %A Spradling, Cloyce %A Lange, Klaus-Dieter %A Kounev, Samuel %B Proceedings of the 7th ACM/SPEC International Conference on Performance Engineering (ICPE 2016) %C New York, NY, USA %D 2016 %I ACM %T Variations in CPU Power Consumption %X Experimental analysis of computer systems' power consumption has become an integral part of system performance evaluation, efficiency management, and model-based analysis. As with all measurements, repeatability and reproducibility of power measurements is a major challenge. Nominally identical systems can have different power consumption running the same workload under otherwise identical conditions. This behavior can also be observed for individual system components. Specifically, CPU power consumption can vary amongst different samples of nominally identical CPUs. This in turn has a significant impact on the overall system power, considering that a system's processor is the largest and most dynamic power consumer of the overall system. The concrete impact of CPU sample power variations is unknown, as comprehensive studies about differences in power consumption for nominally identical systems are currently missing. We address this lack of studies by conducting measurements on four different processor types from two different architectures. For each of these types, we compare up to 30 physical processor samples with a total sum of 90 samples over all processor types. We analyze the variations in power consumption for the different samples using six different workloads over five load levels. Additionally, we analyze how these variations change for different processor core counts and architectures. The results of this paper show that selection of a processor sample can have a statistically significant impact on power consumption. With no correlation to performance, power consumption for nominally identical processors can differ as much as 29.6% in idle and 19.5% at full load. We also show that these variations change over different architectures and processor types.
• 
  Automated Extraction of Network Traffic Models Suitable for Performance Simulation Rygielski, Piotr; Simko, Viliam; Sittner, Felix; Aschenbrenner, Doris; Kounev, Samuel; Schilling, Klaus; in Proceedings of the 7th ACM/SPEC International Conference on Performance Engineering (ICPE 2016) (2016). 27–35. ACM.

  Acceptance rate (Full Paper): 19/57 = 33%
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Data centers are increasingly becoming larger and dynamic due to virtualization. In order to leverage the performance modeling and prediction techniques, such as Palladio Component Model or Descartes Modeling Language, in such a dynamic environments, it is necessary to automate the model extraction. Building and maintaining such models manually is not feasible anymore due to their size and the level of details. This paper is focused on traffic models that are an essential part of network infrastructure. Our goal is to decompose real traffic dumps into models suitable for performance prediction using Descartes Network Infrastructure modeling approach. The main challenge was to efficiently encode an arbitrary signal in the form of simple traffic generators while maintaining the shape of the original signal. We show that a typical 15 minute long tcpdump trace can be compressed to 0.4--15% of its original size whereas the relative median of extraction error is 0% for the most of the 69 examined traces.

  @inproceedings{RySiSiAschKoSch2016-dni-traffic-extraction-icpe2016, abstract = {{Data centers are increasingly becoming larger and dynamic due to virtualization. In order to leverage the performance modeling and prediction techniques, such as Palladio Component Model or Descartes Modeling Language, in such a dynamic environments, it is necessary to automate the model extraction. Building and maintaining such models manually is not feasible anymore due to their size and the level of details. This paper is focused on traffic models that are an essential part of network infrastructure. Our goal is to decompose real traffic dumps into models suitable for performance prediction using Descartes Network Infrastructure modeling approach. The main challenge was to efficiently encode an arbitrary signal in the form of simple traffic generators while maintaining the shape of the original signal. We show that a typical 15 minute long tcpdump trace can be compressed to 0.4–15% of its original size whereas the relative median of extraction error is 0% for the most of the 69 examined traces.}}, author = {Rygielski, Piotr and Simko, Viliam and Sittner, Felix and Aschenbrenner, Doris and Kounev, Samuel and Schilling, Klaus}, booktitle = {Proceedings of the 7th ACM/SPEC International Conference on Performance Engineering (ICPE 2016)}, keywords = {descartes}, month = {March}, note = {{Acceptance rate (Full Paper): 19/57 = 33\%}}, pages = {27–35}, publisher = {ACM}, title = {{Automated Extraction of Network Traffic Models Suitable for Performance Simulation}}, year = 2016 }

  %0 Conference Paper %1 RySiSiAschKoSch2016-dni-traffic-extraction-icpe2016 %A Rygielski, Piotr %A Simko, Viliam %A Sittner, Felix %A Aschenbrenner, Doris %A Kounev, Samuel %A Schilling, Klaus %B Proceedings of the 7th ACM/SPEC International Conference on Performance Engineering (ICPE 2016) %D 2016 %I ACM %P 27--35 %T Automated Extraction of Network Traffic Models Suitable for Performance Simulation %U http://dl.acm.org/citation.cfm?doid=2851553.2851570 %X Data centers are increasingly becoming larger and dynamic due to virtualization. In order to leverage the performance modeling and prediction techniques, such as Palladio Component Model or Descartes Modeling Language, in such a dynamic environments, it is necessary to automate the model extraction. Building and maintaining such models manually is not feasible anymore due to their size and the level of details. This paper is focused on traffic models that are an essential part of network infrastructure. Our goal is to decompose real traffic dumps into models suitable for performance prediction using Descartes Network Infrastructure modeling approach. The main challenge was to efficiently encode an arbitrary signal in the form of simple traffic generators while maintaining the shape of the original signal. We show that a typical 15 minute long tcpdump trace can be compressed to 0.4--15% of its original size whereas the relative median of extraction error is 0% for the most of the 69 examined traces.
• 
  Modeling and Prediction of Software-Defined Networks Performance using Queueing Petri Nets Rygielski, Piotr; Seliuchenko, Marian; Kounev, Samuel; in Proceedings of the Ninth International Conference on Simulation Tools and Techniques (SIMUTools 2016) (2016). 66–75.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Using various modeling and simulation approaches for predicting network performance requires extensive experience and involves a number of time consuming manual steps regarding each of the modeling formalisms. Descartes Network Infrastructure (DNI) is a data center network performance modeling approach that addresses this challenge by offering multiple performance models but requiring to use only a single modeling language. In this paper, we thoroughly extend DNI to support new networking paradigms like, among others, Software-Defined Networking (SDN) and Network-Function Virtualization (NFV). Additionally, we demonstrate how SDN-based networks can be modeled using DNI and how are they transformed later into Queueing Petri Nets (QPN) using a model-to-model transformation. In the analysis of the performance prediction accuracy, we show that automatically generated QPN models represent the performance of heterogeneous SDN hardware with maximal prediction accuracy error of 12 percent.

  @inproceedings{RySeKo2016-SIMUTools-SDN-switches-QPN, abstract = {{Using various modeling and simulation approaches for predicting network performance requires extensive experience and involves a number of time consuming manual steps regarding each of the modeling formalisms. Descartes Network Infrastructure (DNI) is a data center network performance modeling approach that addresses this challenge by offering multiple performance models but requiring to use only a single modeling language. In this paper, we thoroughly extend DNI to support new networking paradigms like, among others, Software-Defined Networking (SDN) and Network-Function Virtualization (NFV). Additionally, we demonstrate how SDN-based networks can be modeled using DNI and how are they transformed later into Queueing Petri Nets (QPN) using a model-to-model transformation. In the analysis of the performance prediction accuracy, we show that automatically generated QPN models represent the performance of heterogeneous SDN hardware with maximal prediction accuracy error of 12 percent.}}, author = {Rygielski, Piotr and Seliuchenko, Marian and Kounev, Samuel}, booktitle = {Proceedings of the Ninth International Conference on Simulation Tools and Techniques (SIMUTools 2016)}, keywords = {myown}, month = {August}, pages = {66–75}, title = {{Modeling and Prediction of Software-Defined Networks Performance using Queueing Petri Nets}}, year = 2016 }

  %0 Conference Paper %1 RySeKo2016-SIMUTools-SDN-switches-QPN %A Rygielski, Piotr %A Seliuchenko, Marian %A Kounev, Samuel %B Proceedings of the Ninth International Conference on Simulation Tools and Techniques (SIMUTools 2016) %D 2016 %P 66--75 %T Modeling and Prediction of Software-Defined Networks Performance using Queueing Petri Nets %U http://dl.acm.org/citation.cfm?id=3021426.3021437 %X Using various modeling and simulation approaches for predicting network performance requires extensive experience and involves a number of time consuming manual steps regarding each of the modeling formalisms. Descartes Network Infrastructure (DNI) is a data center network performance modeling approach that addresses this challenge by offering multiple performance models but requiring to use only a single modeling language. In this paper, we thoroughly extend DNI to support new networking paradigms like, among others, Software-Defined Networking (SDN) and Network-Function Virtualization (NFV). Additionally, we demonstrate how SDN-based networks can be modeled using DNI and how are they transformed later into Queueing Petri Nets (QPN) using a model-to-model transformation. In the analysis of the performance prediction accuracy, we show that automatically generated QPN models represent the performance of heterogeneous SDN hardware with maximal prediction accuracy error of 12 percent.
• 
  Quantifying the Attack Detection Accuracy of Intrusion Detection Systems in Virtualized Environments Milenkoski, Aleksandar; Jayaram, K. R.; Antunes, Nuno; Vieira, Marco; Kounev, Samuel; in Proceedings of The 27th IEEE International Symposium on Software Reliability Engineering (ISSRE 2016) (2016). IEEE Computer Society, Washington DC, USA.

  Acceptance rate (Full Paper): 45/130 = 34%
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  With the widespread adoption of virtualization, intrusion detection systems (IDSes) are increasingly being deployed in virtualized environments. When securing an environment, IT security officers are often faced with the question of how accurate deployed IDSes are at detecting attacks. To this end, metrics for assessing the attack detection accuracy of IDSes have been developed. However, these metrics are defined with respect to a fixed set of hardware resources available to the tested IDS. Therefore, IDSes deployed in virtualized environments featuring elasticity (i.e., on-demand allocation or deallocation of virtualized hardware resources during system operation) cannot be evaluated in an accurate manner using existing metrics. In this paper, we demonstrate the impact of elasticity on IDS attack detection accuracy. In addition, we propose a novel metric and measurement methodology for accurately quantifying the accuracy of IDSes deployed in virtualized environments featuring elasticity. We demonstrate their practical use through case studies involving commonly used IDSes.

  @inproceedings{MiJaAnViKo2016-ISSRE-Quantifying, abstract = {With the widespread adoption of virtualization, intrusion detection systems (IDSes) are increasingly being deployed in virtualized environments. When securing an environment, IT security officers are often faced with the question of how accurate deployed IDSes are at detecting attacks. To this end, metrics for assessing the attack detection accuracy of IDSes have been developed. However, these metrics are defined with respect to a fixed set of hardware resources available to the tested IDS. Therefore, IDSes deployed in virtualized environments featuring elasticity (i.e., on-demand allocation or deallocation of virtualized hardware resources during system operation) cannot be evaluated in an accurate manner using existing metrics. In this paper, we demonstrate the impact of elasticity on IDS attack detection accuracy. In addition, we propose a novel metric and measurement methodology for accurately quantifying the accuracy of IDSes deployed in virtualized environments featuring elasticity. We demonstrate their practical use through case studies involving commonly used IDSes.}, address = {{Washington DC, USA}}, author = {Milenkoski, Aleksandar and Jayaram, K. R. and Antunes, Nuno and Vieira, Marco and Kounev, Samuel}, booktitle = {{Proceedings of The 27th IEEE International Symposium on Software Reliability Engineering (ISSRE 2016)}}, keywords = {descartes}, month = {October}, note = {{Acceptance rate (Full Paper): 45/130 = 34\%}}, organization = {IEEE}, publisher = {IEEE Computer Society}, title = {{Quantifying the Attack Detection Accuracy of Intrusion Detection Systems in Virtualized Environments}}, year = 2016 }

  %0 Conference Paper %1 MiJaAnViKo2016-ISSRE-Quantifying %A Milenkoski, Aleksandar %A Jayaram, K. R. %A Antunes, Nuno %A Vieira, Marco %A Kounev, Samuel %B Proceedings of The 27th IEEE International Symposium on Software Reliability Engineering (ISSRE 2016) %C Washington DC, USA %D 2016 %I IEEE Computer Society %T Quantifying the Attack Detection Accuracy of Intrusion Detection Systems in Virtualized Environments %U http://ieeexplore.ieee.org/document/7774527/?reload=true %X With the widespread adoption of virtualization, intrusion detection systems (IDSes) are increasingly being deployed in virtualized environments. When securing an environment, IT security officers are often faced with the question of how accurate deployed IDSes are at detecting attacks. To this end, metrics for assessing the attack detection accuracy of IDSes have been developed. However, these metrics are defined with respect to a fixed set of hardware resources available to the tested IDS. Therefore, IDSes deployed in virtualized environments featuring elasticity (i.e., on-demand allocation or deallocation of virtualized hardware resources during system operation) cannot be evaluated in an accurate manner using existing metrics. In this paper, we demonstrate the impact of elasticity on IDS attack detection accuracy. In addition, we propose a novel metric and measurement methodology for accurately quantifying the accuracy of IDSes deployed in virtualized environments featuring elasticity. We demonstrate their practical use through case studies involving commonly used IDSes.
• 
  Performance Analysis of SDN Switches with Hardware and Software Flow Tables Rygielski, Piotr; Seliuchenko, Marian; Kounev, Samuel; Klymash, Mykhailo; in Proceedings of the 10th EAI International Conference on Performance Evaluation Methodologies and Tools (ValueTools 2016) (2016).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Nowadays data centers are increasingly becoming larger and dynamic due to virtualization. Software-Defined Networking is the leading approach to network virtualization and flexible management. The wide variety of hardware implementations have brought strong heterogeneity to the market of networking devices which are different in terms of OpenFlow features and performance. In this paper we address the issue of heterogeneity of four hardware OpenFlow switches by characterizing selected performance relevant parameters for the hardware and software flow tables. We characterize maximum size of hardware flow tables for each switch including the behavior of a rule promotion engine that moves the rules between tables. We show that in the worst case forwarding packets using software table decreases the throughput by two orders of magnitude (from 940 to 14 Mbit/s). Our results can help the developers of SDN applications to account performance limitations of hardware and software processing as well as limited hardware support for a specific rule types.

  @inproceedings{RySeKoKl2016-valuetools2016-sdn-switches-hw-sw-flow-tables, abstract = {{Nowadays data centers are increasingly becoming larger and dynamic due to virtualization. Software-Defined Networking is the leading approach to network virtualization and flexible management. The wide variety of hardware implementations have brought strong heterogeneity to the market of networking devices which are different in terms of OpenFlow features and performance. In this paper we address the issue of heterogeneity of four hardware OpenFlow switches by characterizing selected performance relevant parameters for the hardware and software flow tables. We characterize maximum size of hardware flow tables for each switch including the behavior of a rule promotion engine that moves the rules between tables. We show that in the worst case forwarding packets using software table decreases the throughput by two orders of magnitude (from 940 to 14 Mbit/s). Our results can help the developers of SDN applications to account performance limitations of hardware and software processing as well as limited hardware support for a specific rule types. }}, author = {Rygielski, Piotr and Seliuchenko, Marian and Kounev, Samuel and Klymash, Mykhailo}, booktitle = {Proceedings of the 10th EAI International Conference on Performance Evaluation Methodologies and Tools (ValueTools 2016)}, keywords = {myown}, month = {October}, title = {{Performance Analysis of SDN Switches with Hardware and Software Flow Tables}}, year = 2016 }

  %0 Conference Paper %1 RySeKoKl2016-valuetools2016-sdn-switches-hw-sw-flow-tables %A Rygielski, Piotr %A Seliuchenko, Marian %A Kounev, Samuel %A Klymash, Mykhailo %B Proceedings of the 10th EAI International Conference on Performance Evaluation Methodologies and Tools (ValueTools 2016) %D 2016 %T Performance Analysis of SDN Switches with Hardware and Software Flow Tables %X Nowadays data centers are increasingly becoming larger and dynamic due to virtualization. Software-Defined Networking is the leading approach to network virtualization and flexible management. The wide variety of hardware implementations have brought strong heterogeneity to the market of networking devices which are different in terms of OpenFlow features and performance. In this paper we address the issue of heterogeneity of four hardware OpenFlow switches by characterizing selected performance relevant parameters for the hardware and software flow tables. We characterize maximum size of hardware flow tables for each switch including the behavior of a rule promotion engine that moves the rules between tables. We show that in the worst case forwarding packets using software table decreases the throughput by two orders of magnitude (from 940 to 14 Mbit/s). Our results can help the developers of SDN applications to account performance limitations of hardware and software processing as well as limited hardware support for a specific rule types.

• 
  Flexible Performance Prediction of Data Center Networks using Automatically Generated Simulation Models Rygielski, Piotr; Kounev, Samuel; Tran-Gia, Phuoc; in Proceedings of the Eighth International Conference on Simulation Tools and Techniques (SIMUTools 2015) (2015). 119–128.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Using different modeling and simulation approaches for predicting network performance requires extensive experience and involves a number of time consuming manual steps regarding each of the modeling formalisms. In this paper, we propose a generic approach to modeling the performance of data center networks. The approach offers multiple performance models but requires to use only a single modeling language. We propose a two-step modeling methodology, in which a high-level descriptive model of the network is built in the first step, and in the second step model-to-model transformations are used to automatically transform the descriptive model to different network simulation models. We automatically generate three performance models defined at different levels of abstraction to analyze network throughput. By offering multiple simulation models in parallel, we provide flexibility in trading-off between the modeling accuracy and the simulation overhead. We analyze the simulation models by comparing the prediction accuracy with respect to the simulation duration. We observe, that in the investigated scenarios the solution duration of coarser simulation models is up to 300 times shorter, whereas the average prediction accuracy decreases only by 4 percent.

  @inproceedings{RyKoTr2015-SIMUTools-FlexPerformancePrediction, abstract = {{Using different modeling and simulation approaches for predicting network performance requires extensive experience and involves a number of time consuming manual steps regarding each of the modeling formalisms. In this paper, we propose a generic approach to modeling the performance of data center networks. The approach offers multiple performance models but requires to use only a single modeling language. We propose a two-step modeling methodology, in which a high-level descriptive model of the network is built in the first step, and in the second step model-to-model transformations are used to automatically transform the descriptive model to different network simulation models. We automatically generate three performance models defined at different levels of abstraction to analyze network throughput. By offering multiple simulation models in parallel, we provide flexibility in trading-off between the modeling accuracy and the simulation overhead. We analyze the simulation models by comparing the prediction accuracy with respect to the simulation duration. We observe, that in the investigated scenarios the solution duration of coarser simulation models is up to 300 times shorter, whereas the average prediction accuracy decreases only by 4 percent.}}, author = {Rygielski, Piotr and Kounev, Samuel and Tran-Gia, Phuoc}, booktitle = {Proceedings of the Eighth International Conference on Simulation Tools and Techniques (SIMUTools 2015)}, keywords = {myown}, month = {August}, pages = {119–128}, title = {{Flexible Performance Prediction of Data Center Networks using Automatically Generated Simulation Models}}, year = 2015 }

  %0 Conference Paper %1 RyKoTr2015-SIMUTools-FlexPerformancePrediction %A Rygielski, Piotr %A Kounev, Samuel %A Tran-Gia, Phuoc %B Proceedings of the Eighth International Conference on Simulation Tools and Techniques (SIMUTools 2015) %D 2015 %P 119--128 %T Flexible Performance Prediction of Data Center Networks using Automatically Generated Simulation Models %X Using different modeling and simulation approaches for predicting network performance requires extensive experience and involves a number of time consuming manual steps regarding each of the modeling formalisms. In this paper, we propose a generic approach to modeling the performance of data center networks. The approach offers multiple performance models but requires to use only a single modeling language. We propose a two-step modeling methodology, in which a high-level descriptive model of the network is built in the first step, and in the second step model-to-model transformations are used to automatically transform the descriptive model to different network simulation models. We automatically generate three performance models defined at different levels of abstraction to analyze network throughput. By offering multiple simulation models in parallel, we provide flexibility in trading-off between the modeling accuracy and the simulation overhead. We analyze the simulation models by comparing the prediction accuracy with respect to the simulation duration. We observe, that in the investigated scenarios the solution duration of coarser simulation models is up to 300 times shorter, whereas the average prediction accuracy decreases only by 4 percent.
• 
  Proactive Memory Scaling of Virtualized Applications Spinner, Simon; Herbst, Nikolas; Kounev, Samuel; Zhu, Xiaoyun; Lu, Lei; Uysal, Mustafa; Griffith, Rean; in Proceedings of the 2015 IEEE 8th International Conference on Cloud Computing (IEEE CLOUD 2015) (2015). 277–284. IEEE.

  Acceptance Rate: 15%
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Enterprise applications in virtualized environments are often subject to time-varying workloads with multiple seasonal patterns and trends. In order to ensure quality of service for such applications while avoiding over-provisioning, resources need to be dynamically adapted to accommodate the current workload demands. Many memory-intensive applications are not suitable for the traditional horizontal scaling approach often used for runtime performance management, as it relies on complex and expensive state replication. On the other hand, vertical scaling of memory often requires a restart of the application. In this paper, we propose a proactive approach to memory scaling for virtualized applications. It uses statistical forecasting to predict the future workload and reconfigure the memory size of the virtual machine of an application automatically. To this end, we propose an extended forecasting technique that leverages meta-knowledge, such as calendar information, to improve the forecast accuracy. In addition, we develop an application controller to adjust settings associated with application memory management during memory reconfiguration. Our evaluation using real-world traces shows that the forecast accuracy quantified with the MASE error metric can be improved by 11-59%. Furthermore, we demonstrate that the proactive approach can reduce the impact of reconfiguration on application availability by over 80% and significantly improve performance relative to a reactive controller.

  @inproceedings{SpHeKoZhLuUyGr2015-CLOUD-ProactiveMemoryScaling, abstract = {{Enterprise applications in virtualized environments are often subject to time-varying workloads with multiple seasonal patterns and trends. In order to ensure quality of service for such applications while avoiding over-provisioning, resources need to be dynamically adapted to accommodate the current workload demands. Many memory-intensive applications are not suitable for the traditional horizontal scaling approach often used for runtime performance management, as it relies on complex and expensive state replication. On the other hand, vertical scaling of memory often requires a restart of the application. In this paper, we propose a proactive approach to memory scaling for virtualized applications. It uses statistical forecasting to predict the future workload and reconfigure the memory size of the virtual machine of an application automatically. To this end, we propose an extended forecasting technique that leverages meta-knowledge, such as calendar information, to improve the forecast accuracy. In addition, we develop an application controller to adjust settings associated with application memory management during memory reconfiguration. Our evaluation using real-world traces shows that the forecast accuracy quantified with the MASE error metric can be improved by 11-59%. Furthermore, we demonstrate that the proactive approach can reduce the impact of reconfiguration on application availability by over 80% and significantly improve performance relative to a reactive controller.}}, author = {Spinner, Simon and Herbst, Nikolas and Kounev, Samuel and Zhu, Xiaoyun and Lu, Lei and Uysal, Mustafa and Griffith, Rean}, booktitle = {Proceedings of the 2015 IEEE 8th International Conference on Cloud Computing (IEEE CLOUD 2015)}, keywords = {myown}, month = {June}, note = {Acceptance Rate: 15%}, pages = {277–284}, publisher = {IEEE}, title = {{Proactive Memory Scaling of Virtualized Applications}}, year = 2015 }

  %0 Conference Paper %1 SpHeKoZhLuUyGr2015-CLOUD-ProactiveMemoryScaling %A Spinner, Simon %A Herbst, Nikolas %A Kounev, Samuel %A Zhu, Xiaoyun %A Lu, Lei %A Uysal, Mustafa %A Griffith, Rean %B Proceedings of the 2015 IEEE 8th International Conference on Cloud Computing (IEEE CLOUD 2015) %D 2015 %I IEEE %P 277--284 %T Proactive Memory Scaling of Virtualized Applications %X Enterprise applications in virtualized environments are often subject to time-varying workloads with multiple seasonal patterns and trends. In order to ensure quality of service for such applications while avoiding over-provisioning, resources need to be dynamically adapted to accommodate the current workload demands. Many memory-intensive applications are not suitable for the traditional horizontal scaling approach often used for runtime performance management, as it relies on complex and expensive state replication. On the other hand, vertical scaling of memory often requires a restart of the application. In this paper, we propose a proactive approach to memory scaling for virtualized applications. It uses statistical forecasting to predict the future workload and reconfigure the memory size of the virtual machine of an application automatically. To this end, we propose an extended forecasting technique that leverages meta-knowledge, such as calendar information, to improve the forecast accuracy. In addition, we develop an application controller to adjust settings associated with application memory management during memory reconfiguration. Our evaluation using real-world traces shows that the forecast accuracy quantified with the MASE error metric can be improved by 11-59%. Furthermore, we demonstrate that the proactive approach can reduce the impact of reconfiguration on application availability by over 80% and significantly improve performance relative to a reactive controller.
• 
  Analysis of the Influences on Server Power Consumption and Energy Efficiency for CPU-Intensive Workloads von Kistowski, Jóakim; Block, Hansfried; Beckett, John; Lange, Klaus-Dieter; Arnold, Jeremy A.; Kounev, Samuel; in Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering (ICPE 2015) (2015). ACM, New York, NY, USA.

  acceptance rate: 27%
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Energy efficiency of servers has become a significant research topic over the last years, as server energy consumption varies depending on multiple factors, such as server utilization and workload type. Server energy analysis and estimation must take all relevant factors into account to ensure reliable estimates and conclusions. Thorough system analysis requires benchmarks capable of testing different system resources at different load levels using multiple workload types. Server energy estimation approaches, on the other hand, require knowledge about the interactions of these factors for the creation of accurate power models. Common approaches to energy-aware workload classification classify workloads depending on the resource types used by the different workloads. However, they rarely take into account differences in workloads targeting the same resources. Industrial energy-efficiency benchmarks typically do not evaluate the system's energy consumption at different resource load levels, and they only provide data for system analysis at maximum system load. In this paper, we benchmark multiple server configurations using the CPU worklets included in SPEC's Server Efficiency Rating Tool (SERT). We evaluate the impact of load levels and different CPU workloads on power consumption and energy efficiency. We analyze how functions approximating the measured power consumption differ over multiple server configurations and architectures. We show that workloads targeting the same resource can differ significantly in their power draw and energy efficiency. The power consumption of a given workload type varies depending on utilization, hardware and software configuration. The power consumption of CPU-intensive workloads does not scale uniformly with increased load, nor do hardware or software configuration changes affect it in a uniform manner.

  @inproceedings{KiBlBeLaArKo2015-ICPE-SERT, abstract = {{ Energy efficiency of servers has become a significant research topic over the last years, as server energy consumption varies depending on multiple factors, such as server utilization and workload type. Server energy analysis and estimation must take all relevant factors into account to ensure reliable estimates and conclusions. Thorough system analysis requires benchmarks capable of testing different system resources at different load levels using multiple workload types. Server energy estimation approaches, on the other hand, require knowledge about the interactions of these factors for the creation of accurate power models. Common approaches to energy-aware workload classification classify workloads depending on the resource types used by the different workloads. However, they rarely take into account differences in workloads targeting the same resources. Industrial energy-efficiency benchmarks typically do not evaluate the system's energy consumption at different resource load levels, and they only provide data for system analysis at maximum system load. In this paper, we benchmark multiple server configurations using the CPU worklets included in SPEC's Server Efficiency Rating Tool (SERT). We evaluate the impact of load levels and different CPU workloads on power consumption and energy efficiency. We analyze how functions approximating the measured power consumption differ over multiple server configurations and architectures. We show that workloads targeting the same resource can differ significantly in their power draw and energy efficiency. The power consumption of a given workload type varies depending on utilization, hardware and software configuration. The power consumption of CPU-intensive workloads does not scale uniformly with increased load, nor do hardware or software configuration changes affect it in a uniform manner.}}, address = {New York, NY, USA}, author = {von Kistowski, J{\'o}akim and Block, Hansfried and Beckett, John and Lange, Klaus-Dieter and Arnold, Jeremy A. and Kounev, Samuel}, booktitle = {Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering (ICPE 2015)}, keywords = {myown}, month = {February}, note = {acceptance rate: 27\%}, publisher = {ACM}, series = {ICPE '15}, title = {{Analysis of the Influences on Server Power Consumption and Energy Efficiency for CPU-Intensive Workloads}}, year = 2015 }

  %0 Conference Paper %1 KiBlBeLaArKo2015-ICPE-SERT %A von Kistowski, Jóakim %A Block, Hansfried %A Beckett, John %A Lange, Klaus-Dieter %A Arnold, Jeremy A. %A Kounev, Samuel %B Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering (ICPE 2015) %C New York, NY, USA %D 2015 %I ACM %T Analysis of the Influences on Server Power Consumption and Energy Efficiency for CPU-Intensive Workloads %X Energy efficiency of servers has become a significant research topic over the last years, as server energy consumption varies depending on multiple factors, such as server utilization and workload type. Server energy analysis and estimation must take all relevant factors into account to ensure reliable estimates and conclusions. Thorough system analysis requires benchmarks capable of testing different system resources at different load levels using multiple workload types. Server energy estimation approaches, on the other hand, require knowledge about the interactions of these factors for the creation of accurate power models. Common approaches to energy-aware workload classification classify workloads depending on the resource types used by the different workloads. However, they rarely take into account differences in workloads targeting the same resources. Industrial energy-efficiency benchmarks typically do not evaluate the system's energy consumption at different resource load levels, and they only provide data for system analysis at maximum system load. In this paper, we benchmark multiple server configurations using the CPU worklets included in SPEC's Server Efficiency Rating Tool (SERT). We evaluate the impact of load levels and different CPU workloads on power consumption and energy efficiency. We analyze how functions approximating the measured power consumption differ over multiple server configurations and architectures. We show that workloads targeting the same resource can differ significantly in their power draw and energy efficiency. The power consumption of a given workload type varies depending on utilization, hardware and software configuration. The power consumption of CPU-intensive workloads does not scale uniformly with increased load, nor do hardware or software configuration changes affect it in a uniform manner.
• 
  Automated Workload Characterization for I/O Performance Analysis in Virtualized Environments Busch, Axel; Noorshams, Qais; Kounev, Samuel; Koziolek, Anne; Reussner, Ralf; Amrehn, Erich; in Proceedings of the ACM/SPEC International Conference on Performance Engineering (2015). ACM, New York, NY, USA.

  Acceptance Rate (Full Paper): 15/56 = 27%.
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  @inproceedings{BuNoKoKoReAm2015-ICPE-WorkloadCharacterization, address = {New York, NY, USA}, author = {Busch, Axel and Noorshams, Qais and Kounev, Samuel and Koziolek, Anne and Reussner, Ralf and Amrehn, Erich}, booktitle = {Proceedings of the ACM/SPEC International Conference on Performance Engineering}, keywords = {descartes}, month = {February}, note = {Acceptance Rate (Full Paper): 15/56 = 27\%.}, publisher = {ACM}, series = {ICPE '15}, title = {{Automated Workload Characterization for I/O Performance Analysis in Virtualized Environments}}, year = 2015 }

  %0 Conference Paper %1 BuNoKoKoReAm2015-ICPE-WorkloadCharacterization %A Busch, Axel %A Noorshams, Qais %A Kounev, Samuel %A Koziolek, Anne %A Reussner, Ralf %A Amrehn, Erich %B Proceedings of the ACM/SPEC International Conference on Performance Engineering %C New York, NY, USA %D 2015 %I ACM %T Automated Workload Characterization for I/O Performance Analysis in Virtualized Environments %U http://dx.doi.org/10.1145/2668930.2688050
• 
  BUNGEE: An Elasticity Benchmark for Self-Adaptive IaaS Cloud Environments Herbst, Nikolas Roman; Kounev, Samuel; Weber, Andreas; Groenda, Henning; in Proceedings of the 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS 2015) (2015).

  Acceptance rate: 29%
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Today's infrastructure clouds provide resource elasticity (i.e. auto-scaling) mechanisms enabling self-adaptive resource provisioning to reflect variations in the load intensity over time. These mechanisms impact on the application performance, however, their effect in specific situations is hard to quantify and compare. To evaluate the quality of elasticity mechanisms provided by different platforms and configurations, respective metrics and benchmarks are required. Existing metrics for elasticity only consider the time required to provision and deprovision resources or the costs impact of adaptations. Existing benchmarks lack the capability to handle open workloads with realistic load intensity profiles and do not explicitly distinguish between the performance exhibited by the provisioned underlying resources, on the one hand, and the quality of the elasticity mechanisms themselves, on the other hand. In this paper, we propose reliable metrics for quantifying the timing aspects and accuracy of elasticity. Based on these metrics, we propose a novel approach for benchmarking the elasticity of Infrastructure-as-a-Service (IaaS) cloud platforms independent of the performance exhibited by the provisioned underlying resources. We show that the proposed metrics provide consistent ranking of elastic platforms on an ordinal scale. Finally, we present an extensive case study of real-world complexity demonstrating that the proposed approach is applicable in realistic scenarios and can cope with different levels of resource efficiency.

  @inproceedings{HeKoWeGr_2015_SEAMS_BUNGEE, abstract = {Today's infrastructure clouds provide resource elasticity (i.e. auto-scaling) mechanisms enabling self-adaptive resource provisioning to reflect variations in the load intensity over time. These mechanisms impact on the application performance, however, their effect in specific situations is hard to quantify and compare. To evaluate the quality of elasticity mechanisms provided by different platforms and configurations, respective metrics and benchmarks are required. Existing metrics for elasticity only consider the time required to provision and deprovision resources or the costs impact of adaptations. Existing benchmarks lack the capability to handle open workloads with realistic load intensity profiles and do not explicitly distinguish between the performance exhibited by the provisioned underlying resources, on the one hand, and the quality of the elasticity mechanisms themselves, on the other hand. In this paper, we propose reliable metrics for quantifying the timing aspects and accuracy of elasticity. Based on these metrics, we propose a novel approach for benchmarking the elasticity of Infrastructure-as-a-Service (IaaS) cloud platforms independent of the performance exhibited by the provisioned underlying resources. We show that the proposed metrics provide consistent ranking of elastic platforms on an ordinal scale. Finally, we present an extensive case study of real-world complexity demonstrating that the proposed approach is applicable in realistic scenarios and can cope with different levels of resource efficiency.}, author = {Herbst, Nikolas Roman and Kounev, Samuel and Weber, Andreas and Groenda, Henning}, booktitle = {Proceedings of the 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS 2015)}, keywords = {myown}, month = {May}, note = {Acceptance rate: 29\%}, title = {{BUNGEE: An Elasticity Benchmark for Self-Adaptive IaaS Cloud Environments}}, year = 2015 }

  %0 Conference Paper %1 HeKoWeGr_2015_SEAMS_BUNGEE %A Herbst, Nikolas Roman %A Kounev, Samuel %A Weber, Andreas %A Groenda, Henning %B Proceedings of the 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS 2015) %D 2015 %T BUNGEE: An Elasticity Benchmark for Self-Adaptive IaaS Cloud Environments %X Today's infrastructure clouds provide resource elasticity (i.e. auto-scaling) mechanisms enabling self-adaptive resource provisioning to reflect variations in the load intensity over time. These mechanisms impact on the application performance, however, their effect in specific situations is hard to quantify and compare. To evaluate the quality of elasticity mechanisms provided by different platforms and configurations, respective metrics and benchmarks are required. Existing metrics for elasticity only consider the time required to provision and deprovision resources or the costs impact of adaptations. Existing benchmarks lack the capability to handle open workloads with realistic load intensity profiles and do not explicitly distinguish between the performance exhibited by the provisioned underlying resources, on the one hand, and the quality of the elasticity mechanisms themselves, on the other hand. In this paper, we propose reliable metrics for quantifying the timing aspects and accuracy of elasticity. Based on these metrics, we propose a novel approach for benchmarking the elasticity of Infrastructure-as-a-Service (IaaS) cloud platforms independent of the performance exhibited by the provisioned underlying resources. We show that the proposed metrics provide consistent ranking of elastic platforms on an ordinal scale. Finally, we present an extensive case study of real-world complexity demonstrating that the proposed approach is applicable in realistic scenarios and can cope with different levels of resource efficiency.
• 
  Modeling and Extracting Load Intensity Profiles von Kistowski, Jóakim; Herbst, Nikolas Roman; Zoller, Daniel; Kounev, Samuel; Hotho, Andreas; in Proceedings of the 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS 2015) (2015).

  Acceptance rate: 29%
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Today's system developers and operators face the challenge of creating software systems that make efficient use of dynamically allocated resources under highly variable and dynamic load profiles, while at the same time delivering reliable performance. Benchmarking of systems under these constraints is difficult, as state-of-the-art benchmarking frameworks provide only limited support for emulating such dynamic and highly variable load profiles for the creation of realistic workload scenarios. Industrial benchmarks typically confine themselves to workloads with constant or stepwise increasing loads. Alternatively, they support replaying of recorded load traces. Statistical load intensity descriptions also do not sufficiently capture concrete pattern load profile variations over time. To address these issues, we present the Descartes Load Intensity Model (DLIM). DLIM provides a modeling formalism for describing load intensity variations over time. A DLIM instance can be used as a compact representation of a recorded load intensity trace, providing a powerful tool for benchmarking and performance analysis. As manually obtaining DLIM instances can be time consuming, we present three different automated extraction methods, which also help to enable autonomous system analysis for self-adaptive systems. Model expressiveness is validated using the presented extraction methods. Extracted DLIM instances exhibit a median modeling error of 12.4% on average over nine different real-world traces covering between two weeks and seven months. Additionally, extraction methods perform orders of magnitude faster than existing time series decomposition approaches.

  @inproceedings{KiHeZoKoHo2015_SEAMS_DLIMExtraction, abstract = {Today's system developers and operators face the challenge of creating software systems that make efficient use of dynamically allocated resources under highly variable and dynamic load profiles, while at the same time delivering reliable performance. Benchmarking of systems under these constraints is difficult, as state-of-the-art benchmarking frameworks provide only limited support for emulating such dynamic and highly variable load profiles for the creation of realistic workload scenarios. Industrial benchmarks typically confine themselves to workloads with constant or stepwise increasing loads. Alternatively, they support replaying of recorded load traces. Statistical load intensity descriptions also do not sufficiently capture concrete pattern load profile variations over time. To address these issues, we present the Descartes Load Intensity Model (DLIM). DLIM provides a modeling formalism for describing load intensity variations over time. A DLIM instance can be used as a compact representation of a recorded load intensity trace, providing a powerful tool for benchmarking and performance analysis. As manually obtaining DLIM instances can be time consuming, we present three different automated extraction methods, which also help to enable autonomous system analysis for self-adaptive systems. Model expressiveness is validated using the presented extraction methods. Extracted DLIM instances exhibit a median modeling error of 12.4% on average over nine different real-world traces covering between two weeks and seven months. Additionally, extraction methods perform orders of magnitude faster than existing time series decomposition approaches.}, author = {von Kistowski, J{\'o}akim and Herbst, Nikolas Roman and Zoller, Daniel and Kounev, Samuel and Hotho, Andreas}, booktitle = {Proceedings of the 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS 2015)}, keywords = {myown}, month = {may}, note = {Acceptance rate: 29\%}, title = {{Modeling and Extracting Load Intensity Profiles}}, year = 2015 }

  %0 Conference Paper %1 KiHeZoKoHo2015_SEAMS_DLIMExtraction %A von Kistowski, Jóakim %A Herbst, Nikolas Roman %A Zoller, Daniel %A Kounev, Samuel %A Hotho, Andreas %B Proceedings of the 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS 2015) %D 2015 %T Modeling and Extracting Load Intensity Profiles %X Today's system developers and operators face the challenge of creating software systems that make efficient use of dynamically allocated resources under highly variable and dynamic load profiles, while at the same time delivering reliable performance. Benchmarking of systems under these constraints is difficult, as state-of-the-art benchmarking frameworks provide only limited support for emulating such dynamic and highly variable load profiles for the creation of realistic workload scenarios. Industrial benchmarks typically confine themselves to workloads with constant or stepwise increasing loads. Alternatively, they support replaying of recorded load traces. Statistical load intensity descriptions also do not sufficiently capture concrete pattern load profile variations over time. To address these issues, we present the Descartes Load Intensity Model (DLIM). DLIM provides a modeling formalism for describing load intensity variations over time. A DLIM instance can be used as a compact representation of a recorded load intensity trace, providing a powerful tool for benchmarking and performance analysis. As manually obtaining DLIM instances can be time consuming, we present three different automated extraction methods, which also help to enable autonomous system analysis for self-adaptive systems. Model expressiveness is validated using the presented extraction methods. Extracted DLIM instances exhibit a median modeling error of 12.4% on average over nine different real-world traces covering between two weeks and seven months. Additionally, extraction methods perform orders of magnitude faster than existing time series decomposition approaches.
• 
  Parallel Simulation of Queueing Petri Nets Walter, Jürgen; Spinner, Simon; Kounev, Samuel; in Proceedings of the Eighth EAI International Conference on Simulation Tools and Techniques (SIMUTools 2015) (2015).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Queueing Petri nets show high accurracy to model and analyze cloud scenarios. To apply these models in online scenarios a speedup of simulation is desirable. One way to speedup simulation is parallelization. Decades of intensive research showed no general solution for parallel discrete event simulation, which provides reasonable speedups. We research the application of parallel simulation for cloud scenarios. We develop a parallel simulation engine that shows speedup for certain scenarios common in cloud models. We present a parallel simualtion engine that employs application level and event level parallelism. We parallelize the execution of a single run by the use of a conservative barrier-based synchronization algorithm. The speedup for a single run depends on the capability of the model. Hence, a number of experiments on different net characteristics were conducted showing that for certain models a superlinear speedup is possible.

  @inproceedings{WaSpKo-SIMUTools-QPNParallelSimulation, abstract = {Queueing Petri nets show high accurracy to model and analyze cloud scenarios. To apply these models in online scenarios a speedup of simulation is desirable. One way to speedup simulation is parallelization. Decades of intensive research showed no general solution for parallel discrete event simulation, which provides reasonable speedups. We research the application of parallel simulation for cloud scenarios. We develop a parallel simulation engine that shows speedup for certain scenarios common in cloud models. We present a parallel simualtion engine that employs application level and event level parallelism. We parallelize the execution of a single run by the use of a conservative barrier-based synchronization algorithm. The speedup for a single run depends on the capability of the model. Hence, a number of experiments on different net characteristics were conducted showing that for certain models a superlinear speedup is possible.}, author = {Walter, J{\"u}rgen and Spinner, Simon and Kounev, Samuel}, booktitle = {Proceedings of the Eighth EAI International Conference on Simulation Tools and Techniques (SIMUTools 2015)}, keywords = {myown}, month = {August}, title = {{Parallel Simulation of Queueing Petri Nets}}, year = 2015 }

  %0 Conference Paper %1 WaSpKo-SIMUTools-QPNParallelSimulation %A Walter, Jürgen %A Spinner, Simon %A Kounev, Samuel %B Proceedings of the Eighth EAI International Conference on Simulation Tools and Techniques (SIMUTools 2015) %D 2015 %T Parallel Simulation of Queueing Petri Nets %X Queueing Petri nets show high accurracy to model and analyze cloud scenarios. To apply these models in online scenarios a speedup of simulation is desirable. One way to speedup simulation is parallelization. Decades of intensive research showed no general solution for parallel discrete event simulation, which provides reasonable speedups. We research the application of parallel simulation for cloud scenarios. We develop a parallel simulation engine that shows speedup for certain scenarios common in cloud models. We present a parallel simualtion engine that employs application level and event level parallelism. We parallelize the execution of a single run by the use of a conservative barrier-based synchronization algorithm. The speedup for a single run depends on the capability of the model. Hence, a number of experiments on different net characteristics were conducted showing that for certain models a superlinear speedup is possible.
• 
  Specialized Heuristics for the Controller Placement Problem in Large Scale SDN Networks Lange, Stanislav; Gebert, Steffen; Spoerhase, Joachim; Rygielski, Piotr; Zinner, Thomas; Kounev, Samuel; Tran-Gia, Phuoc; in 27th International Teletraffic Congress (ITC 2015}) (2015). 210–218. Ghent, Belgium.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  The Software Defined Networking~(SDN) concept introduces a paradigm shift in the networking world towards an externalized control plane which is logically centralized. When designing an SDN-based WAN architecture, it is of vital importance to find a feasible solution to the controller placement problem, i.e., to decide where to position a limited amount of resources within the network. In addition to time-independent constraints regarding aspects like scalability, resilience, and control plane communication delays, dynamically changing network conditions like traffic patterns or bandwidth demands need to be considered as well. Consequently, such dynamic environments call for a regular and fast recalculation of placements in order to adapt to the current situation in a timely manner. While an exhaustive evaluation of all possible solutions can be performed within a practically feasible time frame for small and medium-sized networks, such an approach is out of scope for large problem instances which have significantly higher time and memory requirements. Therefore, this work investigates a specialized heuristic, which takes into account a particular set of optimization objectives and returns solutions representing the possible trade-offs between them. Due to its low computation time and acceptable margin of error, this heuristic can be employed by automatic decision systems operating in dynamic environments.

  @inproceedings{LaGeSpRyZiKoTG-SDN-Controller-Placement-ITC2015, abstract = {{The Software Defined Networking (SDN) concept introduces a paradigm shift in the networking world towards an externalized control plane which is logically centralized. When designing an SDN-based WAN architecture, it is of vital importance to find a feasible solution to the controller placement problem, i.e., to decide where to position a limited amount of resources within the network. In addition to time-independent constraints regarding aspects like scalability, resilience, and control plane communication delays, dynamically changing network conditions like traffic patterns or bandwidth demands need to be considered as well. Consequently, such dynamic environments call for a regular and fast recalculation of placements in order to adapt to the current situation in a timely manner. While an exhaustive evaluation of all possible solutions can be performed within a practically feasible time frame for small and medium-sized networks, such an approach is out of scope for large problem instances which have significantly higher time and memory requirements. Therefore, this work investigates a specialized heuristic, which takes into account a particular set of optimization objectives and returns solutions representing the possible trade-offs between them. Due to its low computation time and acceptable margin of error, this heuristic can be employed by automatic decision systems operating in dynamic environments.}}, address = {Ghent, Belgium}, author = {Lange, Stanislav and Gebert, Steffen and Spoerhase, Joachim and Rygielski, Piotr and Zinner, Thomas and Kounev, Samuel and Tran-Gia, Phuoc}, booktitle = {27th International Teletraffic Congress ({ITC 2015})}, keywords = {myown}, month = {September}, pages = {210–218}, title = {{Specialized Heuristics for the Controller Placement Problem in Large Scale SDN Networks}}, year = 2015 }

  %0 Conference Paper %1 LaGeSpRyZiKoTG-SDN-Controller-Placement-ITC2015 %A Lange, Stanislav %A Gebert, Steffen %A Spoerhase, Joachim %A Rygielski, Piotr %A Zinner, Thomas %A Kounev, Samuel %A Tran-Gia, Phuoc %B 27th International Teletraffic Congress (ITC 2015}) %C Ghent, Belgium %D 2015 %P 210--218 %T Specialized Heuristics for the Controller Placement Problem in Large Scale SDN Networks %U http://dx.doi.org/10.1109/ITC.2015.32 %X The Software Defined Networking~(SDN) concept introduces a paradigm shift in the networking world towards an externalized control plane which is logically centralized. When designing an SDN-based WAN architecture, it is of vital importance to find a feasible solution to the controller placement problem, i.e., to decide where to position a limited amount of resources within the network. In addition to time-independent constraints regarding aspects like scalability, resilience, and control plane communication delays, dynamically changing network conditions like traffic patterns or bandwidth demands need to be considered as well. Consequently, such dynamic environments call for a regular and fast recalculation of placements in order to adapt to the current situation in a timely manner. While an exhaustive evaluation of all possible solutions can be performed within a practically feasible time frame for small and medium-sized networks, such an approach is out of scope for large problem instances which have significantly higher time and memory requirements. Therefore, this work investigates a specialized heuristic, which takes into account a particular set of optimization objectives and returns solutions representing the possible trade-offs between them. Due to its low computation time and acceptable margin of error, this heuristic can be employed by automatic decision systems operating in dynamic environments.
• 
  Energy Efficiency of Hierarchical Server Load Distribution Strategies von Kistowski, Jóakim; Beckett, John; Lange, Klaus-Dieter; Block, Hansfried; Arnold, Jeremy A.; Kounev, Samuel; in Proceedings of the IEEE 23nd International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS 2015) (2015). IEEE.

  Full paper acceptance rate: 19%
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Energy efficiency of servers has become a significant issue over the last years. Load distribution plays a crucial role in the improvement of energy efficiency as (un-)balancing strategies can be leveraged to distribute load over one or multiple systems in a way in which resources are utilized at high performance, yet low overall power consumption. This can be achieved on multiple levels, from load distribution on single CPU cores to machine level load balancing on distributed systems. With modern day server architectures providing load balancing opportunities at several layers, answering the question of optimal load distribution has become non-trivial. Work has to be distributed hierarchically in a fashion that enables maximum energy efficiency at each level. Current approaches balance load based on generalized assumptions about the energy efficiency of servers. These assumptions are based either on very machine-specific or highly generalized observations that may or may not hold true over a variety of systems and configurations. In this paper, we use a modified version of the SPEC SERT suite to measure the energy efficiency of a variety of hierarchical load distribution strategies on single and multi-node systems. We introduce a new strategy and evaluate energy efficiency for homogeneous and heterogeneous workloads over different hardware configurations. Our results show that the selection of a load distribution strategy depends heavily on workload, system utilization, as well as hardware. Used in conjunction with existing strategies, our new load distribution strategy can reduce a single system's power consumption by up to 10.7%.

  @inproceedings{KiBeLaBlArKo2015-MASCOTS, abstract = {{Energy efficiency of servers has become a significant issue over the last years. Load distribution plays a crucial role in the improvement of energy efficiency as (un-)balancing strategies can be leveraged to distribute load over one or multiple systems in a way in which resources are utilized at high performance, yet low overall power consumption. This can be achieved on multiple levels, from load distribution on single CPU cores to machine level load balancing on distributed systems. With modern day server architectures providing load balancing opportunities at several layers, answering the question of optimal load distribution has become non-trivial. Work has to be distributed hierarchically in a fashion that enables maximum energy efficiency at each level. Current approaches balance load based on generalized assumptions about the energy efficiency of servers. These assumptions are based either on very machine-specific or highly generalized observations that may or may not hold true over a variety of systems and configurations. In this paper, we use a modified version of the SPEC SERT suite to measure the energy efficiency of a variety of hierarchical load distribution strategies on single and multi-node systems. We introduce a new strategy and evaluate energy efficiency for homogeneous and heterogeneous workloads over different hardware configurations. Our results show that the selection of a load distribution strategy depends heavily on workload, system utilization, as well as hardware. Used in conjunction with existing strategies, our new load distribution strategy can reduce a single system's power consumption by up to 10.7%.}}, author = {von Kistowski, J{\'o}akim and Beckett, John and Lange, Klaus-Dieter and Block, Hansfried and Arnold, Jeremy A. and Kounev, Samuel}, booktitle = {Proceedings of the IEEE 23nd International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS 2015)}, keywords = {myown}, month = {October}, note = {Full paper acceptance rate: 19\%}, publisher = {IEEE}, title = {{Energy Efficiency of Hierarchical Server Load Distribution Strategies}}, year = 2015 }

  %0 Conference Paper %1 KiBeLaBlArKo2015-MASCOTS %A von Kistowski, Jóakim %A Beckett, John %A Lange, Klaus-Dieter %A Block, Hansfried %A Arnold, Jeremy A. %A Kounev, Samuel %B Proceedings of the IEEE 23nd International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS 2015) %D 2015 %I IEEE %T Energy Efficiency of Hierarchical Server Load Distribution Strategies %X Energy efficiency of servers has become a significant issue over the last years. Load distribution plays a crucial role in the improvement of energy efficiency as (un-)balancing strategies can be leveraged to distribute load over one or multiple systems in a way in which resources are utilized at high performance, yet low overall power consumption. This can be achieved on multiple levels, from load distribution on single CPU cores to machine level load balancing on distributed systems. With modern day server architectures providing load balancing opportunities at several layers, answering the question of optimal load distribution has become non-trivial. Work has to be distributed hierarchically in a fashion that enables maximum energy efficiency at each level. Current approaches balance load based on generalized assumptions about the energy efficiency of servers. These assumptions are based either on very machine-specific or highly generalized observations that may or may not hold true over a variety of systems and configurations. In this paper, we use a modified version of the SPEC SERT suite to measure the energy efficiency of a variety of hierarchical load distribution strategies on single and multi-node systems. We introduce a new strategy and evaluate energy efficiency for homogeneous and heterogeneous workloads over different hardware configurations. Our results show that the selection of a load distribution strategy depends heavily on workload, system utilization, as well as hardware. Used in conjunction with existing strategies, our new load distribution strategy can reduce a single system's power consumption by up to 10.7%.
• 
  Model-based Autonomic and Performance-aware System Adaptation in Heterogeneous Resource Environments: A Case Study Huber, Nikolaus; Walter, Jürgen; Bähr, Manuel; Kounev, Samuel; in Proceedings of the 2015 IEEE International Conference on Cloud and Autonomic Computing (ICCAC) (2015). IEEE.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Recent trends like cloud computing show that service providers increasingly adopt to modern self-adaptive system architectures promising higher resource efficiency and lower operating costs. Blue Yonder is such a service provider in the field of predictive analytics, looking for novel approaches to automatically adapt their service infrastructure to changing customer workloads and service-level agreements. In this paper, we present a realistic case study in which we apply a holistic modelbased approach to build a self-adaptive system that automatically maintains performance requirements and resource efficiency in the heterogeneous resource environment of Blue Yonder. As modeling formalism, we employ the Descartes Modeling Language (DML), a domain-specific language for modeling the performance behavior and run-time adaptation processes of modern dynamic IT systems. This case study is an exemplar for applying a holistic model-based approach to engineering performance-aware system adaptation. More specifically, the results evaluate the applicability and effectiveness of our approach and demonstrate that DML provides suitable modeling abstractions that can be used as a basis for self-adaptive performance and resource management.

  @inproceedings{HuWaBaKo2015-ICCAC-BlueYonder, abstract = {Recent trends like cloud computing show that service providers increasingly adopt to modern self-adaptive system architectures promising higher resource efficiency and lower operating costs. Blue Yonder is such a service provider in the field of predictive analytics, looking for novel approaches to automatically adapt their service infrastructure to changing customer workloads and service-level agreements. In this paper, we present a realistic case study in which we apply a holistic modelbased approach to build a self-adaptive system that automatically maintains performance requirements and resource efficiency in the heterogeneous resource environment of Blue Yonder. As modeling formalism, we employ the Descartes Modeling Language (DML), a domain-specific language for modeling the performance behavior and run-time adaptation processes of modern dynamic IT systems. This case study is an exemplar for applying a holistic model-based approach to engineering performance-aware system adaptation. More specifically, the results evaluate the applicability and effectiveness of our approach and demonstrate that DML provides suitable modeling abstractions that can be used as a basis for self-adaptive performance and resource management.}, author = {Huber, Nikolaus and Walter, J{\"u}rgen and B{\"a}hr, Manuel and Kounev, Samuel}, booktitle = {Proceedings of the 2015 IEEE International Conference on Cloud and Autonomic Computing (ICCAC)}, keywords = {myown}, month = {September}, publisher = {IEEE}, title = {{Model-based Autonomic and Performance-aware System Adaptation in Heterogeneous Resource Environments: A Case Study}}, year = 2015 }

  %0 Conference Paper %1 HuWaBaKo2015-ICCAC-BlueYonder %A Huber, Nikolaus %A Walter, Jürgen %A Bähr, Manuel %A Kounev, Samuel %B Proceedings of the 2015 IEEE International Conference on Cloud and Autonomic Computing (ICCAC) %D 2015 %I IEEE %T Model-based Autonomic and Performance-aware System Adaptation in Heterogeneous Resource Environments: A Case Study %X Recent trends like cloud computing show that service providers increasingly adopt to modern self-adaptive system architectures promising higher resource efficiency and lower operating costs. Blue Yonder is such a service provider in the field of predictive analytics, looking for novel approaches to automatically adapt their service infrastructure to changing customer workloads and service-level agreements. In this paper, we present a realistic case study in which we apply a holistic modelbased approach to build a self-adaptive system that automatically maintains performance requirements and resource efficiency in the heterogeneous resource environment of Blue Yonder. As modeling formalism, we employ the Descartes Modeling Language (DML), a domain-specific language for modeling the performance behavior and run-time adaptation processes of modern dynamic IT systems. This case study is an exemplar for applying a holistic model-based approach to engineering performance-aware system adaptation. More specifically, the results evaluate the applicability and effectiveness of our approach and demonstrate that DML provides suitable modeling abstractions that can be used as a basis for self-adaptive performance and resource management.
• 
  Evaluation of Intrusion Detection Systems in Virtualized Environments Using Attack Injection Milenkoski, Aleksandar; Payne, Bryan D.; Antunes, Nuno; Vieira, Marco; Kounev, Samuel; Avritzer, Alberto; Luft, Matthias; in The 18th International Symposium on Research in Attacks, Intrusions, and Defenses (RAID 2015) (2015). Springer.

  Acceptance Rate: 23%.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  The evaluation of intrusion detection systems (IDSes) is an active research area with many open challenges, one of which is the generation of representative workloads that contain attacks. In this paper, we propose a novel approach for the rigorous evaluation of IDSes in virtualized environments, with a focus on IDSes designed to detect attacks leveraging or targeting the hypervisor via its hypercall interface. We present hInjector, a tool for generating IDS evaluation workloads by injecting such attacks during regular operation of a virtualized environment. We demonstrate the application of our approach and show its practical usefulness by evaluating a representative IDS designed to operate in virtualized environments. The virtualized environment of the industry-standard benchmark SPECvirt_sc2013 is used as a testbed, whose drivers generate workloads representative of workloads seen in production environments. This work enables for the first time the injection of attacks in virtualized environments for the purpose of generating representative IDS evaluation workloads.

  @inproceedings{MiPaAnViKoAvLu2015-RAID-Challenges, abstract = {{The evaluation of intrusion detection systems (IDSes) is an active research area with many open challenges, one of which is the generation of representative workloads that contain attacks. In this paper, we propose a novel approach for the rigorous evaluation of IDSes in virtualized environments, with a focus on IDSes designed to detect attacks leveraging or targeting the hypervisor via its hypercall interface. We present hInjector, a tool for generating IDS evaluation workloads by injecting such attacks during regular operation of a virtualized environment. We demonstrate the application of our approach and show its practical usefulness by evaluating a representative IDS designed to operate in virtualized environments. The virtualized environment of the industry-standard benchmark SPECvirt_sc2013 is used as a testbed, whose drivers generate workloads representative of workloads seen in production environments. This work enables for the first time the injection of attacks in virtualized environments for the purpose of generating representative IDS evaluation workloads.}}, author = {Milenkoski, Aleksandar and Payne, Bryan D. and Antunes, Nuno and Vieira, Marco and Kounev, Samuel and Avritzer, Alberto and Luft, Matthias}, booktitle = {The 18th International Symposium on Research in Attacks, Intrusions, and Defenses (RAID 2015)}, keywords = {myown}, month = {{November}}, note = {{Acceptance Rate: 23%.}}, publisher = {{Springer}}, title = {{Evaluation of Intrusion Detection Systems in Virtualized Environments Using Attack Injection}}, year = 2015 }

  %0 Conference Paper %1 MiPaAnViKoAvLu2015-RAID-Challenges %A Milenkoski, Aleksandar %A Payne, Bryan D. %A Antunes, Nuno %A Vieira, Marco %A Kounev, Samuel %A Avritzer, Alberto %A Luft, Matthias %B The 18th International Symposium on Research in Attacks, Intrusions, and Defenses (RAID 2015) %D 2015 %I Springer %T Evaluation of Intrusion Detection Systems in Virtualized Environments Using Attack Injection %U http://link.springer.com/chapter/10.1007/978-3-319-26362-5_22 %X The evaluation of intrusion detection systems (IDSes) is an active research area with many open challenges, one of which is the generation of representative workloads that contain attacks. In this paper, we propose a novel approach for the rigorous evaluation of IDSes in virtualized environments, with a focus on IDSes designed to detect attacks leveraging or targeting the hypervisor via its hypercall interface. We present hInjector, a tool for generating IDS evaluation workloads by injecting such attacks during regular operation of a virtualized environment. We demonstrate the application of our approach and show its practical usefulness by evaluating a representative IDS designed to operate in virtualized environments. The virtualized environment of the industry-standard benchmark SPECvirt_sc2013 is used as a testbed, whose drivers generate workloads representative of workloads seen in production environments. This work enables for the first time the injection of attacks in virtualized environments for the purpose of generating representative IDS evaluation workloads.

• 
  Platform-as-a-Service Architecture for Performance Isolated Multi-Tenant Applications Krebs, Rouven; Loesch, Manuel; Kounev, Samuel; in Proceedings of the 7th IEEE International Conference on Cloud Computing (IEEE Cloud 2014) (2014). IEEE.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]