Publications

• [ 2020 ]
• [ 2017 ]
• [ 2015 ]
• [ 2013 ]
• [ 2011 ]
• [ 2008 ]
• [ 2005 ]

• 
  {Systems Benchmarking} Kounev, Samuel; Lange, Klaus-Dieter; von Kistowski, Jóakim; (1st ed.) Springer International Publishing.
  • [ BibTeX ]
  • [ EndNote ]
  @book{KoLaKi-2020-SystemsBenchmarking, author = {Kounev, Samuel and Lange, Klaus-Dieter and von Kistowski, Jóakim}, edition = 1, publisher = {Springer International Publishing}, title = {{Systems Benchmarking}}, year = 2020 }

  %0 Book %1 KoLaKi-2020-SystemsBenchmarking %A Kounev, Samuel %A Lange, Klaus-Dieter %A von Kistowski, Jóakim %D 2020 %I Springer International Publishing %T {Systems Benchmarking} %7 1

• 
  {Self-Aware Computing Systems} Kounev, Samuel; Kephart, Jeffrey O.; Milenkoski, Aleksandar; Zhu, Xiaoyun;{Springer Verlag}, {Berlin Heidelberg, Germany}.
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  @book{KoKeMiZh2017-self, address = {{Berlin Heidelberg, Germany}}, author = {Kounev, Samuel and Kephart, Jeffrey O. and Milenkoski, Aleksandar and Zhu, Xiaoyun}, publisher = {{Springer Verlag}}, title = {{Self-Aware Computing Systems}}, year = 2017 }

  %0 Book %1 KoKeMiZh2017-self %A Kounev, Samuel %A Kephart, Jeffrey O. %A Milenkoski, Aleksandar %A Zhu, Xiaoyun %C {Berlin Heidelberg, Germany} %D 2017 %I {Springer Verlag} %T {Self-Aware Computing Systems} %U http://www.springer.com/de/book/9783319474724

• 
  Proceeding of IEEE International Conference on Autonomic Computing (ICAC2015), Grenoble, France, July 8--10, 2015 Kounev, Samuel; (P. Lalanda; A. Diaconescu; L. Cherkasova; S. Kounev, eds.)IEEE Computer Society Conference Publishing Services (CPS), Los Alamitos, California.
  • [ BibTeX ]
  • [ EndNote ]
  @proceedings{proceedings-icac-2015, address = {Los Alamitos, California}, author = {Kounev, Samuel}, editor = {Lalanda, Philippe and Diaconescu, Ada and Cherkasova, Lucy and Kounev, Samuel}, month = {July}, publisher = {IEEE Computer Society Conference Publishing Services (CPS)}, title = {Proceeding of IEEE International Conference on Autonomic Computing (ICAC2015), Grenoble, France, July 8--10, 2015}, year = 2015 }

  %0 Conference Proceedings %1 proceedings-icac-2015 %A Kounev, Samuel %C Los Alamitos, California %D 2015 %E Lalanda, Philippe %E Diaconescu, Ada %E Cherkasova, Lucy %E Kounev, Samuel %I IEEE Computer Society Conference Publishing Services (CPS) %T Proceeding of IEEE International Conference on Autonomic Computing (ICAC2015), Grenoble, France, July 8--10, 2015
• 
  {Model-driven Algorithms and Architectures for Self-Aware Computing Systems} Kounev, Samuel; in Dagstuhl Reports, (S. Kounev; X. Zhu; J. O. Kephart; M. Kwiatkowska; A. M. [Assistant Editor], eds.)Dagstuhl, Germany.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  {This report documents the program and the outcomes of Dagstuhl Seminar 15041 "Model-driven Algorithms and Architectures for Self-Aware Computing Systems". The design of self-aware computing systems calls for an integrated interdisciplinary approach building on results from multiple areas of computer science and engineering, including software and systems engineering, systems modeling, simulation and analysis, autonomic and organic computing, machine learning and artificial intelligence, data center resource management, and so on. The Dagstuhl Seminar 15041 served as a platform to raise the awareness about the relevant research efforts in the respective research communities as well as existing synergies that can be exploited to advance the state-of-the-art, formulate a new research agenda that takes a broader view on the problem following an integrated and interdisciplinary approach, and establish collaborations between academia and industry.}

  @proceedings{KoZhKeKw2015, abstract = {{This report documents the program and the outcomes of Dagstuhl Seminar 15041 "Model-driven Algorithms and Architectures for Self-Aware Computing Systems". The design of self-aware computing systems calls for an integrated interdisciplinary approach building on results from multiple areas of computer science and engineering, including software and systems engineering, systems modeling, simulation and analysis, autonomic and organic computing, machine learning and artificial intelligence, data center resource management, and so on. The Dagstuhl Seminar 15041 served as a platform to raise the awareness about the relevant research efforts in the respective research communities as well as existing synergies that can be exploited to advance the state-of-the-art, formulate a new research agenda that takes a broader view on the problem following an integrated and interdisciplinary approach, and establish collaborations between academia and industry.}}, address = {Dagstuhl, Germany}, author = {Kounev, Samuel}, editor = {Kounev, Samuel and Zhu, Xiaoyun and Kephart, Jeffrey O. and Kwiatkowska, Marta and Editor], Aleksandar Milenkoski [Assistant}, month = {January}, series = {Dagstuhl Reports}, title = {{Model-driven Algorithms and Architectures for Self-Aware Computing Systems}}, year = 2015 }

  %0 Conference Proceedings %1 KoZhKeKw2015 %A Kounev, Samuel %B Dagstuhl Reports %C Dagstuhl, Germany %D 2015 %E Kounev, Samuel %E Zhu, Xiaoyun %E Kephart, Jeffrey O. %E Kwiatkowska, Marta %E Editor], Aleksandar Milenkoski [Assistant %T {Model-driven Algorithms and Architectures for Self-Aware Computing Systems} %U http://drops.dagstuhl.de/opus/volltexte/2015/5038/ %X {This report documents the program and the outcomes of Dagstuhl Seminar 15041 "Model-driven Algorithms and Architectures for Self-Aware Computing Systems". The design of self-aware computing systems calls for an integrated interdisciplinary approach building on results from multiple areas of computer science and engineering, including software and systems engineering, systems modeling, simulation and analysis, autonomic and organic computing, machine learning and artificial intelligence, data center resource management, and so on. The Dagstuhl Seminar 15041 served as a platform to raise the awareness about the relevant research efforts in the respective research communities as well as existing synergies that can be exploited to advance the state-of-the-art, formulate a new research agenda that takes a broader view on the problem following an integrated and interdisciplinary approach, and establish collaborations between academia and industry.}

• 
  Proceedings of the 2013 International Workshop on Hot Topics in Cloud Services (HotTopiCS 2013) Kounev, Samuel; (S. Kounev; S. Zschaler; K. Sachs, eds.)ACM.
  • [ BibTeX ]
  • [ EndNote ]
  @proceedings{KoZsSa2013-HOTTOPICS-Proceedings, author = {Kounev, Samuel}, editor = {Kounev, Samuel and Zschaler, Steffen and Sachs, Kai}, month = {April}, publisher = {ACM}, title = {Proceedings of the 2013 International Workshop on Hot Topics in Cloud Services (HotTopiCS 2013)}, year = 2013 }

  %0 Conference Proceedings %1 KoZsSa2013-HOTTOPICS-Proceedings %A Kounev, Samuel %D 2013 %E Kounev, Samuel %E Zschaler, Steffen %E Sachs, Kai %I ACM %T Proceedings of the 2013 International Workshop on Hot Topics in Cloud Services (HotTopiCS 2013)

• 
  ICPE’11 - 2nd Joint ACM/SPEC International Conference on Performance Engineering, Karlsruhe, Germany, March 14--16, 2011 Kounev, Samuel; (S. Kounev; V. Cortellessa; R. Mirandola; D. J. Lilja, eds.)ACM, New York, NY, USA.
  • [ BibTeX ]
  • [ EndNote ]
  @proceedings{DBLP-conf-wosp-2011, address = {New York, NY, USA}, author = {Kounev, Samuel}, editor = {Kounev, Samuel and Cortellessa, Vittorio and Mirandola, Raffaela and Lilja, David J.}, month = {March}, publisher = {ACM}, title = {ICPE'11 - 2nd Joint ACM/SPEC International Conference on Performance Engineering, Karlsruhe, Germany, March 14--16, 2011}, year = 2011 }

  %0 Conference Proceedings %1 DBLP-conf-wosp-2011 %A Kounev, Samuel %C New York, NY, USA %D 2011 %E Kounev, Samuel %E Cortellessa, Vittorio %E Mirandola, Raffaela %E Lilja, David J. %I ACM %T ICPE'11 - 2nd Joint ACM/SPEC International Conference on Performance Engineering, Karlsruhe, Germany, March 14--16, 2011

• 
  {Performance Evaluation: Metrics, Models and Benchmarks, Proceedings of the 2008 SPEC International Performance Evaluation Workshop (SIPEW 2008), Darmstadt, Germany, June 27-28} Kounev, Samuel; in Lecture Notes in Computer Science (LNCS), (S. Kounev; I. Gorton; K. Sachs, eds.) (Vol. 5119) Springer, Heidelberg, Germany.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  This book constitutes the refereed proceedings of the SPEC International Performance Evaluation Workshop, SIPEW 2008, held in Darmstadt, Germany, in June 2008. The 17 revised full papers presented together with 3 keynote talks were carefully reviewed and selected out of 39 submissions for inclusion in the book. The papers are organized in topical sections on models for software performance engineering; benchmarks and workload characterization; Web services and service-oriented architectures; power and performance; and profiling, monitoring and optimization.

  @proceedings{kounev2008c, abstract = {This book constitutes the refereed proceedings of the SPEC International Performance Evaluation Workshop, SIPEW 2008, held in Darmstadt, Germany, in June 2008. The 17 revised full papers presented together with 3 keynote talks were carefully reviewed and selected out of 39 submissions for inclusion in the book. The papers are organized in topical sections on models for software performance engineering; benchmarks and workload characterization; Web services and service-oriented architectures; power and performance; and profiling, monitoring and optimization.}, address = {Heidelberg, Germany}, author = {Kounev, Samuel}, editor = {Kounev, Samuel and Gorton, Ian and Sachs, Kai}, month = {June}, publisher = {Springer}, series = {Lecture Notes in Computer Science (LNCS)}, title = {{Performance Evaluation: Metrics, Models and Benchmarks, Proceedings of the 2008 SPEC International Performance Evaluation Workshop (SIPEW 2008), Darmstadt, Germany, June 27-28}}, volume = 5119, year = 2008 }

  %0 Conference Proceedings %1 kounev2008c %A Kounev, Samuel %B Lecture Notes in Computer Science (LNCS) %C Heidelberg, Germany %D 2008 %E Kounev, Samuel %E Gorton, Ian %E Sachs, Kai %I Springer %T {Performance Evaluation: Metrics, Models and Benchmarks, Proceedings of the 2008 SPEC International Performance Evaluation Workshop (SIPEW 2008), Darmstadt, Germany, June 27-28} %U http://www.springer.com/computer/programming/book/978-3-540-69813-5 %V 5119 %X This book constitutes the refereed proceedings of the SPEC International Performance Evaluation Workshop, SIPEW 2008, held in Darmstadt, Germany, in June 2008. The 17 revised full papers presented together with 3 keynote talks were carefully reviewed and selected out of 39 submissions for inclusion in the book. The papers are organized in topical sections on models for software performance engineering; benchmarks and workload characterization; Web services and service-oriented architectures; power and performance; and profiling, monitoring and optimization.

• 
  Performance Engineering of Distributed Component-Based Systems - Benchmarking, Modeling and Performance Prediction Kounev, Samuel;Shaker Verlag, Ph.D. Thesis, Technische Universit{\"a}t Darmstadt, Germany.

  Distinguished Dissertation Award
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @book{Ko2005-TUD-PhD_Thesis, author = {Kounev, Samuel}, month = {December}, note = {<b>Distinguished Dissertation Award</b>}, publisher = {Shaker Verlag, Ph.D. Thesis, Technische Universit{\"a}t Darmstadt, Germany}, title = {Performance Engineering of Distributed Component-Based Systems - Benchmarking, Modeling and Performance Prediction}, year = 2005 }

  %0 Book %1 Ko2005-TUD-PhD_Thesis %A Kounev, Samuel %D 2005 %I Shaker Verlag, Ph.D. Thesis, Technische Universit{\"a}t Darmstadt, Germany %T Performance Engineering of Distributed Component-Based Systems - Benchmarking, Modeling and Performance Prediction

• [ 2022 ]
• [ 2021 ]
• [ 2020 ]
• [ 2019 ]
• [ 2018 ]
• [ 2017 ]
• [ 2016 ]
• [ 2015 ]
• [ 2014 ]
• [ 2013 ]
• [ 2011 ]
• [ 2009 ]
• [ 2008 ]
• [ 2006 ]
• [ 2004 ]
• [ 1999 ]

• 
  A literature review on optimization techniques for adaptation planning in adaptive systems: State of the art and research directions. Henrichs, Elia; Lesch, Veronika; Straesser, Martin; Kounev, Samuel; Krupitzer, Christian; in Information and Software Technology 149 106940.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Context: Recent developments in modern IT systems including internet of things, edge/fog computing, or cyber–physical systems support intelligent and seamless interaction between users and systems. This requires a reaction to changes in their environment or the system. Adaptive systems provide mechanisms for these reactions. Objective: To implement this functionality, several approaches for the planning of adaptations exist that rely on rules, utility functions, or advanced techniques, such as machine learning. As the adaptation space with possible options is often extensively huge, optimization techniques might support efficient determination of the adaptation space and identify the system’s optimal configuration. With this paper, we provide a systematic review of adaptation planning as the optimization target. Method: In this paper, we review which optimization techniques are applied for adaptation planning in adaptive systems using a systematic literature review approach. Results: We reviewed 115 paper in detail out of an initial search set of 9,588 papers. Our analysis reveals that learning techniques and genetic algorithms are by far dominant; in total, heuristics (anytime learning) are more frequently applied as exact algorithms. We observed that around 57% of the approaches target multi-objectiveness and around 30% integrate distributed optimization. As last dimension, we focused on situation-awareness, which is only supported by two approaches. Conclusion: In this paper, we provide an overview of the current state of the art of approaches that rely on optimization techniques for planning adaptations in adaptive systems and further derive open research challenges, in particular regarding the integration of distributed optimization and situation-awareness.

  @article{HENRICHS2022106940, abstract = {Context: Recent developments in modern IT systems including internet of things, edge/fog computing, or cyber–physical systems support intelligent and seamless interaction between users and systems. This requires a reaction to changes in their environment or the system. Adaptive systems provide mechanisms for these reactions. Objective: To implement this functionality, several approaches for the planning of adaptations exist that rely on rules, utility functions, or advanced techniques, such as machine learning. As the adaptation space with possible options is often extensively huge, optimization techniques might support efficient determination of the adaptation space and identify the system’s optimal configuration. With this paper, we provide a systematic review of adaptation planning as the optimization target. Method: In this paper, we review which optimization techniques are applied for adaptation planning in adaptive systems using a systematic literature review approach. Results: We reviewed 115 paper in detail out of an initial search set of 9,588 papers. Our analysis reveals that learning techniques and genetic algorithms are by far dominant; in total, heuristics (anytime learning) are more frequently applied as exact algorithms. We observed that around 57% of the approaches target multi-objectiveness and around 30% integrate distributed optimization. As last dimension, we focused on situation-awareness, which is only supported by two approaches. Conclusion: In this paper, we provide an overview of the current state of the art of approaches that rely on optimization techniques for planning adaptations in adaptive systems and further derive open research challenges, in particular regarding the integration of distributed optimization and situation-awareness.}, author = {Henrichs, Elia and Lesch, Veronika and Straesser, Martin and Kounev, Samuel and Krupitzer, Christian}, journal = {Information and Software Technology}, pages = 106940, title = {A literature review on optimization techniques for adaptation planning in adaptive systems: State of the art and research directions}, volume = 149, year = 2022 }

  %0 Journal Article %1 HENRICHS2022106940 %A Henrichs, Elia %A Lesch, Veronika %A Straesser, Martin %A Kounev, Samuel %A Krupitzer, Christian %D 2022 %J Information and Software Technology %P 106940 %T A literature review on optimization techniques for adaptation planning in adaptive systems: State of the art and research directions %U https://www.sciencedirect.com/science/article/pii/S0950584922000891 %V 149 %X Context: Recent developments in modern IT systems including internet of things, edge/fog computing, or cyber–physical systems support intelligent and seamless interaction between users and systems. This requires a reaction to changes in their environment or the system. Adaptive systems provide mechanisms for these reactions. Objective: To implement this functionality, several approaches for the planning of adaptations exist that rely on rules, utility functions, or advanced techniques, such as machine learning. As the adaptation space with possible options is often extensively huge, optimization techniques might support efficient determination of the adaptation space and identify the system’s optimal configuration. With this paper, we provide a systematic review of adaptation planning as the optimization target. Method: In this paper, we review which optimization techniques are applied for adaptation planning in adaptive systems using a systematic literature review approach. Results: We reviewed 115 paper in detail out of an initial search set of 9,588 papers. Our analysis reveals that learning techniques and genetic algorithms are by far dominant; in total, heuristics (anytime learning) are more frequently applied as exact algorithms. We observed that around 57% of the approaches target multi-objectiveness and around 30% integrate distributed optimization. As last dimension, we focused on situation-awareness, which is only supported by two approaches. Conclusion: In this paper, we provide an overview of the current state of the art of approaches that rely on optimization techniques for planning adaptations in adaptive systems and further derive open research challenges, in particular regarding the integration of distributed optimization and situation-awareness.
• 
  A case study on the stability of performance tests for serverless applications. Eismann, Simon; Costa, Diego; Liao, Lizhi; Bezemer, Cor-Paul; Shang, Weiyi; van Hoorn, André; Kounev, Samuel; in Journal of Systems and Software (JSS) 189
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Context: While in serverless computing, application resource management and operational concerns are generally delegated to the cloud provider, ensuring that serverless applications meet their performance requirements is still a responsibility of the developers. Performance testing is a commonly used performance assessment practice; however, it traditionally requires visibility of the resource environment. Objective: In this study, we investigate whether performance tests of serverless applications are stable, that is, if their results are reproducible, and what implications the serverless paradigm has for performance tests. Method: We conduct a case study where we collect two datasets of performance test results: (a) repetitions of performance tests for varying memory size and load intensities and (b) three repetitions of the same performance test every day for ten months. Results: We find that performance tests of serverless applications are comparatively stable if conducted on the same day. However, we also observe short-term performance variations and frequent long-term performance changes. Conclusion: Performance tests for serverless applications can be stable; however, the serverless model impacts the planning, execution, and analysis of performance tests.

  @article{eismann2022study, abstract = {Context: While in serverless computing, application resource management and operational concerns are generally delegated to the cloud provider, ensuring that serverless applications meet their performance requirements is still a responsibility of the developers. Performance testing is a commonly used performance assessment practice; however, it traditionally requires visibility of the resource environment. Objective: In this study, we investigate whether performance tests of serverless applications are stable, that is, if their results are reproducible, and what implications the serverless paradigm has for performance tests. Method: We conduct a case study where we collect two datasets of performance test results: (a) repetitions of performance tests for varying memory size and load intensities and (b) three repetitions of the same performance test every day for ten months. Results: We find that performance tests of serverless applications are comparatively stable if conducted on the same day. However, we also observe short-term performance variations and frequent long-term performance changes. Conclusion: Performance tests for serverless applications can be stable; however, the serverless model impacts the planning, execution, and analysis of performance tests.}, author = {Eismann, Simon and Costa, Diego and Liao, Lizhi and Bezemer, Cor-Paul and Shang, Weiyi and van Hoorn, André and Kounev, Samuel}, journal = {Journal of Systems and Software (JSS)}, title = {A case study on the stability of performance tests for serverless applications}, volume = 189, year = 2022 }

  %0 Journal Article %1 eismann2022study %A Eismann, Simon %A Costa, Diego %A Liao, Lizhi %A Bezemer, Cor-Paul %A Shang, Weiyi %A van Hoorn, André %A Kounev, Samuel %D 2022 %J Journal of Systems and Software (JSS) %T A case study on the stability of performance tests for serverless applications %U https://www.sciencedirect.com/science/article/abs/pii/S0164121222000498 %V 189 %X Context: While in serverless computing, application resource management and operational concerns are generally delegated to the cloud provider, ensuring that serverless applications meet their performance requirements is still a responsibility of the developers. Performance testing is a commonly used performance assessment practice; however, it traditionally requires visibility of the resource environment. Objective: In this study, we investigate whether performance tests of serverless applications are stable, that is, if their results are reproducible, and what implications the serverless paradigm has for performance tests. Method: We conduct a case study where we collect two datasets of performance test results: (a) repetitions of performance tests for varying memory size and load intensities and (b) three repetitions of the same performance test every day for ten months. Results: We find that performance tests of serverless applications are comparatively stable if conducted on the same day. However, we also observe short-term performance variations and frequent long-term performance changes. Conclusion: Performance tests for serverless applications can be stable; however, the serverless model impacts the planning, execution, and analysis of performance tests.

• 
  A Machine Learning-based Workflow for Automatic Detection of Anomalies in Machine Tools. Züfle, Marwin; Moog, Felix; Lesch, Veronika; Krupitzer, Christian; Kounev, Samuel; in ISA Transactions: The Journal of Automation

  (in press)
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  Despite the increased sensor-based data collection in Industry 4.0, the practical use of this data is still in its infancy. In contrast, academic literature provides several approaches to detect machine failures but, in most cases, relies on simulations and vast amounts of training data. Since it is often not practical to collect such amounts of data in an industrial context, we propose an approach to detect the current production mode and machine degradation states on a comparably small data set. Our approach integrates domain knowledge about manufacturing systems into a highly generalizable end-to-end workflow ranging from raw data processing, phase segmentation, data resampling, and feature extraction to machine tool anomaly detection. The workflow applies unsupervised clustering techniques to identify the current production mode and supervised classification models for detecting the present degradation. A resampling strategy and classical machine learning models enable the workflow to handle small data sets and distinguish between normal and abnormal machine tool behavior. To the best of our knowledge, there exists no such end-to-end workflow in the literature that uses the entire machine signal as input to identify anomalies for individual tools. Our evaluation with data from a real multi-purpose machine shows that the proposed workflow detects anomalies with an average F1-score of almost 93%.

  @article{zuefle2021anomalydetection, abstract = {Despite the increased sensor-based data collection in Industry 4.0, the practical use of this data is still in its infancy. In contrast, academic literature provides several approaches to detect machine failures but, in most cases, relies on simulations and vast amounts of training data. Since it is often not practical to collect such amounts of data in an industrial context, we propose an approach to detect the current production mode and machine degradation states on a comparably small data set. Our approach integrates domain knowledge about manufacturing systems into a highly generalizable end-to-end workflow ranging from raw data processing, phase segmentation, data resampling, and feature extraction to machine tool anomaly detection. The workflow applies unsupervised clustering techniques to identify the current production mode and supervised classification models for detecting the present degradation. A resampling strategy and classical machine learning models enable the workflow to handle small data sets and distinguish between normal and abnormal machine tool behavior. To the best of our knowledge, there exists no such end-to-end workflow in the literature that uses the entire machine signal as input to identify anomalies for individual tools. Our evaluation with data from a real multi-purpose machine shows that the proposed workflow detects anomalies with an average F1-score of almost 93%.}, author = {Züfle, Marwin and Moog, Felix and Lesch, Veronika and Krupitzer, Christian and Kounev, Samuel}, journal = {ISA Transactions: The Journal of Automation}, note = {(in press)}, title = {A Machine Learning-based Workflow for Automatic Detection of Anomalies in Machine Tools}, year = 2021 }

  %0 Journal Article %1 zuefle2021anomalydetection %A Züfle, Marwin %A Moog, Felix %A Lesch, Veronika %A Krupitzer, Christian %A Kounev, Samuel %D 2021 %J ISA Transactions: The Journal of Automation %T A Machine Learning-based Workflow for Automatic Detection of Anomalies in Machine Tools %U https://www.sciencedirect.com/science/article/pii/S0019057821003773 %X Despite the increased sensor-based data collection in Industry 4.0, the practical use of this data is still in its infancy. In contrast, academic literature provides several approaches to detect machine failures but, in most cases, relies on simulations and vast amounts of training data. Since it is often not practical to collect such amounts of data in an industrial context, we propose an approach to detect the current production mode and machine degradation states on a comparably small data set. Our approach integrates domain knowledge about manufacturing systems into a highly generalizable end-to-end workflow ranging from raw data processing, phase segmentation, data resampling, and feature extraction to machine tool anomaly detection. The workflow applies unsupervised clustering techniques to identify the current production mode and supervised classification models for detecting the present degradation. A resampling strategy and classical machine learning models enable the workflow to handle small data sets and distinguish between normal and abnormal machine tool behavior. To the best of our knowledge, there exists no such end-to-end workflow in the literature that uses the entire machine signal as input to identify anomalies for individual tools. Our evaluation with data from a real multi-purpose machine shows that the proposed workflow detects anomalies with an average F1-score of almost 93%.
• 
  SARDE: A Framework for Continuous and Self-Adaptive Resource Demand Estimation. Grohmann, Johannes; Eismann, Simon; Bauer, Andr\’{e}; Spinner, Simon; Blum, Johannes; Herbst, Nikolas; Kounev, Samuel; in ACM Transactions on Autonomous and Adaptive Systems 15(2) Association for Computing Machinery, New York, NY, USA.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Resource demands are crucial parameters for modeling and predicting the performance of software systems. Currently, resource demand estimators are usually executed once for system analysis. However, the monitored system, as well as the resource demand itself, are subject to constant change in runtime environments. These changes additionally impact the applicability, the required parametrization as well as the resulting accuracy of individual estimation approaches. Over time, this leads to invalid or outdated estimates, which in turn negatively influence the decision-making of adaptive systems. In this article, we present SARDE, a framework for self-adaptive resource demand estimation in continuous environments. SARDE dynamically and continuously tunes, selects, and executes an ensemble of resource demand estimation approaches to adapt to changes in the environment. This creates an autonomous and unsupervised ensemble estimation technique, providing reliable resource demand estimations in dynamic environments. We evaluate SARDE using two realistic datasets. One set of different micro-benchmarks reflecting different possible system states and one dataset consisting of a continuously running application in a changing environment. Our results show that by continuously applying online optimization, selection and estimation, SARDE is able to efficiently adapt to the online trace and reduce the model error using the resulting ensemble technique.

  @article{GrEiBaSpBlHeKo2021-TaaS-SARDE, abstract = {Resource demands are crucial parameters for modeling and predicting the performance of software systems. Currently, resource demand estimators are usually executed once for system analysis. However, the monitored system, as well as the resource demand itself, are subject to constant change in runtime environments. These changes additionally impact the applicability, the required parametrization as well as the resulting accuracy of individual estimation approaches. Over time, this leads to invalid or outdated estimates, which in turn negatively influence the decision-making of adaptive systems. In this article, we present SARDE, a framework for self-adaptive resource demand estimation in continuous environments. SARDE dynamically and continuously tunes, selects, and executes an ensemble of resource demand estimation approaches to adapt to changes in the environment. This creates an autonomous and unsupervised ensemble estimation technique, providing reliable resource demand estimations in dynamic environments. We evaluate SARDE using two realistic datasets. One set of different micro-benchmarks reflecting different possible system states and one dataset consisting of a continuously running application in a changing environment. Our results show that by continuously applying online optimization, selection and estimation, SARDE is able to efficiently adapt to the online trace and reduce the model error using the resulting ensemble technique.}, address = {New York, NY, USA}, author = {Grohmann, Johannes and Eismann, Simon and Bauer, Andr\'{e} and Spinner, Simon and Blum, Johannes and Herbst, Nikolas and Kounev, Samuel}, journal = {ACM Transactions on Autonomous and Adaptive Systems}, month = {jun}, number = 2, publisher = {Association for Computing Machinery}, title = {SARDE: A Framework for Continuous and Self-Adaptive Resource Demand Estimation}, volume = 15, year = 2021 }

  %0 Journal Article %1 GrEiBaSpBlHeKo2021-TaaS-SARDE %A Grohmann, Johannes %A Eismann, Simon %A Bauer, Andr\'{e} %A Spinner, Simon %A Blum, Johannes %A Herbst, Nikolas %A Kounev, Samuel %C New York, NY, USA %D 2021 %I Association for Computing Machinery %J ACM Transactions on Autonomous and Adaptive Systems %N 2 %T SARDE: A Framework for Continuous and Self-Adaptive Resource Demand Estimation %U https://doi.org/10.1145/3463369 %V 15 %X Resource demands are crucial parameters for modeling and predicting the performance of software systems. Currently, resource demand estimators are usually executed once for system analysis. However, the monitored system, as well as the resource demand itself, are subject to constant change in runtime environments. These changes additionally impact the applicability, the required parametrization as well as the resulting accuracy of individual estimation approaches. Over time, this leads to invalid or outdated estimates, which in turn negatively influence the decision-making of adaptive systems. In this article, we present SARDE, a framework for self-adaptive resource demand estimation in continuous environments. SARDE dynamically and continuously tunes, selects, and executes an ensemble of resource demand estimation approaches to adapt to changes in the environment. This creates an autonomous and unsupervised ensemble estimation technique, providing reliable resource demand estimations in dynamic environments. We evaluate SARDE using two realistic datasets. One set of different micro-benchmarks reflecting different possible system states and one dataset consisting of a continuously running application in a changing environment. Our results show that by continuously applying online optimization, selection and estimation, SARDE is able to efficiently adapt to the online trace and reduce the model error using the resulting ensemble technique.
• 
  A Comparison of Mechanisms for Compensating Negative Impacts of System Integration. Lesch, Veronika; Krupitzer, Christian; Stubenrauch, Kevin; Keil, Nico; Becker, Christian; Kounev, Samuel; Segata, Michele; in Future Generation Computer Systems 116 117–131.
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @article{lesch2020comparison, author = {Lesch, Veronika and Krupitzer, Christian and Stubenrauch, Kevin and Keil, Nico and Becker, Christian and Kounev, Samuel and Segata, Michele}, journal = {Future Generation Computer Systems}, month = {March}, pages = {117-131}, title = {A Comparison of Mechanisms for Compensating Negative Impacts of System Integration}, volume = 116, year = 2021 }

  %0 Journal Article %1 lesch2020comparison %A Lesch, Veronika %A Krupitzer, Christian %A Stubenrauch, Kevin %A Keil, Nico %A Becker, Christian %A Kounev, Samuel %A Segata, Michele %D 2021 %J Future Generation Computer Systems %P 117-131 %T A Comparison of Mechanisms for Compensating Negative Impacts of System Integration %V 116
• 
  An Overview on Approaches for Coordination of Platoons. Lesch, Veronika; Breitbach, Martin; Segata, Michele; Becker, Christian; Kounev, Samuel; Krupitzer, Christian; in IEEE Transactions on Intelligent Transportation Systems

  Early Access on IEEE Xplore
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @article{lesch2021overview, author = {Lesch, Veronika and Breitbach, Martin and Segata, Michele and Becker, Christian and Kounev, Samuel and Krupitzer, Christian}, journal = {IEEE Transactions on Intelligent Transportation Systems}, note = {Early Access on IEEE Xplore}, title = {An Overview on Approaches for Coordination of Platoons}, year = 2021 }

  %0 Journal Article %1 lesch2021overview %A Lesch, Veronika %A Breitbach, Martin %A Segata, Michele %A Becker, Christian %A Kounev, Samuel %A Krupitzer, Christian %D 2021 %J IEEE Transactions on Intelligent Transportation Systems %T An Overview on Approaches for Coordination of Platoons
• 
  Tackling the Rich Vehicle Routing Problem with Nature-Inspired Algorithms. Lesch, Veronika; König, Maximilian; Kounev, Samuel; Stein, Anthony; Krupitzer, Christian; in Applied Intelligence 52 9476–9500.
  • [ BibTeX ]
  • [ EndNote ]
  @article{lesch2021tackling, author = {Lesch, Veronika and König, Maximilian and Kounev, Samuel and Stein, Anthony and Krupitzer, Christian}, journal = {Applied Intelligence}, pages = {9476–9500}, title = {Tackling the Rich Vehicle Routing Problem with Nature-Inspired Algorithms}, volume = 52, year = 2021 }

  %0 Journal Article %1 lesch2021tackling %A Lesch, Veronika %A König, Maximilian %A Kounev, Samuel %A Stein, Anthony %A Krupitzer, Christian %D 2021 %J Applied Intelligence %P 9476–9500 %T Tackling the Rich Vehicle Routing Problem with Nature-Inspired Algorithms %V 52
• 
  People Detection with Depth Silhouettes and Convolutional Neural Networks on a Mobile Robot. Spiess, Florian; Reinhart, Lucas; Strobel, Norbert; Kaiser, Dennis; Kounev, Samuel; Kaupp, Tobias; in Journal of Image and Graphics, (J. of Image; Graphics, eds.) 9(4) 135–139.
  • [ BibTeX ]
  • [ EndNote ]
  @article{spiess2021people, author = {Spiess, Florian and Reinhart, Lucas and Strobel, Norbert and Kaiser, Dennis and Kounev, Samuel and Kaupp, Tobias}, editor = {of Image, Journal and Graphics}, journal = {Journal of Image and Graphics}, month = {December 2021}, number = 4, pages = {135-139}, title = {People Detection with Depth Silhouettes and Convolutional Neural Networks on a Mobile Robot}, volume = 9, year = 2021 }

  %0 Journal Article %1 spiess2021people %A Spiess, Florian %A Reinhart, Lucas %A Strobel, Norbert %A Kaiser, Dennis %A Kounev, Samuel %A Kaupp, Tobias %D 2021 %E of Image, Journal %E Graphics, %J Journal of Image and Graphics %N 4 %P 135-139 %T People Detection with Depth Silhouettes and Convolutional Neural Networks on a Mobile Robot %V 9

• 
  Survey and Experimental Comparison of RGB-D Indoor Robot Navigation Methods Supported by ROS and Their Expansion via Fusion with Wheel Odometry and IMU Data. Spiess, Florian; Friesslich, Jonas; Tobias, Kaupp; Kounev, Samuel; Strobel, Norbert; in International Journal of Mechanical Engineering and Robotics Research (IJMERR), (P. R. (Chunhui) Yang, ed.) 9(12) 1532–1540.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  This paper presents an experimental evaluation and comparison of selected Visual Odometry (VO) and Visual-SLAM (V-SLAM) algorithms for indoor mobile robot navigation supported by the Robot Operating System (ROS). The focus is on algorithms involving RGB-D cameras. Since RGB-D cameras integrate color and depth information, they output coherent measurement data and facilitate an efficient processing pipeline. The various underlying methods of vision-based algorithms are described and evaluated on two datasets covering different indoor situations as well as various lighting and movement conditions. In general, V-SLAM algorithms yielded better results. They were superior with respect to handling drift, in particular when loop closures were involved. However, the results confirmed that VO algorithms could outperform V-SLAM methods under certain circumstances. This happened when there was a very good match between an algorithm’s design objectives and the situation at hand. While the experiments showed that there is no single best algorithm for every scenario, ORB-SLAM2 is recommended as a robust stand-alone RGB-D based localization method available under ROS. Furthermore, we observed that the position estimation error could be reduced by around 67% on average when combining vision-based position estimates with sensor data obtained from wheel odometry and an inertial measurement unit (IMU), respectively. This clearly demonstrates the potential of sensor fusion techniques. The best results in case of sensor fusion were obtained with RGB-DSLAMv2.

  @article{yang2020survey, abstract = {This paper presents an experimental evaluation and comparison of selected Visual Odometry (VO) and Visual-SLAM (V-SLAM) algorithms for indoor mobile robot navigation supported by the Robot Operating System (ROS). The focus is on algorithms involving RGB-D cameras. Since RGB-D cameras integrate color and depth information, they output coherent measurement data and facilitate an efficient processing pipeline. The various underlying methods of vision-based algorithms are described and evaluated on two datasets covering different indoor situations as well as various lighting and movement conditions. In general, V-SLAM algorithms yielded better results. They were superior with respect to handling drift, in particular when loop closures were involved. However, the results confirmed that VO algorithms could outperform V-SLAM methods under certain circumstances. This happened when there was a very good match between an algorithm’s design objectives and the situation at hand. While the experiments showed that there is no single best algorithm for every scenario, ORB-SLAM2 is recommended as a robust stand-alone RGB-D based localization method available under ROS. Furthermore, we observed that the position estimation error could be reduced by around 67% on average when combining vision-based position estimates with sensor data obtained from wheel odometry and an inertial measurement unit (IMU), respectively. This clearly demonstrates the potential of sensor fusion techniques. The best results in case of sensor fusion were obtained with RGB-DSLAMv2.}, author = {Spiess, Florian and Friesslich, Jonas and Tobias, Kaupp and Kounev, Samuel and Strobel, Norbert}, editor = {Yang, Prof Richard (Chunhui)}, journal = {International Journal of Mechanical Engineering and Robotics Research (IJMERR)}, month = {dec}, number = 12, pages = {1532-1540}, title = {Survey and Experimental Comparison of RGB-D Indoor Robot Navigation Methods Supported by ROS and Their Expansion via Fusion with Wheel Odometry and IMU Data}, volume = 9, year = 2020 }

  %0 Journal Article %1 yang2020survey %A Spiess, Florian %A Friesslich, Jonas %A Tobias, Kaupp %A Kounev, Samuel %A Strobel, Norbert %D 2020 %E Yang, Prof Richard (Chunhui) %J International Journal of Mechanical Engineering and Robotics Research (IJMERR) %N 12 %P 1532-1540 %T Survey and Experimental Comparison of RGB-D Indoor Robot Navigation Methods Supported by ROS and Their Expansion via Fusion with Wheel Odometry and IMU Data %U http://www.ijmerr.com/index.php?m=content&c=index&a=show&catid=185&id=1529 %V 9 %X This paper presents an experimental evaluation and comparison of selected Visual Odometry (VO) and Visual-SLAM (V-SLAM) algorithms for indoor mobile robot navigation supported by the Robot Operating System (ROS). The focus is on algorithms involving RGB-D cameras. Since RGB-D cameras integrate color and depth information, they output coherent measurement data and facilitate an efficient processing pipeline. The various underlying methods of vision-based algorithms are described and evaluated on two datasets covering different indoor situations as well as various lighting and movement conditions. In general, V-SLAM algorithms yielded better results. They were superior with respect to handling drift, in particular when loop closures were involved. However, the results confirmed that VO algorithms could outperform V-SLAM methods under certain circumstances. This happened when there was a very good match between an algorithm’s design objectives and the situation at hand. While the experiments showed that there is no single best algorithm for every scenario, ORB-SLAM2 is recommended as a robust stand-alone RGB-D based localization method available under ROS. Furthermore, we observed that the position estimation error could be reduced by around 67% on average when combining vision-based position estimates with sensor data obtained from wheel odometry and an inertial measurement unit (IMU), respectively. This clearly demonstrates the potential of sensor fusion techniques. The best results in case of sensor fusion were obtained with RGB-DSLAMv2.
• 
  The impact of the German strategy for containment of Coronavirus SARS-CoV-2 on the training characteristics, physical activity, sleep of highly trained kayakers and canoeists: A retrospective observational study. Zinner, Christoph; Matzka, Manuel; Leppich, Robert; Kounev, Samuel; Holmberg, HC; Sperlich, Billy; in Frontiers in Sports and Active Living 2 127. Frontiers.
  • [ BibTeX ]
  • [ EndNote ]
  @article{zinner2020impact, author = {Zinner, Christoph and Matzka, Manuel and Leppich, Robert and Kounev, Samuel and Holmberg, HC and Sperlich, Billy}, journal = {Frontiers in Sports and Active Living}, pages = 127, publisher = {Frontiers}, title = {The impact of the German strategy for containment of Coronavirus SARS-CoV-2 on the training characteristics, physical activity, sleep of highly trained kayakers and canoeists: A retrospective observational study}, volume = 2, year = 2020 }

  %0 Journal Article %1 zinner2020impact %A Zinner, Christoph %A Matzka, Manuel %A Leppich, Robert %A Kounev, Samuel %A Holmberg, HC %A Sperlich, Billy %D 2020 %I Frontiers %J Frontiers in Sports and Active Living %P 127 %T The impact of the German strategy for containment of Coronavirus SARS-CoV-2 on the training characteristics, physical activity, sleep of highly trained kayakers and canoeists: A retrospective observational study %V 2
• 
  {Time Series Forecasting for Self-Aware Systems}. Bauer, Andr{\’e}; Z{\"u}fle, Marwin; Herbst, Nikolas; Zehe, Albin; Hotho, Andreas; Kounev, Samuel; in Proceedings of the IEEE 108(7) 1068–1093.
  • [ BibTeX ]
  • [ EndNote ]
  @article{BaZuHeZeHoKo-PIEEE-Forecasting, author = {Bauer, Andr{\'e} and Z{\"u}fle, Marwin and Herbst, Nikolas and Zehe, Albin and Hotho, Andreas and Kounev, Samuel}, journal = {Proceedings of the IEEE}, month = 7, number = 7, pages = {1068 - 1093}, title = {{Time Series Forecasting for Self-Aware Systems}}, volume = 108, year = 2020 }

  %0 Journal Article %1 BaZuHeZeHoKo-PIEEE-Forecasting %A Bauer, Andr{\'e} %A Z{\"u}fle, Marwin %A Herbst, Nikolas %A Zehe, Albin %A Hotho, Andreas %A Kounev, Samuel %D 2020 %J Proceedings of the IEEE %N 7 %P 1068 - 1093 %T {Time Series Forecasting for Self-Aware Systems} %V 108
• 
  A Taxonomy of Techniques for SLO Failure Prediction in Software Systems. Grohmann, Johannes; Herbst, Nikolas; Chalbani, Avi; Arian, Yair; Peretz, Noam; Kounev, Samuel; in Computers 9(1) 10. Multidisciplinary Digital Publishing Institute (MDPI).
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  Failure prediction is an important aspect of self-aware computing systems. Therefore, a multitude of different approaches has been proposed in the literature over the past few years. In this work, we propose a taxonomy for organizing works focusing on the prediction of Service Level Objective (SLO) failures. Our taxonomy classifies related work along the dimensions of the prediction target (e.g., anomaly detection, performance prediction, or failure prediction), the time horizon (e.g., detection or prediction, online or offline application), and the applied modeling type (e.g., time series forecasting, machine learning, or queueing theory). The classification is derived based on a systematic mapping of relevant papers in the area. Additionally, we give an overview of different techniques in each sub-group and address remaining challenges in order to guide future research.

  @article{grohmann2020-MDPI-taxonomy, abstract = {Failure prediction is an important aspect of self-aware computing systems. Therefore, a multitude of different approaches has been proposed in the literature over the past few years. In this work, we propose a taxonomy for organizing works focusing on the prediction of Service Level Objective (SLO) failures. Our taxonomy classifies related work along the dimensions of the prediction target (e.g., anomaly detection, performance prediction, or failure prediction), the time horizon (e.g., detection or prediction, online or offline application), and the applied modeling type (e.g., time series forecasting, machine learning, or queueing theory). The classification is derived based on a systematic mapping of relevant papers in the area. Additionally, we give an overview of different techniques in each sub-group and address remaining challenges in order to guide future research.}, author = {Grohmann, Johannes and Herbst, Nikolas and Chalbani, Avi and Arian, Yair and Peretz, Noam and Kounev, Samuel}, journal = {Computers}, number = 1, pages = 10, publisher = {Multidisciplinary Digital Publishing Institute (MDPI)}, title = {A Taxonomy of Techniques for SLO Failure Prediction in Software Systems}, volume = 9, year = 2020 }

  %0 Journal Article %1 grohmann2020-MDPI-taxonomy %A Grohmann, Johannes %A Herbst, Nikolas %A Chalbani, Avi %A Arian, Yair %A Peretz, Noam %A Kounev, Samuel %D 2020 %I Multidisciplinary Digital Publishing Institute (MDPI) %J Computers %N 1 %P 10 %T A Taxonomy of Techniques for SLO Failure Prediction in Software Systems %U https://doi.org/10.3390/computers9010010 %V 9 %X Failure prediction is an important aspect of self-aware computing systems. Therefore, a multitude of different approaches has been proposed in the literature over the past few years. In this work, we propose a taxonomy for organizing works focusing on the prediction of Service Level Objective (SLO) failures. Our taxonomy classifies related work along the dimensions of the prediction target (e.g., anomaly detection, performance prediction, or failure prediction), the time horizon (e.g., detection or prediction, online or offline application), and the applied modeling type (e.g., time series forecasting, machine learning, or queueing theory). The classification is derived based on a systematic mapping of relevant papers in the area. Additionally, we give an overview of different techniques in each sub-group and address remaining challenges in order to guide future research.
• 
  {Towards Self-Aware Multirotor Formations}. Kaiser, Dennis; Lesch, Veronika; Rothe, Julian; Strohmeier, Michael; Spiess, Florian; Krupitzer, Christian; Montenegro, Sergio; Kounev, Samuel; in Computers 9(7) MDPI.

  Special Issue on Self-Aware Computing
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @article{Kaiser2020Towards, author = {Kaiser, Dennis and Lesch, Veronika and Rothe, Julian and Strohmeier, Michael and Spiess, Florian and Krupitzer, Christian and Montenegro, Sergio and Kounev, Samuel}, journal = {Computers}, month = {Februar}, note = {Special Issue on Self-Aware Computing}, number = 7, publisher = {MDPI}, title = {{Towards Self-Aware Multirotor Formations}}, volume = 9, year = 2020 }

  %0 Journal Article %1 Kaiser2020Towards %A Kaiser, Dennis %A Lesch, Veronika %A Rothe, Julian %A Strohmeier, Michael %A Spiess, Florian %A Krupitzer, Christian %A Montenegro, Sergio %A Kounev, Samuel %D 2020 %I MDPI %J Computers %N 7 %T {Towards Self-Aware Multirotor Formations} %V 9

• 
  {Modeling of Aggregated IoT Traffic and Its Application to an IoT Cloud}. Metzger, Florian; Ho{\ss}feld, Tobias; Bauer, Andr{\’e}; Kounev, Samuel; Heegaard, Poul. E.; in Proceedings of the IEEE 107(4) 679–694.
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @article{MeHoBaKoHe2019-PIEEE-IoTTraffic, author = {Metzger, Florian and Ho{\ss}feld, Tobias and Bauer, Andr{\'e} and Kounev, Samuel and Heegaard, Poul. E.}, journal = {Proceedings of the IEEE}, month = {April}, number = 4, pages = {679-694}, title = {{Modeling of Aggregated IoT Traffic and Its Application to an IoT Cloud}}, volume = 107, year = 2019 }

  %0 Journal Article %1 MeHoBaKoHe2019-PIEEE-IoTTraffic %A Metzger, Florian %A Ho{\ss}feld, Tobias %A Bauer, Andr{\'e} %A Kounev, Samuel %A Heegaard, Poul. E. %D 2019 %J Proceedings of the IEEE %N 4 %P 679-694 %T {Modeling of Aggregated IoT Traffic and Its Application to an IoT Cloud} %V 107
• 
  Online model learning for self-aware computing infrastructures. Spinner, Simon; Grohmann, Johannes; Eismann, Simon; Kounev, Samuel; in Journal of Systems and Software 147 1–16.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Performance models are valuable and powerful tools for performance prediction. However, the creation of performance models usually requires significant manual effort. Furthermore, as the modeled structures are subject to frequent change in modern infrastructures, such performance models need to be adapted as well. We therefore propose a reference architecture for online model learning in virtualized environments, which enables the automatic extraction of the aforementioned performance models. We follow an agent-based approach, which enables us to incorporate the extraction of information about the application structure as well as the virtualization structures present in modern computing centers. Our evaluation shows that our collaborating agents are able to reduce the manual effort of performance model extraction by 85.4%. The resulting performance model is able to predict the system utilization with an absolute error of less than 4% and the end-to-end response time with a relative error of less than 21%.

  @article{SpGrEiKo2019-JSS-ModelLearning, abstract = {Performance models are valuable and powerful tools for performance prediction. However, the creation of performance models usually requires significant manual effort. Furthermore, as the modeled structures are subject to frequent change in modern infrastructures, such performance models need to be adapted as well. We therefore propose a reference architecture for online model learning in virtualized environments, which enables the automatic extraction of the aforementioned performance models. We follow an agent-based approach, which enables us to incorporate the extraction of information about the application structure as well as the virtualization structures present in modern computing centers. Our evaluation shows that our collaborating agents are able to reduce the manual effort of performance model extraction by 85.4%. The resulting performance model is able to predict the system utilization with an absolute error of less than 4% and the end-to-end response time with a relative error of less than 21%.}, author = {Spinner, Simon and Grohmann, Johannes and Eismann, Simon and Kounev, Samuel}, journal = {Journal of Systems and Software}, pages = {1--16}, title = {Online model learning for self-aware computing infrastructures}, volume = 147, year = 2019 }

  %0 Journal Article %1 SpGrEiKo2019-JSS-ModelLearning %A Spinner, Simon %A Grohmann, Johannes %A Eismann, Simon %A Kounev, Samuel %D 2019 %J Journal of Systems and Software %P 1--16 %T Online model learning for self-aware computing infrastructures %U https://doi.org/10.1016/j.jss.2018.09.089 %V 147 %X Performance models are valuable and powerful tools for performance prediction. However, the creation of performance models usually requires significant manual effort. Furthermore, as the modeled structures are subject to frequent change in modern infrastructures, such performance models need to be adapted as well. We therefore propose a reference architecture for online model learning in virtualized environments, which enables the automatic extraction of the aforementioned performance models. We follow an agent-based approach, which enables us to incorporate the extraction of information about the application structure as well as the virtualization structures present in modern computing centers. Our evaluation shows that our collaborating agents are able to reduce the manual effort of performance model extraction by 85.4%. The resulting performance model is able to predict the system utilization with an absolute error of less than 4% and the end-to-end response time with a relative error of less than 21%.
• 
  {Chameleon: A Hybrid, Proactive Auto-Scaling Mechanism on a Level-Playing Field}. Bauer, Andr{\’e}; Herbst, Nikolas; Spinner, Simon; Ali-Eldin, Ahmed; Kounev, Samuel; in IEEE Transactions on Parallel and Distributed Systems 30(4) 800–813. IEEE.
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @article{BaHeSpAEKo2019-TPDS-ChameleonAutoscaler, author = {Bauer, Andr{\'e} and Herbst, Nikolas and Spinner, Simon and Ali-Eldin, Ahmed and Kounev, Samuel}, journal = {IEEE Transactions on Parallel and Distributed Systems}, month = {April}, number = 4, pages = {800--813}, publisher = {IEEE}, title = {{Chameleon: A Hybrid, Proactive Auto-Scaling Mechanism on a Level-Playing Field}}, volume = 30, year = 2019 }

  %0 Journal Article %1 BaHeSpAEKo2019-TPDS-ChameleonAutoscaler %A Bauer, Andr{\'e} %A Herbst, Nikolas %A Spinner, Simon %A Ali-Eldin, Ahmed %A Kounev, Samuel %D 2019 %I IEEE %J IEEE Transactions on Parallel and Distributed Systems %N 4 %P 800--813 %T {Chameleon: A Hybrid, Proactive Auto-Scaling Mechanism on a Level-Playing Field} %V 30
• 
  {Measuring and Rating the Energy-Efficiency of Servers}. von Kistowski, J{\’o}akim; Lange, Klaus-Dieter; Arnold, Jeremy A.; Beckett, John; Block, Hansfried; Tricker, Mike; Sharma, Sanjay; Pais, Johann; Kounev, Samuel; in Future Generation Computer Systems 100 579–589.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  Data centers and servers consume a significant amount of electrical power. As future data centers increase in size, it becomes increasingly important to be able to select servers based on their energy efficiency. Rating the efficiency of servers is challenging, as it depends on the software executed on that server and on its load profile. To account for this, a measurement and rating methodology for servers must make use of both realistic and varying workloads. Existing energy-efficiency benchmarks either run standardized application loads at multiple load levels or multiple micro-benchmarks at full load. This does not enable a full analysis of system behavior as the energy efficiency for different kinds of software at low load levels remains unknown. This article introduces a measurement methodology and metrics for energy-efficiency rating of servers. The methodology defines system setup and instrumentation for reproducible results. It is designed to use multiple, specifically chosen workloads at different load levels for a full system characterization. All partial results are aggregated in a final metric that can be used by regulators to define server-labeling criteria. The methodology and workloads have been implemented in the standardized Server Efficiency Rating Tool. We show the applicability and use of the measurement methodology specifically considering its reproducibility, fairness, and relevance. A reproducibility analysis shows that efficiency scores vary with a maximum coefficient of 1.04\% for repeated experiments. In addition, we evaluate the proposed metrics by investigating their energy-efficiency rating on a set of 385 different servers and show the relevance of the selected workloads by analyzing relationships to real-world applications.

  @article{KiLaArBeBlTrShPaKo-FGCS2019-ServerEfficiency, abstract = {Data centers and servers consume a significant amount of electrical power. As future data centers increase in size, it becomes increasingly important to be able to select servers based on their energy efficiency. Rating the efficiency of servers is challenging, as it depends on the software executed on that server and on its load profile. To account for this, a measurement and rating methodology for servers must make use of both realistic and varying workloads. Existing energy-efficiency benchmarks either run standardized application loads at multiple load levels or multiple micro-benchmarks at full load. This does not enable a full analysis of system behavior as the energy efficiency for different kinds of software at low load levels remains unknown. This article introduces a measurement methodology and metrics for energy-efficiency rating of servers. The methodology defines system setup and instrumentation for reproducible results. It is designed to use multiple, specifically chosen workloads at different load levels for a full system characterization. All partial results are aggregated in a final metric that can be used by regulators to define server-labeling criteria. The methodology and workloads have been implemented in the standardized Server Efficiency Rating Tool. We show the applicability and use of the measurement methodology specifically considering its reproducibility, fairness, and relevance. A reproducibility analysis shows that efficiency scores vary with a maximum coefficient of 1.04\% for repeated experiments. In addition, we evaluate the proposed metrics by investigating their energy-efficiency rating on a set of 385 different servers and show the relevance of the selected workloads by analyzing relationships to real-world applications.}, author = {von Kistowski, J{\'o}akim and Lange, Klaus-Dieter and Arnold, Jeremy A. and Beckett, John and Block, Hansfried and Tricker, Mike and Sharma, Sanjay and Pais, Johann and Kounev, Samuel}, journal = {Future Generation Computer Systems}, pages = {579 - 589}, title = {{Measuring and Rating the Energy-Efficiency of Servers}}, volume = 100, year = 2019 }

  %0 Journal Article %1 KiLaArBeBlTrShPaKo-FGCS2019-ServerEfficiency %A von Kistowski, J{\'o}akim %A Lange, Klaus-Dieter %A Arnold, Jeremy A. %A Beckett, John %A Block, Hansfried %A Tricker, Mike %A Sharma, Sanjay %A Pais, Johann %A Kounev, Samuel %D 2019 %J Future Generation Computer Systems %P 579 - 589 %T {Measuring and Rating the Energy-Efficiency of Servers} %U http://www.sciencedirect.com/science/article/pii/S0167739X18312123 %V 100 %X Data centers and servers consume a significant amount of electrical power. As future data centers increase in size, it becomes increasingly important to be able to select servers based on their energy efficiency. Rating the efficiency of servers is challenging, as it depends on the software executed on that server and on its load profile. To account for this, a measurement and rating methodology for servers must make use of both realistic and varying workloads. Existing energy-efficiency benchmarks either run standardized application loads at multiple load levels or multiple micro-benchmarks at full load. This does not enable a full analysis of system behavior as the energy efficiency for different kinds of software at low load levels remains unknown. This article introduces a measurement methodology and metrics for energy-efficiency rating of servers. The methodology defines system setup and instrumentation for reproducible results. It is designed to use multiple, specifically chosen workloads at different load levels for a full system characterization. All partial results are aggregated in a final metric that can be used by regulators to define server-labeling criteria. The methodology and workloads have been implemented in the standardized Server Efficiency Rating Tool. We show the applicability and use of the measurement methodology specifically considering its reproducibility, fairness, and relevance. A reproducibility analysis shows that efficiency scores vary with a maximum coefficient of 1.04\% for repeated experiments. In addition, we evaluate the proposed metrics by investigating their energy-efficiency rating on a set of 385 different servers and show the relevance of the selected workloads by analyzing relationships to real-world applications.

• 
  {CUP: A Formalism for Expressing Cloud Usage Patterns for Experts and Non-Experts}. Milenkoski, Aleksandar; Iosup, Alexandru; Kounev, Samuel; Sachs, Kai; Mularz, Diane E.; Curtiss, Jonathan A.; Ding, Jason J.; Rosenberg, Florian; Rygielski, Piotr; in IEEE Cloud Computing 5(3) 65–76. IEEE.
  • [ BibTeX ]
  • [ EndNote ]
  @article{MiIoKoSaMuCuDiRoRy2018-IEEECC-cup, author = {Milenkoski, Aleksandar and Iosup, Alexandru and Kounev, Samuel and Sachs, Kai and Mularz, Diane E. and Curtiss, Jonathan A. and Ding, Jason J. and Rosenberg, Florian and Rygielski, Piotr}, journal = {IEEE Cloud Computing}, month = 6, number = 3, pages = {65--76}, publisher = {IEEE}, title = {{CUP: A Formalism for Expressing Cloud Usage Patterns for Experts and Non-Experts}}, volume = 5, year = 2018 }

  %0 Journal Article %1 MiIoKoSaMuCuDiRoRy2018-IEEECC-cup %A Milenkoski, Aleksandar %A Iosup, Alexandru %A Kounev, Samuel %A Sachs, Kai %A Mularz, Diane E. %A Curtiss, Jonathan A. %A Ding, Jason J. %A Rosenberg, Florian %A Rygielski, Piotr %D 2018 %I IEEE %J IEEE Cloud Computing %N 3 %P 65--76 %T {CUP: A Formalism for Expressing Cloud Usage Patterns for Experts and Non-Experts} %V 5
• 
  {Quantifying Cloud Performance and Dependability: Taxonomy, Metric Design, and Emerging Challenges}. Herbst, Nikolas; Bauer, Andr{\’e}; Kounev, Samuel; Oikonomou, Giorgos; van Eyk, Erwin; Kousiouris, George; Evangelinou, Athanasia; Krebs, Rouven; Brecht, Tim; Abad, Cristina L.; Iosup, Alexandru; in ACM Transactions on Modeling and Performance Evaluation of Computing Systems (ToMPECS) 3(4) 19:1–19:36. ACM, New York, NY, USA.
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  @article{HeBaKoOiEyKoEvKrBrAbIo2018-TOMPECS-CloudMetrics, address = {New York, NY, USA}, author = {Herbst, Nikolas and Bauer, Andr{\'e} and Kounev, Samuel and Oikonomou, Giorgos and van Eyk, Erwin and Kousiouris, George and Evangelinou, Athanasia and Krebs, Rouven and Brecht, Tim and Abad, Cristina L. and Iosup, Alexandru}, journal = {ACM Transactions on Modeling and Performance Evaluation of Computing Systems (ToMPECS)}, month = 8, number = 4, pages = {19:1--19:36}, publisher = {ACM}, title = {{Quantifying Cloud Performance and Dependability: Taxonomy, Metric Design, and Emerging Challenges}}, volume = 3, year = 2018 }

  %0 Journal Article %1 HeBaKoOiEyKoEvKrBrAbIo2018-TOMPECS-CloudMetrics %A Herbst, Nikolas %A Bauer, Andr{\'e} %A Kounev, Samuel %A Oikonomou, Giorgos %A van Eyk, Erwin %A Kousiouris, George %A Evangelinou, Athanasia %A Krebs, Rouven %A Brecht, Tim %A Abad, Cristina L. %A Iosup, Alexandru %C New York, NY, USA %D 2018 %I ACM %J ACM Transactions on Modeling and Performance Evaluation of Computing Systems (ToMPECS) %N 4 %P 19:1--19:36 %T {Quantifying Cloud Performance and Dependability: Taxonomy, Metric Design, and Emerging Challenges} %U http://doi.acm.org/10.1145/3236332 %V 3

• 
  {Modeling and Extracting Load Intensity Profiles}. von Kistowski, J{\’o}akim; Herbst, Nikolas; Kounev, Samuel; Groenda, Henning; Stier, Christian; Lehrig, Sebastian; in ACM Transactions on Autonomous and Adaptive Systems (TAAS) 11(4) 23:1–23:28. ACM, New York, NY, USA.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Today's system developers and operators face the challenge of creating software systems that make efficient use of dynamically allocated resources under highly variable and dynamic load profiles, while at the same time delivering reliable performance. Autonomic controllers, e.g., an advanced auto-scaling mechanism in a cloud computing context, can benefit from an abstracted load model as knowledge to reconfigure on time and precisely. Existing workload characterization approaches have limited support to capture variations the inter-arrival times of incoming work units over time (i.e., a variable load profile). For example, industrial and scientific benchmarks support constant or stepwise increasing load, or inter-arrival times defined by statistical distributions or recorded traces. These options show shortcomings either in representative character of load variation patterns or in abstraction and flexibility of their format. In this article, we present the Descartes Load Intensity Model (DLIM) approach addressing these issues. DLIM provides a modeling formalism for describing load intensity variations over time. A DLIM instance is a compact formal description of a load intensity trace. DLIM-based tools provide features for benchmarking, performance and recorded load intensity trace analysis. As manually obtaining and maintaining DLIM instances becomes time consuming, we contribute three automated extraction methods and devised metrics for comparison and method selection. We discuss how these features are used to enhance system management approaches for adaptations during run-time, and how they are integrated into simulation contexts and enable benchmarking of elastic or adaptive behavior. We show that automatically extracted DLIM instances exhibit an average modeling error of 15.2\% over ten different real-world traces that cover between two weeks and seven months. These results underline DLIM model expressiveness. In terms of accuracy and processing speed, our proposed extraction methods for the descriptive models are comparable to existing time series decomposition methods. Additionally, we illustrate DLIM applicability by outlining approaches of workload modeling in systems engineering that employ or rely on our proposed load intensity modeling formalism.

  @article{KiHeKo-TAAS17-ModelExtractLoadProfiles, abstract = {Today's system developers and operators face the challenge of creating software systems that make efficient use of dynamically allocated resources under highly variable and dynamic load profiles, while at the same time delivering reliable performance. Autonomic controllers, e.g., an advanced auto-scaling mechanism in a cloud computing context, can benefit from an abstracted load model as knowledge to reconfigure on time and precisely. Existing workload characterization approaches have limited support to capture variations the inter-arrival times of incoming work units over time (i.e., a variable load profile). For example, industrial and scientific benchmarks support constant or stepwise increasing load, or inter-arrival times defined by statistical distributions or recorded traces. These options show shortcomings either in representative character of load variation patterns or in abstraction and flexibility of their format. In this article, we present the Descartes Load Intensity Model (DLIM) approach addressing these issues. DLIM provides a modeling formalism for describing load intensity variations over time. A DLIM instance is a compact formal description of a load intensity trace. DLIM-based tools provide features for benchmarking, performance and recorded load intensity trace analysis. As manually obtaining and maintaining DLIM instances becomes time consuming, we contribute three automated extraction methods and devised metrics for comparison and method selection. We discuss how these features are used to enhance system management approaches for adaptations during run-time, and how they are integrated into simulation contexts and enable benchmarking of elastic or adaptive behavior. We show that automatically extracted DLIM instances exhibit an average modeling error of 15.2\% over ten different real-world traces that cover between two weeks and seven months. These results underline DLIM model expressiveness. In terms of accuracy and processing speed, our proposed extraction methods for the descriptive models are comparable to existing time series decomposition methods. Additionally, we illustrate DLIM applicability by outlining approaches of workload modeling in systems engineering that employ or rely on our proposed load intensity modeling formalism.}, address = {New York, NY, USA}, author = {von Kistowski, J{\'o}akim and Herbst, Nikolas and Kounev, Samuel and Groenda, Henning and Stier, Christian and Lehrig, Sebastian}, journal = {ACM Transactions on Autonomous and Adaptive Systems (TAAS)}, month = {January}, number = 4, pages = {23:1--23:28}, publisher = {ACM}, title = {{Modeling and Extracting Load Intensity Profiles}}, volume = 11, year = 2017 }

  %0 Journal Article %1 KiHeKo-TAAS17-ModelExtractLoadProfiles %A von Kistowski, J{\'o}akim %A Herbst, Nikolas %A Kounev, Samuel %A Groenda, Henning %A Stier, Christian %A Lehrig, Sebastian %C New York, NY, USA %D 2017 %I ACM %J ACM Transactions on Autonomous and Adaptive Systems (TAAS) %N 4 %P 23:1--23:28 %T {Modeling and Extracting Load Intensity Profiles} %U http://doi.acm.org/10.1145/3019596 %V 11 %X Today's system developers and operators face the challenge of creating software systems that make efficient use of dynamically allocated resources under highly variable and dynamic load profiles, while at the same time delivering reliable performance. Autonomic controllers, e.g., an advanced auto-scaling mechanism in a cloud computing context, can benefit from an abstracted load model as knowledge to reconfigure on time and precisely. Existing workload characterization approaches have limited support to capture variations the inter-arrival times of incoming work units over time (i.e., a variable load profile). For example, industrial and scientific benchmarks support constant or stepwise increasing load, or inter-arrival times defined by statistical distributions or recorded traces. These options show shortcomings either in representative character of load variation patterns or in abstraction and flexibility of their format. In this article, we present the Descartes Load Intensity Model (DLIM) approach addressing these issues. DLIM provides a modeling formalism for describing load intensity variations over time. A DLIM instance is a compact formal description of a load intensity trace. DLIM-based tools provide features for benchmarking, performance and recorded load intensity trace analysis. As manually obtaining and maintaining DLIM instances becomes time consuming, we contribute three automated extraction methods and devised metrics for comparison and method selection. We discuss how these features are used to enhance system management approaches for adaptations during run-time, and how they are integrated into simulation contexts and enable benchmarking of elastic or adaptive behavior. We show that automatically extracted DLIM instances exhibit an average modeling error of 15.2\% over ten different real-world traces that cover between two weeks and seven months. These results underline DLIM model expressiveness. In terms of accuracy and processing speed, our proposed extraction methods for the descriptive models are comparable to existing time series decomposition methods. Additionally, we illustrate DLIM applicability by outlining approaches of workload modeling in systems engineering that employ or rely on our proposed load intensity modeling formalism.
• 
  {Model-Based Self-Aware Performance and Resource Management Using the Descartes Modeling Language}. Huber, Nikolaus; Brosig, Fabian; Spinner, Simon; Kounev, Samuel; B{\"a}hr, Manuel; in IEEE Transactions on Software Engineering (TSE) 43(5) 432–452. IEEE Computer Society.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Modern IT systems have increasingly distributed and dynamic architectures providing flexibility to adapt to changes in the environment and thus enabling higher resource efficiency. However, these benefits come at the cost of higher system complexity and dynamics. Thus, engineering systems that manage their end-to-end application performance and resource efficiency in an autonomic manner is a challenge. In this article, we present a holistic model-based approach for self-aware performance and resource management leveraging the Descartes Modeling Language (DML), an architecture-level modeling language for online performance and resource management. We propose a novel online performance prediction process that dynamically tailors the model solving depending on the requirements regarding accuracy and overhead. Using these prediction capabilities, we implement a generic model-based control loop for proactive system adaptation. We evaluate our model-based approach in the context of two representative case studies showing that with the proposed methods, significant resource efficiency gains can be achieved while maintaining performance requirements. These results represent the first end-to-end validation of our approach, demonstrating its potential for self-aware performance and resource management in the context of modern IT systems and infrastructures.

  @article{HuBrSpKoBa2017-TSE-DML, abstract = {Modern IT systems have increasingly distributed and dynamic architectures providing flexibility to adapt to changes in the environment and thus enabling higher resource efficiency. However, these benefits come at the cost of higher system complexity and dynamics. Thus, engineering systems that manage their end-to-end application performance and resource efficiency in an autonomic manner is a challenge. In this article, we present a holistic model-based approach for self-aware performance and resource management leveraging the Descartes Modeling Language (DML), an architecture-level modeling language for online performance and resource management. We propose a novel online performance prediction process that dynamically tailors the model solving depending on the requirements regarding accuracy and overhead. Using these prediction capabilities, we implement a generic model-based control loop for proactive system adaptation. We evaluate our model-based approach in the context of two representative case studies showing that with the proposed methods, significant resource efficiency gains can be achieved while maintaining performance requirements. These results represent the first end-to-end validation of our approach, demonstrating its potential for self-aware performance and resource management in the context of modern IT systems and infrastructures.}, author = {Huber, Nikolaus and Brosig, Fabian and Spinner, Simon and Kounev, Samuel and B{\"a}hr, Manuel}, journal = {IEEE Transactions on Software Engineering (TSE)}, number = 5, pages = {432--452}, publisher = {IEEE Computer Society}, title = {{Model-Based Self-Aware Performance and Resource Management Using the Descartes Modeling Language}}, volume = 43, year = 2017 }

  %0 Journal Article %1 HuBrSpKoBa2017-TSE-DML %A Huber, Nikolaus %A Brosig, Fabian %A Spinner, Simon %A Kounev, Samuel %A B{\"a}hr, Manuel %D 2017 %I IEEE Computer Society %J IEEE Transactions on Software Engineering (TSE) %N 5 %P 432--452 %T {Model-Based Self-Aware Performance and Resource Management Using the Descartes Modeling Language} %U http://dx.doi.org/10.1109/TSE.2016.2613863 %V 43 %X Modern IT systems have increasingly distributed and dynamic architectures providing flexibility to adapt to changes in the environment and thus enabling higher resource efficiency. However, these benefits come at the cost of higher system complexity and dynamics. Thus, engineering systems that manage their end-to-end application performance and resource efficiency in an autonomic manner is a challenge. In this article, we present a holistic model-based approach for self-aware performance and resource management leveraging the Descartes Modeling Language (DML), an architecture-level modeling language for online performance and resource management. We propose a novel online performance prediction process that dynamically tailors the model solving depending on the requirements regarding accuracy and overhead. Using these prediction capabilities, we implement a generic model-based control loop for proactive system adaptation. We evaluate our model-based approach in the context of two representative case studies showing that with the proposed methods, significant resource efficiency gains can be achieved while maintaining performance requirements. These results represent the first end-to-end validation of our approach, demonstrating its potential for self-aware performance and resource management in the context of modern IT systems and infrastructures.

• 
  {Enabling Fluid Analysis for Queueing Petri Nets via Model Transformation}. M{\"u}ller, Christoph; Rygielski, Piotr; Spinner, Simon; Kounev, Samuel; in Electronic Notes in Theoretical Computer Science 327 71–91. Elsevier.

  The 8th International Workshop on Practical Application of Stochastic Modeling, PASM 2016
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Abstract Due to the growing size of modern \{IT\} systems, their performance analysis becomes an even more challenging task. Existing simulators are unable to analyze the behavior of large systems in a reasonable time, whereas analytical methods suffer from the state space explosion problem. Fluid analysis techniques can be used to approximate the solution of high-order Markov chain models enabling time efficient analysis of large performance models. In this paper, we describe a model-to-model transformation from queueing Petri nets (QPN) into layered queueing networks (LQN). Obtained \{LQN\} models can benefit from three existing solvers: LINE, LQNS, LQSIM. \{LINE\} internally utilize fluid limits approximation to speed up the solving process for large models. We present the incentives for developing the automated model-to-model transformation and present a systematic approach that we followed in its design. We demonstrate the transformations using representative examples. Finally, we evaluate and compare the performance predictions of existing analytical, simulation and fluid analysis solvers. We analyze solvers' limitations, solving time, and memory consumption.

  @article{MuRySpKo2016-PASM-QPN-LQN-Transformation, abstract = {Abstract Due to the growing size of modern \{IT\} systems, their performance analysis becomes an even more challenging task. Existing simulators are unable to analyze the behavior of large systems in a reasonable time, whereas analytical methods suffer from the state space explosion problem. Fluid analysis techniques can be used to approximate the solution of high-order Markov chain models enabling time efficient analysis of large performance models. In this paper, we describe a model-to-model transformation from queueing Petri nets (QPN) into layered queueing networks (LQN). Obtained \{LQN\} models can benefit from three existing solvers: LINE, LQNS, LQSIM. \{LINE\} internally utilize fluid limits approximation to speed up the solving process for large models. We present the incentives for developing the automated model-to-model transformation and present a systematic approach that we followed in its design. We demonstrate the transformations using representative examples. Finally, we evaluate and compare the performance predictions of existing analytical, simulation and fluid analysis solvers. We analyze solvers' limitations, solving time, and memory consumption.}, author = {M{\"u}ller, Christoph and Rygielski, Piotr and Spinner, Simon and Kounev, Samuel}, journal = {Electronic Notes in Theoretical Computer Science}, note = {The 8th International Workshop on Practical Application of Stochastic Modeling, PASM 2016}, pages = {71--91}, publisher = {Elsevier}, title = {{Enabling Fluid Analysis for Queueing Petri Nets via Model Transformation}}, volume = 327, year = 2016 }

  %0 Journal Article %1 MuRySpKo2016-PASM-QPN-LQN-Transformation %A M{\"u}ller, Christoph %A Rygielski, Piotr %A Spinner, Simon %A Kounev, Samuel %D 2016 %I Elsevier %J Electronic Notes in Theoretical Computer Science %P 71--91 %T {Enabling Fluid Analysis for Queueing Petri Nets via Model Transformation} %U http://www.sciencedirect.com/science/article/pii/S157106611630069X %V 327 %X Abstract Due to the growing size of modern \{IT\} systems, their performance analysis becomes an even more challenging task. Existing simulators are unable to analyze the behavior of large systems in a reasonable time, whereas analytical methods suffer from the state space explosion problem. Fluid analysis techniques can be used to approximate the solution of high-order Markov chain models enabling time efficient analysis of large performance models. In this paper, we describe a model-to-model transformation from queueing Petri nets (QPN) into layered queueing networks (LQN). Obtained \{LQN\} models can benefit from three existing solvers: LINE, LQNS, LQSIM. \{LINE\} internally utilize fluid limits approximation to speed up the solving process for large models. We present the incentives for developing the automated model-to-model transformation and present a systematic approach that we followed in its design. We demonstrate the transformations using representative examples. Finally, we evaluate and compare the performance predictions of existing analytical, simulation and fluid analysis solvers. We analyze solvers' limitations, solving time, and memory consumption.
• 
  {A Model-Based Approach to Designing Self-Aware IT Systems and Infrastructures}. Kounev, Samuel; Huber, Nikolaus; Brosig, Fabian; Zhu, Xiaoyun; in IEEE Computer 49(7) 53–61. IEEE.
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  @article{KoHuBrZh2016-IEEEComputer-Descartes, author = {Kounev, Samuel and Huber, Nikolaus and Brosig, Fabian and Zhu, Xiaoyun}, journal = {IEEE Computer}, month = {July}, number = 7, pages = {53-61}, publisher = {IEEE}, title = {{A Model-Based Approach to Designing Self-Aware IT Systems and Infrastructures}}, volume = 49, year = 2016 }

  %0 Journal Article %1 KoHuBrZh2016-IEEEComputer-Descartes %A Kounev, Samuel %A Huber, Nikolaus %A Brosig, Fabian %A Zhu, Xiaoyun %D 2016 %I IEEE %J IEEE Computer %N 7 %P 53-61 %T {A Model-Based Approach to Designing Self-Aware IT Systems and Infrastructures} %U http://dx.doi.org/10.1109/MC.2016.198 %V 49

• 
  {Evaluating Approaches to Resource Demand Estimation}. Spinner, Simon; Casale, Giuliano; Brosig, Fabian; Kounev, Samuel; in Performance Evaluation 92 51–71. Elsevier B.V.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Resource demands are a key parameter of stochastic performance models that needs to be determined when performing a quantitative performance analysis of a system. However, the direct measurement of resource demands is not feasible in most realistic systems. Therefore, statistical approaches that estimate resource demands based on coarse-grained monitoring data (e.g., CPU utilization, and response times) have been proposed in the literature. These approaches have different assumptions and characteristics that need to be considered when estimating resource demands. This paper surveys the state-of-the-art in resource demand estimation and proposes a classification scheme for estimation approaches. Furthermore, it contains an experimental evaluation comparing the impact of different factors (monitoring window size, number of workload classes, load level, collinearity, and model mismatch) on the estimation accuracy of seven different approaches. The classification scheme and the experimental comparison helps performance engineers to select an approach to resource demand estimation that fulfills the requirements of a given analysis scenario.

  @article{SpCaBrKo2015-PEVA-ResDemEstSurvey, abstract = {Resource demands are a key parameter of stochastic performance models that needs to be determined when performing a quantitative performance analysis of a system. However, the direct measurement of resource demands is not feasible in most realistic systems. Therefore, statistical approaches that estimate resource demands based on coarse-grained monitoring data (e.g., CPU utilization, and response times) have been proposed in the literature. These approaches have different assumptions and characteristics that need to be considered when estimating resource demands. This paper surveys the state-of-the-art in resource demand estimation and proposes a classification scheme for estimation approaches. Furthermore, it contains an experimental evaluation comparing the impact of different factors (monitoring window size, number of workload classes, load level, collinearity, and model mismatch) on the estimation accuracy of seven different approaches. The classification scheme and the experimental comparison helps performance engineers to select an approach to resource demand estimation that fulfills the requirements of a given analysis scenario.}, author = {Spinner, Simon and Casale, Giuliano and Brosig, Fabian and Kounev, Samuel}, journal = {Performance Evaluation}, month = {October}, pages = {51--71}, publisher = {Elsevier B.V.}, title = {{Evaluating Approaches to Resource Demand Estimation}}, volume = 92, year = 2015 }

  %0 Journal Article %1 SpCaBrKo2015-PEVA-ResDemEstSurvey %A Spinner, Simon %A Casale, Giuliano %A Brosig, Fabian %A Kounev, Samuel %D 2015 %I Elsevier B.V. %J Performance Evaluation %P 51--71 %T {Evaluating Approaches to Resource Demand Estimation} %U http://www.sciencedirect.com/science/article/pii/S0166531615000711 %V 92 %X Resource demands are a key parameter of stochastic performance models that needs to be determined when performing a quantitative performance analysis of a system. However, the direct measurement of resource demands is not feasible in most realistic systems. Therefore, statistical approaches that estimate resource demands based on coarse-grained monitoring data (e.g., CPU utilization, and response times) have been proposed in the literature. These approaches have different assumptions and characteristics that need to be considered when estimating resource demands. This paper surveys the state-of-the-art in resource demand estimation and proposes a classification scheme for estimation approaches. Furthermore, it contains an experimental evaluation comparing the impact of different factors (monitoring window size, number of workload classes, load level, collinearity, and model mismatch) on the estimation accuracy of seven different approaches. The classification scheme and the experimental comparison helps performance engineers to select an approach to resource demand estimation that fulfills the requirements of a given analysis scenario.
• 
  {Evaluating Computer Intrusion Detection Systems: A Survey of Common Practices}. Milenkoski, Aleksandar; Vieira, Marco; Kounev, Samuel; Avritzer, Alberto; Payne, Bryan D.; in {ACM Computing Surveys} 48(1) 12:1–12:41. ACM, New York, NY, USA.

  {5-year Impact Factor (2014): 5.949}
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  {The evaluation of computer intrusion detection systems (which we refer to as intrusion detection systems) is an active research area. In this paper, we survey and systematize common practices in the area of evaluation of intrusion detection systems. For this purpose, we define a design space structured into three parts: workload, metrics, and measurement methodology. We then provide an overview of the common practices in evaluation of intrusion detection systems by surveying evaluation approaches and methods related to each part of the design space. Finally, we discuss open issues and challenges focusing on evaluation methodologies for novel intrusion detection systems.}

  @article{MiViKoAvPa2015-CSUR-IDSEval, abstract = {{The evaluation of computer intrusion detection systems (which we refer to as intrusion detection systems) is an active research area. In this paper, we survey and systematize common practices in the area of evaluation of intrusion detection systems. For this purpose, we define a design space structured into three parts: workload, metrics, and measurement methodology. We then provide an overview of the common practices in evaluation of intrusion detection systems by surveying evaluation approaches and methods related to each part of the design space. Finally, we discuss open issues and challenges focusing on evaluation methodologies for novel intrusion detection systems.}}, address = {New York, NY, USA}, author = {Milenkoski, Aleksandar and Vieira, Marco and Kounev, Samuel and Avritzer, Alberto and Payne, Bryan D.}, journal = {{ACM Computing Surveys}}, month = {{September}}, note = {{<b>5-year Impact Factor (2014): 5.949</b>}}, number = 1, pages = {12:1--12:41}, publisher = {ACM}, title = {{Evaluating Computer Intrusion Detection Systems: A Survey of Common Practices}}, volume = 48, year = 2015 }

  %0 Journal Article %1 MiViKoAvPa2015-CSUR-IDSEval %A Milenkoski, Aleksandar %A Vieira, Marco %A Kounev, Samuel %A Avritzer, Alberto %A Payne, Bryan D. %C New York, NY, USA %D 2015 %I ACM %J {ACM Computing Surveys} %N 1 %P 12:1--12:41 %T {Evaluating Computer Intrusion Detection Systems: A Survey of Common Practices} %U http://dl.acm.org/authorize?N06203 %V 48 %X {The evaluation of computer intrusion detection systems (which we refer to as intrusion detection systems) is an active research area. In this paper, we survey and systematize common practices in the area of evaluation of intrusion detection systems. For this purpose, we define a design space structured into three parts: workload, metrics, and measurement methodology. We then provide an overview of the common practices in evaluation of intrusion detection systems by surveying evaluation approaches and methods related to each part of the design space. Finally, we discuss open issues and challenges focusing on evaluation methodologies for novel intrusion detection systems.}
• 
  {Quantitative Evaluation of Model-Driven Performance Analysis and Simulation of Component-based Architectures}. Brosig, Fabian; Meier, Philipp; Becker, Steffen; Koziolek, Anne; Koziolek, Heiko; Kounev, Samuel; in IEEE Transactions on Software Engineering (TSE) 41(2) 157–175. IEEE.
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  @article{BrMeBeKoKoKo2014-TSE-ModelSolutionTradeOffs, author = {Brosig, Fabian and Meier, Philipp and Becker, Steffen and Koziolek, Anne and Koziolek, Heiko and Kounev, Samuel}, journal = {IEEE Transactions on Software Engineering (TSE)}, month = {{February}}, number = 2, pages = {157--175}, publisher = {IEEE}, title = {{Quantitative Evaluation of Model-Driven Performance Analysis and Simulation of Component-based Architectures}}, volume = 41, year = 2015 }

  %0 Journal Article %1 BrMeBeKoKoKo2014-TSE-ModelSolutionTradeOffs %A Brosig, Fabian %A Meier, Philipp %A Becker, Steffen %A Koziolek, Anne %A Koziolek, Heiko %A Kounev, Samuel %D 2015 %I IEEE %J IEEE Transactions on Software Engineering (TSE) %N 2 %P 157--175 %T {Quantitative Evaluation of Model-Driven Performance Analysis and Simulation of Component-based Architectures} %U http://dx.doi.org/10.1109/TSE.2014.2362755 %V 41
• 
  {Parallel Simulation of Queueing Petri Nets}. Walter, J{\"u}rgen; Spinner, Simon; Kounev, Samuel; in EAI Endorsed Transactions on Industrial Networks and Intelligent Systems 16(8) ACM.
  • [ BibTeX ]
  • [ EndNote ]
  @article{WaSpKo2016-EAI-parallelSimulation, author = {Walter, J{\"u}rgen and Spinner, Simon and Kounev, Samuel}, journal = {EAI Endorsed Transactions on Industrial Networks and Intelligent Systems}, month = 8, number = 8, publisher = {ACM}, title = {{Parallel Simulation of Queueing Petri Nets}}, volume = 16, year = 2015 }

  %0 Journal Article %1 WaSpKo2016-EAI-parallelSimulation %A Walter, J{\"u}rgen %A Spinner, Simon %A Kounev, Samuel %D 2015 %I ACM %J EAI Endorsed Transactions on Industrial Networks and Intelligent Systems %N 8 %T {Parallel Simulation of Queueing Petri Nets} %V 16

• 
  {Self-Adaptive Workload Classification and Forecasting for Proactive Resource Provisioning}. Herbst, Nikolas Roman; Huber, Nikolaus; Kounev, Samuel; Amrehn, Erich; in Concurrency and Computation - Practice and Experience, John Wiley and Sons, Ltd. 26(12) 2053–2078.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  As modern enterprise software systems become increasingly dynamic, workload forecasting techniques are gaining in importance as a foundation for online capacity planning and resource management. Time series analysis covers a broad spectrum of methods to calculate workload forecasts based on history monitoring data. Related work in the field of workload forecasting mostly concentrates on evaluating specific methods and their individual optimisation potential or on predicting Quality-of-Service (QoS) metrics directly. As a basis, we present a survey on established forecasting methods of the time series analysis concerning their benefits and drawbacks and group them according to their computational overheads. In this paper, we propose a novel self-adaptive approach that selects suitable forecasting methods for a given context based on a decision tree and direct feedback cycles together with a corresponding implementation. The user needs to provide only his general forecasting objectives. In several experiments and case studies based on real world workload traces, we show that our implementation of the approach provides continuous and reliable forecast results at run-time. The results of this extensive evaluation show that the relative error of the individual forecast points is significantly reduced compared to statically applied forecasting methods, e.g. in an exemplary scenario on average by 37%. In a case study, between 55% and 75% of the violations of a given service level agreement can be prevented by applying proactive resource provisioning based on the forecast results of our implementation.

  @article{HuKoAm2013-CCPE-WorkloadClassificationAndForecasting, abstract = {As modern enterprise software systems become increasingly dynamic, workload forecasting techniques are gaining in importance as a foundation for online capacity planning and resource management. Time series analysis covers a broad spectrum of methods to calculate workload forecasts based on history monitoring data. Related work in the field of workload forecasting mostly concentrates on evaluating specific methods and their individual optimisation potential or on predicting Quality-of-Service (QoS) metrics directly. As a basis, we present a survey on established forecasting methods of the time series analysis concerning their benefits and drawbacks and group them according to their computational overheads. In this paper, we propose a novel self-adaptive approach that selects suitable forecasting methods for a given context based on a decision tree and direct feedback cycles together with a corresponding implementation. The user needs to provide only his general forecasting objectives. In several experiments and case studies based on real world workload traces, we show that our implementation of the approach provides continuous and reliable forecast results at run-time. The results of this extensive evaluation show that the relative error of the individual forecast points is significantly reduced compared to statically applied forecasting methods, e.g. in an exemplary scenario on average by 37%. In a case study, between 55% and 75% of the violations of a given service level agreement can be prevented by applying proactive resource provisioning based on the forecast results of our implementation.}, author = {Herbst, Nikolas Roman and Huber, Nikolaus and Kounev, Samuel and Amrehn, Erich}, journal = {Concurrency and Computation - Practice and Experience, John Wiley and Sons, Ltd.}, month = {March}, number = 12, pages = {2053--2078}, title = {{Self-Adaptive Workload Classification and Forecasting for Proactive Resource Provisioning}}, volume = 26, year = 2014 }

  %0 Journal Article %1 HuKoAm2013-CCPE-WorkloadClassificationAndForecasting %A Herbst, Nikolas Roman %A Huber, Nikolaus %A Kounev, Samuel %A Amrehn, Erich %D 2014 %J Concurrency and Computation - Practice and Experience, John Wiley and Sons, Ltd. %N 12 %P 2053--2078 %T {Self-Adaptive Workload Classification and Forecasting for Proactive Resource Provisioning} %U http://dx.doi.org/10.1002/cpe.3224 %V 26 %X As modern enterprise software systems become increasingly dynamic, workload forecasting techniques are gaining in importance as a foundation for online capacity planning and resource management. Time series analysis covers a broad spectrum of methods to calculate workload forecasts based on history monitoring data. Related work in the field of workload forecasting mostly concentrates on evaluating specific methods and their individual optimisation potential or on predicting Quality-of-Service (QoS) metrics directly. As a basis, we present a survey on established forecasting methods of the time series analysis concerning their benefits and drawbacks and group them according to their computational overheads. In this paper, we propose a novel self-adaptive approach that selects suitable forecasting methods for a given context based on a decision tree and direct feedback cycles together with a corresponding implementation. The user needs to provide only his general forecasting objectives. In several experiments and case studies based on real world workload traces, we show that our implementation of the approach provides continuous and reliable forecast results at run-time. The results of this extensive evaluation show that the relative error of the individual forecast points is significantly reduced compared to statically applied forecasting methods, e.g. in an exemplary scenario on average by 37%. In a case study, between 55% and 75% of the violations of a given service level agreement can be prevented by applying proactive resource provisioning based on the forecast results of our implementation.
• 
  {Architecture-Level Software Performance Abstractions for Online Performance Prediction}. Brosig, Fabian; Huber, Nikolaus; Kounev, Samuel; in Elsevier Science of Computer Programming Journal (SciCo) Vol. 90, Part B 71–92. Elsevier.
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  @article{BrHuKo2013-SciCo-SoftwarePerformanceAbstractions, author = {Brosig, Fabian and Huber, Nikolaus and Kounev, Samuel}, journal = {Elsevier Science of Computer Programming Journal (SciCo)}, month = {September}, pages = {71--92}, publisher = {Elsevier}, title = {{Architecture-Level Software Performance Abstractions for Online Performance Prediction}}, volume = {Vol. 90, Part B}, year = 2014 }

  %0 Journal Article %1 BrHuKo2013-SciCo-SoftwarePerformanceAbstractions %A Brosig, Fabian %A Huber, Nikolaus %A Kounev, Samuel %D 2014 %I Elsevier %J Elsevier Science of Computer Programming Journal (SciCo) %P 71--92 %T {Architecture-Level Software Performance Abstractions for Online Performance Prediction} %U http://authors.elsevier.com/sd/article/S0167642313001421 %V Vol. 90, Part B
• 
  {Metrics and Techniques for Quantifying Performance Isolation in Cloud Environments}. Krebs, Rouven; Momm, Christof; Kounev, Samuel; in Elsevier Science of Computer Programming Journal (SciCo) Vol. 90, Part B 116–134. Elsevier B.V.
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @article{KrMoKo2013-SciCo-MetricsAndTechniquesForPerformanceIsolation, author = {Krebs, Rouven and Momm, Christof and Kounev, Samuel}, journal = {Elsevier Science of Computer Programming Journal (SciCo)}, month = {February}, pages = {116--134}, publisher = {Elsevier B.V.}, title = {{Metrics and Techniques for Quantifying Performance Isolation in Cloud Environments}}, volume = {Vol. 90, Part B}, year = 2014 }

  %0 Journal Article %1 KrMoKo2013-SciCo-MetricsAndTechniquesForPerformanceIsolation %A Krebs, Rouven %A Momm, Christof %A Kounev, Samuel %D 2014 %I Elsevier B.V. %J Elsevier Science of Computer Programming Journal (SciCo) %P 116--134 %T {Metrics and Techniques for Quantifying Performance Isolation in Cloud Environments} %V Vol. 90, Part B
• 
  {Modeling Run-Time Adaptation at the System Architecture Level in Dynamic Service-Oriented Environments}. Huber, Nikolaus; van Hoorn, Andr{\’e}; Koziolek, Anne; Brosig, Fabian; Kounev, Samuel; in Service Oriented Computing and Applications Journal (SOCA) 8(1) 73–89. Springer-Verlag.
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @article{HuHoKoBrKo2014-SOCA-ModelingRuntimeAdaptation, author = {Huber, Nikolaus and van Hoorn, Andr{\'e} and Koziolek, Anne and Brosig, Fabian and Kounev, Samuel}, journal = {Service Oriented Computing and Applications Journal (SOCA)}, month = {{March}}, number = 1, pages = {73--89}, publisher = {Springer-Verlag}, title = {{Modeling Run-Time Adaptation at the System Architecture Level in Dynamic Service-Oriented Environments}}, volume = 8, year = 2014 }

  %0 Journal Article %1 HuHoKoBrKo2014-SOCA-ModelingRuntimeAdaptation %A Huber, Nikolaus %A van Hoorn, Andr{\'e} %A Koziolek, Anne %A Brosig, Fabian %A Kounev, Samuel %D 2014 %I Springer-Verlag %J Service Oriented Computing and Applications Journal (SOCA) %N 1 %P 73--89 %T {Modeling Run-Time Adaptation at the System Architecture Level in Dynamic Service-Oriented Environments} %V 8

• 
  Performance modeling and analysis of message-oriented event-driven systems. Sachs, Kai; Kounev, Samuel; Buchmann, Alejandro; in Journal of Software and Systems Modeling 12(4) 705–729. Springer-Verlag.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Message-oriented event-driven systems are becoming increasingly ubiquitous in many industry domains including telecommunications, transportation and supply chain management. Applications in these areas typically have stringent requirements for performance and scalability. To guarantee adequate quality-of-service, systems must be subjected to a rigorous performance and scalability analysis before they are put into production. In this paper, we present a comprehensive modeling methodology for message-oriented event-driven systems in the context of a case study of a representative application in the supply chain management domain. The methodology, which is based on queueing Petri nets, provides a basis for performance analysis and capacity planning. We study a deployment of the SPECjms2007 standard benchmark on a leading commercial middleware platform. A detailed system model is built in a step-by-step fashion and then used to predict the system performance under various workload and configuration scenarios. After the case study, we present a set of generic performance modeling patterns that can be used as building blocks when modeling message-oriented event-driven systems. The results demonstrate the effectiveness, practicality and accuracy of the proposed modeling and prediction approach.

  @article{SaKoBu2011-SoSyM-PerfModMoEdSys, abstract = {Message-oriented event-driven systems are becoming increasingly ubiquitous in many industry domains including telecommunications, transportation and supply chain management. Applications in these areas typically have stringent requirements for performance and scalability. To guarantee adequate quality-of-service, systems must be subjected to a rigorous performance and scalability analysis before they are put into production. In this paper, we present a comprehensive modeling methodology for message-oriented event-driven systems in the context of a case study of a representative application in the supply chain management domain. The methodology, which is based on queueing Petri nets, provides a basis for performance analysis and capacity planning. We study a deployment of the SPECjms2007 standard benchmark on a leading commercial middleware platform. A detailed system model is built in a step-by-step fashion and then used to predict the system performance under various workload and configuration scenarios. After the case study, we present a set of generic performance modeling patterns that can be used as building blocks when modeling message-oriented event-driven systems. The results demonstrate the effectiveness, practicality and accuracy of the proposed modeling and prediction approach.}, author = {Sachs, Kai and Kounev, Samuel and Buchmann, Alejandro}, journal = {Journal of Software and Systems Modeling}, number = 4, pages = {705--729}, publisher = {Springer-Verlag}, title = {Performance modeling and analysis of message-oriented event-driven systems}, volume = 12, year = 2013 }

  %0 Journal Article %1 SaKoBu2011-SoSyM-PerfModMoEdSys %A Sachs, Kai %A Kounev, Samuel %A Buchmann, Alejandro %D 2013 %I Springer-Verlag %J Journal of Software and Systems Modeling %N 4 %P 705--729 %T Performance modeling and analysis of message-oriented event-driven systems %V 12 %X Message-oriented event-driven systems are becoming increasingly ubiquitous in many industry domains including telecommunications, transportation and supply chain management. Applications in these areas typically have stringent requirements for performance and scalability. To guarantee adequate quality-of-service, systems must be subjected to a rigorous performance and scalability analysis before they are put into production. In this paper, we present a comprehensive modeling methodology for message-oriented event-driven systems in the context of a case study of a representative application in the supply chain management domain. The methodology, which is based on queueing Petri nets, provides a basis for performance analysis and capacity planning. We study a deployment of the SPECjms2007 standard benchmark on a leading commercial middleware platform. A detailed system model is built in a step-by-step fashion and then used to predict the system performance under various workload and configuration scenarios. After the case study, we present a set of generic performance modeling patterns that can be used as building blocks when modeling message-oriented event-driven systems. The results demonstrate the effectiveness, practicality and accuracy of the proposed modeling and prediction approach.
• 
  {Network Virtualization for QoS-Aware Resource Management in Cloud Data Centers: A Survey}. Rygielski, Piotr; Kounev, Samuel; in PIK --- Praxis der Informationsverarbeitung und Kommunikation 36(1) 55–64. de Gruyter.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  The increasing popularity of Cloud Computing is leading to the emergence of large virtualized data centers hosting increasingly complex and dynamic IT systems and services. Over the past decade, the efficient sharing of computational resources through virtualization has been subject to intensive research, while network management in cloud data centers has received less attention. A variety of network-intensive applications require QoS (Quality-of-Service) provisioning, performance isolation and support for flexible and efficient migration of virtual machines. In this paper, we survey existing network virtualization approaches and evaluate the extent to which they can be used as a basis for realizing the mentioned requirements in a cloud data center. More specifically, we identify generic network virtualization techniques, characterize them according to their features related to QoS management and performance isolation, and show how they can be composed together and used as building blocks for complex network virtualization solutions. We then present an overview of selected representative cloud platforms and show how they leverage the generic techniques as a basis for network resource management. Finally, we outline open issues and research challenges in the area of performance modeling and proactive resource management of virtualized data center infrastructures.

  @article{RyKo2013, abstract = {The increasing popularity of Cloud Computing is leading to the emergence of large virtualized data centers hosting increasingly complex and dynamic IT systems and services. Over the past decade, the efficient sharing of computational resources through virtualization has been subject to intensive research, while network management in cloud data centers has received less attention. A variety of network-intensive applications require QoS (Quality-of-Service) provisioning, performance isolation and support for flexible and efficient migration of virtual machines. In this paper, we survey existing network virtualization approaches and evaluate the extent to which they can be used as a basis for realizing the mentioned requirements in a cloud data center. More specifically, we identify generic network virtualization techniques, characterize them according to their features related to QoS management and performance isolation, and show how they can be composed together and used as building blocks for complex network virtualization solutions. We then present an overview of selected representative cloud platforms and show how they leverage the generic techniques as a basis for network resource management. Finally, we outline open issues and research challenges in the area of performance modeling and proactive resource management of virtualized data center infrastructures.}, author = {Rygielski, Piotr and Kounev, Samuel}, journal = {PIK --- Praxis der Informationsverarbeitung und Kommunikation}, month = {February}, number = 1, pages = {55--64}, publisher = {de Gruyter}, title = {{Network Virtualization for QoS-Aware Resource Management in Cloud Data Centers: A Survey}}, volume = 36, year = 2013 }

  %0 Journal Article %1 RyKo2013 %A Rygielski, Piotr %A Kounev, Samuel %D 2013 %I de Gruyter %J PIK --- Praxis der Informationsverarbeitung und Kommunikation %N 1 %P 55--64 %T {Network Virtualization for QoS-Aware Resource Management in Cloud Data Centers: A Survey} %U http://www.degruyter.com/view/j/piko-2013-36-issue-1/pik-2012-0136/pik-2012-0136.xml?format=INT %V 36 %X The increasing popularity of Cloud Computing is leading to the emergence of large virtualized data centers hosting increasingly complex and dynamic IT systems and services. Over the past decade, the efficient sharing of computational resources through virtualization has been subject to intensive research, while network management in cloud data centers has received less attention. A variety of network-intensive applications require QoS (Quality-of-Service) provisioning, performance isolation and support for flexible and efficient migration of virtual machines. In this paper, we survey existing network virtualization approaches and evaluate the extent to which they can be used as a basis for realizing the mentioned requirements in a cloud data center. More specifically, we identify generic network virtualization techniques, characterize them according to their features related to QoS management and performance isolation, and show how they can be composed together and used as building blocks for complex network virtualization solutions. We then present an overview of selected representative cloud platforms and show how they leverage the generic techniques as a basis for network resource management. Finally, we outline open issues and research challenges in the area of performance modeling and proactive resource management of virtualized data center infrastructures.
• 
  Modeling Event-based Communication in Component-based Software Architectures for Performance Predictions. Rathfelder, Christoph; Klatt, Benjamin; Sachs, Kai; Kounev, Samuel; in Journal of Software and Systems Modeling (SoSyM) 1–27. Springer Verlag.
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  @article{rathfelder2013a, author = {Rathfelder, Christoph and Klatt, Benjamin and Sachs, Kai and Kounev, Samuel}, journal = {Journal of Software and Systems Modeling (SoSyM)}, month = {March}, pages = {1--27}, publisher = {Springer Verlag}, title = {Modeling Event-based Communication in Component-based Software Architectures for Performance Predictions}, year = 2013 }

  %0 Journal Article %1 rathfelder2013a %A Rathfelder, Christoph %A Klatt, Benjamin %A Sachs, Kai %A Kounev, Samuel %D 2013 %I Springer Verlag %J Journal of Software and Systems Modeling (SoSyM) %P 1--27 %T Modeling Event-based Communication in Component-based Software Architectures for Performance Predictions %U http://dx.doi.org/10.1007/s10270-013-0316-x
• 
  {Modeling of Event-based Communication in Component-based Architectures: State-of-the-Art and Future Directions}. Kounev, Samuel; Rathfelder, Christoph; Klatt, Benjamin; in {Electronic Notes in Theoretical Computer Science (ENTCS)} 295 3–9. Elsevier Science Publishers B. V., Amsterdam, The Netherlands.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Event-based communication is used in different domains including telecommunications, transportation, and business information systems to build scalable distributed systems. Such systems typically have stringent requirements for performance and scalability as they provide business and mission critical services. While the use of event-based communication enables loosely-coupled interactions between components and leads to improved system scalability, it makes it much harder for developers to estimate the system's behavior and performance under load due to the decoupling of components and control flow. We present an overview on our approach enabling the modeling and performance prediction of event-based system at the architecture level. Applying a model-to-model transformation, our approach integrates platform-specific performance influences of the underlying middleware while enabling the use of different existing analytical and simulation-based prediction techniques. The results of two real world case studies demonstrate the effectiveness, practicability and accuracy of the proposed modeling and prediction approach.

  @article{KoRaKl2012-FESCA-Keynote, abstract = {Event-based communication is used in different domains including telecommunications, transportation, and business information systems to build scalable distributed systems. Such systems typically have stringent requirements for performance and scalability as they provide business and mission critical services. While the use of event-based communication enables loosely-coupled interactions between components and leads to improved system scalability, it makes it much harder for developers to estimate the system's behavior and performance under load due to the decoupling of components and control flow. We present an overview on our approach enabling the modeling and performance prediction of event-based system at the architecture level. Applying a model-to-model transformation, our approach integrates platform-specific performance influences of the underlying middleware while enabling the use of different existing analytical and simulation-based prediction techniques. The results of two real world case studies demonstrate the effectiveness, practicability and accuracy of the proposed modeling and prediction approach.}, address = {Amsterdam, The Netherlands}, author = {Kounev, Samuel and Rathfelder, Christoph and Klatt, Benjamin}, journal = {{Electronic Notes in Theoretical Computer Science (ENTCS)}}, month = {may}, pages = {3--9}, publisher = {Elsevier Science Publishers B. V.}, title = {{Modeling of Event-based Communication in Component-based Architectures: State-of-the-Art and Future Directions}}, volume = 295, year = 2013 }

  %0 Journal Article %1 KoRaKl2012-FESCA-Keynote %A Kounev, Samuel %A Rathfelder, Christoph %A Klatt, Benjamin %C Amsterdam, The Netherlands %D 2013 %I Elsevier Science Publishers B. V. %J {Electronic Notes in Theoretical Computer Science (ENTCS)} %P 3--9 %T {Modeling of Event-based Communication in Component-based Architectures: State-of-the-Art and Future Directions} %U http://www.sciencedirect.com/science/article/pii/S1571066113000248 %V 295 %X Event-based communication is used in different domains including telecommunications, transportation, and business information systems to build scalable distributed systems. Such systems typically have stringent requirements for performance and scalability as they provide business and mission critical services. While the use of event-based communication enables loosely-coupled interactions between components and leads to improved system scalability, it makes it much harder for developers to estimate the system's behavior and performance under load due to the decoupling of components and control flow. We present an overview on our approach enabling the modeling and performance prediction of event-based system at the architecture level. Applying a model-to-model transformation, our approach integrates platform-specific performance influences of the underlying middleware while enabling the use of different existing analytical and simulation-based prediction techniques. The results of two real world case studies demonstrate the effectiveness, practicability and accuracy of the proposed modeling and prediction approach.

• 
  {Self-Aware Software and Systems Engineering: A Vision and Research Roadmap}. Kounev, Samuel; in {GI Softwaretechnik-Trends, 31(4), November 2011, ISSN 0720-8928}Karlsruhe, Germany.
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  @inproceedings{Ko2011-SE-DescartesResearch, address = {Karlsruhe, Germany}, author = {Kounev, Samuel}, booktitle = {{GI Softwaretechnik-Trends, 31(4), November 2011, ISSN 0720-8928}}, title = {{Self-Aware Software and Systems Engineering: A Vision and Research Roadmap}}, year = 2011 }

  %0 Conference Paper %1 Ko2011-SE-DescartesResearch %A Kounev, Samuel %B {GI Softwaretechnik-Trends, 31(4), November 2011, ISSN 0720-8928} %C Karlsruhe, Germany %D 2011 %T {Self-Aware Software and Systems Engineering: A Vision and Research Roadmap} %U http://pi.informatik.uni-siegen.de/stt/31_4/index.html

• 
  {Benchmarking and Performance Modeling of Event-Based Systems}. Kounev, Samuel; Sachs, Kai; in it - Information Technology 51(5) 262–269. Oldenbourg Wissenschaftsverlag, Munich, Germany.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  Event-based systems are used increasingly often to build loosely-coupled distributed applications. With their growing popularity and gradual adoption in mission critical areas, the need for novel techniques for benchmarking and performance modeling of event-based systems is increasing. In this article, we provide an overview of the state-of-the-art in this area considering both centralized systems based on message-oriented middleware as well as large-scale distributed publish/subscribe systems. We consider a number of specific techniques for benchmarking and performance modeling, discuss their advantages and disadvantages, and provide references for further information. The techniques we review help to ensure that systems are designed and sized to meet their quality-of-service requirements.

  @article{KoSa2009-DEBSPerformance, abstract = {Event-based systems are used increasingly often to build loosely-coupled distributed applications. With their growing popularity and gradual adoption in mission critical areas, the need for novel techniques for benchmarking and performance modeling of event-based systems is increasing. In this article, we provide an overview of the state-of-the-art in this area considering both centralized systems based on message-oriented middleware as well as large-scale distributed publish/subscribe systems. We consider a number of specific techniques for benchmarking and performance modeling, discuss their advantages and disadvantages, and provide references for further information. The techniques we review help to ensure that systems are designed and sized to meet their quality-of-service requirements.}, address = {Munich, Germany}, author = {Kounev, Samuel and Sachs, Kai}, journal = {it - Information Technology}, month = {October}, number = 5, pages = {262--269}, publisher = {Oldenbourg Wissenschaftsverlag}, title = {{Benchmarking and Performance Modeling of Event-Based Systems}}, volume = 51, year = 2009 }

  %0 Journal Article %1 KoSa2009-DEBSPerformance %A Kounev, Samuel %A Sachs, Kai %C Munich, Germany %D 2009 %I Oldenbourg Wissenschaftsverlag %J it - Information Technology %N 5 %P 262--269 %T {Benchmarking and Performance Modeling of Event-Based Systems} %V 51 %X Event-based systems are used increasingly often to build loosely-coupled distributed applications. With their growing popularity and gradual adoption in mission critical areas, the need for novel techniques for benchmarking and performance modeling of event-based systems is increasing. In this article, we provide an overview of the state-of-the-art in this area considering both centralized systems based on message-oriented middleware as well as large-scale distributed publish/subscribe systems. We consider a number of specific techniques for benchmarking and performance modeling, discuss their advantages and disadvantages, and provide references for further information. The techniques we review help to ensure that systems are designed and sized to meet their quality-of-service requirements.
• 
  {Performance evaluation of message-oriented middleware using the SPECjms2007 benchmark}. Sachs, Kai; Kounev, Samuel; Bacon, Jean; Buchmann, Alejandro; in Performance Evaluation 66(8) 410–434. Elsevier Science Publishers B. V., Amsterdam, The Netherlands.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Message-oriented middleware (MOM) is at the core of a vast number of financial services and telco applications, and is gaining increasing traction in other industries, such as manufacturing, transportation, health-care and supply chain management. Novel messaging applications, however, pose some serious performance and scalability challenges. In this paper, we present a methodology for performance evaluation of MOM platforms using the SPECjms2007 benchmark which is the world's first industry-standard benchmark specialized for MOM. SPECjms2007 is based on a novel application in the supply chain management domain designed to stress MOM infrastructures in a manner representative of real-world applications. In addition to providing a standard workload and metrics for MOM performance, the benchmark provides a flexible performance analysis framework that allows users to tailor the workload to their requirements. The contributions of this paper are: i) we present a detailed workload characterization of SPECjms2007 with the goal to help users understand the internal components of the workload and the way they are scaled, ii) we show how the workload can be customized to exercise and evaluate selected aspects of MOM performance, iii) we present a case study of a leading JMS platform, the BEA WebLogic server, conducting an in-depth performance analysis of the platform under a number of different workload and configuration scenarios. The methodology we propose is the first one that uses an industry-standard benchmark providing both a representative workload as well as the ability to customize it to evaluate the features of MOM platforms selectively.

  @article{SaKoBaBu2008-PERFEVAL-SPECjms2007, abstract = {Message-oriented middleware (MOM) is at the core of a vast number of financial services and telco applications, and is gaining increasing traction in other industries, such as manufacturing, transportation, health-care and supply chain management. Novel messaging applications, however, pose some serious performance and scalability challenges. In this paper, we present a methodology for performance evaluation of MOM platforms using the SPECjms2007 benchmark which is the world's first industry-standard benchmark specialized for MOM. SPECjms2007 is based on a novel application in the supply chain management domain designed to stress MOM infrastructures in a manner representative of real-world applications. In addition to providing a standard workload and metrics for MOM performance, the benchmark provides a flexible performance analysis framework that allows users to tailor the workload to their requirements. The contributions of this paper are: i) we present a detailed workload characterization of SPECjms2007 with the goal to help users understand the internal components of the workload and the way they are scaled, ii) we show how the workload can be customized to exercise and evaluate selected aspects of MOM performance, iii) we present a case study of a leading JMS platform, the BEA WebLogic server, conducting an in-depth performance analysis of the platform under a number of different workload and configuration scenarios. The methodology we propose is the first one that uses an industry-standard benchmark providing both a representative workload as well as the ability to customize it to evaluate the features of MOM platforms selectively.}, address = {Amsterdam, The Netherlands}, author = {Sachs, Kai and Kounev, Samuel and Bacon, Jean and Buchmann, Alejandro}, journal = {Performance Evaluation}, month = {August}, number = 8, pages = {410--434}, publisher = {Elsevier Science Publishers B. V.}, title = {{Performance evaluation of message-oriented middleware using the SPECjms2007 benchmark}}, volume = 66, year = 2009 }

  %0 Journal Article %1 SaKoBaBu2008-PERFEVAL-SPECjms2007 %A Sachs, Kai %A Kounev, Samuel %A Bacon, Jean %A Buchmann, Alejandro %C Amsterdam, The Netherlands %D 2009 %I Elsevier Science Publishers B. V. %J Performance Evaluation %N 8 %P 410--434 %T {Performance evaluation of message-oriented middleware using the SPECjms2007 benchmark} %U http://www.elsevier.com/wps/find/journaldescription.cws_home/505618/description %V 66 %X Message-oriented middleware (MOM) is at the core of a vast number of financial services and telco applications, and is gaining increasing traction in other industries, such as manufacturing, transportation, health-care and supply chain management. Novel messaging applications, however, pose some serious performance and scalability challenges. In this paper, we present a methodology for performance evaluation of MOM platforms using the SPECjms2007 benchmark which is the world's first industry-standard benchmark specialized for MOM. SPECjms2007 is based on a novel application in the supply chain management domain designed to stress MOM infrastructures in a manner representative of real-world applications. In addition to providing a standard workload and metrics for MOM performance, the benchmark provides a flexible performance analysis framework that allows users to tailor the workload to their requirements. The contributions of this paper are: i) we present a detailed workload characterization of SPECjms2007 with the goal to help users understand the internal components of the workload and the way they are scaled, ii) we show how the workload can be customized to exercise and evaluate selected aspects of MOM performance, iii) we present a case study of a leading JMS platform, the BEA WebLogic server, conducting an in-depth performance analysis of the platform under a number of different workload and configuration scenarios. The methodology we propose is the first one that uses an industry-standard benchmark providing both a representative workload as well as the ability to customize it to evaluate the features of MOM platforms selectively.
• 
  {Autonomic QoS control in enterprise Grid environments using online simulation}. Nou, Ramon; Kounev, Samuel; Julia, Ferran; Torres, Jordi; in Journal of Systems and Software 82(3) 486–502. Elsevier Science Publishers B. V., Amsterdam, The Netherlands.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  As Grid Computing increasingly enters the commercial domain, performance and Quality of Service (QoS) issues are becoming a major concern. The inherent complexity, heterogeneity and dynamics of Grid computing environments pose some challenges in managing their capacity to ensure that QoS requirements are continuously met. In this paper, a comprehensive framework for autonomic QoS control in enterprise Grid environments using online simulation is proposed. The paper presents a novel methodology for designing autonomic QoS-aware resource managers that have the capability to predict the performance of the Grid components they manage and allocate resources in such a way that service level agreements are honored. Support for advanced features such as autonomic workload characterization on-the-fly, dynamic deployment of Grid servers on demand, as well as dynamic system reconfiguration after a server failure is provided. The goal is to make the Grid middleware self-configurable and adaptable to changes in the system environment and workload. The approach is subjected to an extensive experimental evaluation in the context of a real-world Grid environment and its effectiveness, practicality and performance are demonstrated.

  @article{NoKoJuTo2008-JSS-GridAutoQoS, abstract = {As Grid Computing increasingly enters the commercial domain, performance and Quality of Service (QoS) issues are becoming a major concern. The inherent complexity, heterogeneity and dynamics of Grid computing environments pose some challenges in managing their capacity to ensure that QoS requirements are continuously met. In this paper, a comprehensive framework for autonomic QoS control in enterprise Grid environments using online simulation is proposed. The paper presents a novel methodology for designing autonomic QoS-aware resource managers that have the capability to predict the performance of the Grid components they manage and allocate resources in such a way that service level agreements are honored. Support for advanced features such as autonomic workload characterization on-the-fly, dynamic deployment of Grid servers on demand, as well as dynamic system reconfiguration after a server failure is provided. The goal is to make the Grid middleware self-configurable and adaptable to changes in the system environment and workload. The approach is subjected to an extensive experimental evaluation in the context of a real-world Grid environment and its effectiveness, practicality and performance are demonstrated.}, address = {Amsterdam, The Netherlands}, author = {Nou, Ramon and Kounev, Samuel and Julia, Ferran and Torres, Jordi}, journal = {Journal of Systems and Software}, month = {March}, number = 3, pages = {486--502}, publisher = {Elsevier Science Publishers B. V.}, title = {{Autonomic QoS control in enterprise Grid environments using online simulation}}, volume = 82, year = 2009 }

  %0 Journal Article %1 NoKoJuTo2008-JSS-GridAutoQoS %A Nou, Ramon %A Kounev, Samuel %A Julia, Ferran %A Torres, Jordi %C Amsterdam, The Netherlands %D 2009 %I Elsevier Science Publishers B. V. %J Journal of Systems and Software %N 3 %P 486--502 %T {Autonomic QoS control in enterprise Grid environments using online simulation} %V 82 %X As Grid Computing increasingly enters the commercial domain, performance and Quality of Service (QoS) issues are becoming a major concern. The inherent complexity, heterogeneity and dynamics of Grid computing environments pose some challenges in managing their capacity to ensure that QoS requirements are continuously met. In this paper, a comprehensive framework for autonomic QoS control in enterprise Grid environments using online simulation is proposed. The paper presents a novel methodology for designing autonomic QoS-aware resource managers that have the capability to predict the performance of the Grid components they manage and allocate resources in such a way that service level agreements are honored. Support for advanced features such as autonomic workload characterization on-the-fly, dynamic deployment of Grid servers on demand, as well as dynamic system reconfiguration after a server failure is provided. The goal is to make the Grid middleware self-configurable and adaptable to changes in the system environment and workload. The approach is subjected to an extensive experimental evaluation in the context of a real-world Grid environment and its effectiveness, practicality and performance are demonstrated.
• 
  {QPME - A Performance Modeling Tool Based on Queueing Petri Nets}. Kounev, Samuel; Dutz, Christofer; in ACM SIGMETRICS Performance Evaluation Review (PER), Special Issue on Tools for Computer Performance Modeling and Reliability Analysis 36 46–51. ACM.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Queueing Petri nets are a powerful formalism that can be exploited for modeling distributed systems and analyzing their performance and scalability. By combining the modeling power and expressiveness of queueing networks and stochastic Petri nets, queueing Petri nets provide a number of advantages. In this paper, we present QPME (Queueing Petri net Modeling Environment) - a tool that supports the modeling and analysis of systems using queueing Petri nets. QPME provides an Eclipse-based editor for designing queueing Petri net models and a powerful simulation engine for analyzing the models. After presenting the tool, we discuss the ongoing work on the QPME project and the planned future enhancements of the tool.

  @article{KoDu2009-SIGMETRICS_PER-QPME, abstract = {Queueing Petri nets are a powerful formalism that can be exploited for modeling distributed systems and analyzing their performance and scalability. By combining the modeling power and expressiveness of queueing networks and stochastic Petri nets, queueing Petri nets provide a number of advantages. In this paper, we present QPME (Queueing Petri net Modeling Environment) - a tool that supports the modeling and analysis of systems using queueing Petri nets. QPME provides an Eclipse-based editor for designing queueing Petri net models and a powerful simulation engine for analyzing the models. After presenting the tool, we discuss the ongoing work on the QPME project and the planned future enhancements of the tool.}, author = {Kounev, Samuel and Dutz, Christofer}, journal = {ACM SIGMETRICS Performance Evaluation Review (PER), Special Issue on Tools for Computer Performance Modeling and Reliability Analysis}, month = 3, pages = {46--51}, publisher = {ACM}, title = {{QPME - A Performance Modeling Tool Based on Queueing Petri Nets}}, volume = 36, year = 2009 }

  %0 Journal Article %1 KoDu2009-SIGMETRICS_PER-QPME %A Kounev, Samuel %A Dutz, Christofer %D 2009 %I ACM %J ACM SIGMETRICS Performance Evaluation Review (PER), Special Issue on Tools for Computer Performance Modeling and Reliability Analysis %P 46--51 %T {QPME - A Performance Modeling Tool Based on Queueing Petri Nets} %V 36 %X Queueing Petri nets are a powerful formalism that can be exploited for modeling distributed systems and analyzing their performance and scalability. By combining the modeling power and expressiveness of queueing networks and stochastic Petri nets, queueing Petri nets provide a number of advantages. In this paper, we present QPME (Queueing Petri net Modeling Environment) - a tool that supports the modeling and analysis of systems using queueing Petri nets. QPME provides an Eclipse-based editor for designing queueing Petri net models and a powerful simulation engine for analyzing the models. After presenting the tool, we discuss the ongoing work on the QPME project and the planned future enhancements of the tool.

• 
  {Kaffeekunde - SPECjms misst Message-oriented Middleware}. Kounev, Samuel; Sachs, Kai;
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  @misc{sachs2008a, author = {Kounev, Samuel and Sachs, Kai}, howpublished = {iX Magazin, Heft 02/2008, Heise Zeitschriften Verlag}, title = {{Kaffeekunde - SPECjms misst Message-oriented Middleware}}, year = 2008 }

  %0 Generic %1 sachs2008a %A Kounev, Samuel %A Sachs, Kai %D 2008 %T {Kaffeekunde - SPECjms misst Message-oriented Middleware} %U http://www.heise.de/kiosk/archiv/ix/2008/2/121

• 
  {SimQPN - a tool and methodology for analyzing queueing Petri net models by means of simulation}. Kounev, Samuel; Buchmann, Alejandro; in Performance Evaluation 63(4-5) 364–394. Elsevier Science Publishers B. V., Amsterdam, The Netherlands.
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @article{KoBu2006-PERFEVAL-SimQPN, address = {Amsterdam, The Netherlands}, author = {Kounev, Samuel and Buchmann, Alejandro}, journal = {Performance Evaluation}, month = 5, number = {4-5}, pages = {364--394}, publisher = {Elsevier Science Publishers B. V.}, title = {{SimQPN - a tool and methodology for analyzing queueing Petri net models by means of simulation}}, volume = 63, year = 2006 }

  %0 Journal Article %1 KoBu2006-PERFEVAL-SimQPN %A Kounev, Samuel %A Buchmann, Alejandro %C Amsterdam, The Netherlands %D 2006 %I Elsevier Science Publishers B. V. %J Performance Evaluation %N 4-5 %P 364--394 %T {SimQPN - a tool and methodology for analyzing queueing Petri net models by means of simulation} %V 63
• 
  {Performance Modeling and Evaluation of Distributed Component-Based Systems using Queueing Petri Nets}. Kounev, Samuel; in IEEE Transactions on Software Engineering 32(7) 486–502.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Performance models are used increasingly throughout the phases of the software engineering lifecycle of distributed component-based systems. However, as systems grow in size and complexity, building models that accurately capture the different aspects of their behavior becomes a more and more challenging task. In this paper, we present a novel case study of a realistic distributed component-based system, showing how Queueing Petri Net models can be exploited as a powerful performance prediction tool in the software engineering process. A detailed system model is built in a step-by-step fashion, validated, and then used to evaluate the system performance and scalability. Along with the case study, a practical performance modeling methodology is presented which helps to construct models that accurately reflect the system performance and scalability characteristics. Taking advantage of the modeling power and expressiveness of Queueing Petri Nets, our approach makes it possible to model the system at a higher degree of accuracy, providing a number of important benefits.

  @article{Ko2006-IEEE_TSE-QPN_ModelingMethod, abstract = {Performance models are used increasingly throughout the phases of the software engineering lifecycle of distributed component-based systems. However, as systems grow in size and complexity, building models that accurately capture the different aspects of their behavior becomes a more and more challenging task. In this paper, we present a novel case study of a realistic distributed component-based system, showing how Queueing Petri Net models can be exploited as a powerful performance prediction tool in the software engineering process. A detailed system model is built in a step-by-step fashion, validated, and then used to evaluate the system performance and scalability. Along with the case study, a practical performance modeling methodology is presented which helps to construct models that accurately reflect the system performance and scalability characteristics. Taking advantage of the modeling power and expressiveness of Queueing Petri Nets, our approach makes it possible to model the system at a higher degree of accuracy, providing a number of important benefits.}, author = {Kounev, Samuel}, journal = {IEEE Transactions on Software Engineering}, month = {July}, number = 7, pages = {486--502}, title = {{Performance Modeling and Evaluation of Distributed Component-Based Systems using Queueing Petri Nets}}, volume = 32, year = 2006 }

  %0 Journal Article %1 Ko2006-IEEE_TSE-QPN_ModelingMethod %A Kounev, Samuel %D 2006 %J IEEE Transactions on Software Engineering %N 7 %P 486--502 %T {Performance Modeling and Evaluation of Distributed Component-Based Systems using Queueing Petri Nets} %V 32 %X Performance models are used increasingly throughout the phases of the software engineering lifecycle of distributed component-based systems. However, as systems grow in size and complexity, building models that accurately capture the different aspects of their behavior becomes a more and more challenging task. In this paper, we present a novel case study of a realistic distributed component-based system, showing how Queueing Petri Net models can be exploited as a powerful performance prediction tool in the software engineering process. A detailed system model is built in a step-by-step fashion, validated, and then used to evaluate the system performance and scalability. Along with the case study, a practical performance modeling methodology is presented which helps to construct models that accurately reflect the system performance and scalability characteristics. Taking advantage of the modeling power and expressiveness of Queueing Petri Nets, our approach makes it possible to model the system at a higher degree of accuracy, providing a number of important benefits.

• 
  {Performance Tuning and Optimization of J2EE Applications on the JBoss Platform}. Kounev, Samuel; Weis, B{\"o}rn; Buchmann, Alejandro; in Journal of Computer Resource Management 113 Computer Measurement Group (CMG).
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @article{KoWeBu2004-CMG_JCRM-JBoss, author = {Kounev, Samuel and Weis, B{\"o}rn and Buchmann, Alejandro}, journal = {Journal of Computer Resource Management}, month = {September}, publisher = {Computer Measurement Group (CMG)}, title = {{Performance Tuning and Optimization of J2EE Applications on the JBoss Platform}}, volume = 113, year = 2004 }

  %0 Journal Article %1 KoWeBu2004-CMG_JCRM-JBoss %A Kounev, Samuel %A Weis, B{\"o}rn %A Buchmann, Alejandro %D 2004 %I Computer Measurement Group (CMG) %J Journal of Computer Resource Management %T {Performance Tuning and Optimization of J2EE Applications on the JBoss Platform} %V 113

• 
  {Distributed Video-Conferencing System Organized for Work on the Internet with the Use of Multimedia Server}. Nenov, Plamen; Kounev, Samuel; Mihailov, Dimiter; in Journal of Computing and Information
  • [ BibTeX ]
  • [ EndNote ]
  @article{NeKoMi1999-JCI-VideoConf, author = {Nenov, Plamen and Kounev, Samuel and Mihailov, Dimiter}, journal = {Journal of Computing and Information}, title = {{Distributed Video-Conferencing System Organized for Work on the Internet with the Use of Multimedia Server}}, year = 1999 }

  %0 Journal Article %1 NeKoMi1999-JCI-VideoConf %A Nenov, Plamen %A Kounev, Samuel %A Mihailov, Dimiter %D 1999 %J Journal of Computing and Information %T {Distributed Video-Conferencing System Organized for Work on the Internet with the Use of Multimedia Server}

• [ 2022 ]
• [ 2021 ]
• [ 2020 ]
• [ 2019 ]
• [ 2018 ]
• [ 2017 ]
• [ 2016 ]
• [ 2015 ]
• [ 2014 ]
• [ 2013 ]
• [ 2012 ]
• [ 2011 ]
• [ 2010 ]
• [ 2009 ]
• [ 2008 ]
• [ 2007 ]
• [ 2003 ]
• [ 2002 ]
• [ 1999 ]

• 
  Towards a Cryptography Benchmark: A View on Attribute Based Encryption Schemes. Prantl, Thomas; Zeck, Timo; Iffländer, Lukas; Beierlieb, Lukas; Dmitrenko, Alexandra; Krupitzer, Christian; Kounev, Samuel; in 2022 5th Conference on Cloud and Internet of Things (CIoT)IEEE.
  • [ BibTeX ]
  • [ EndNote ]
  @inproceedings{PrZeIfBeDmKrKo2022-CIoT-ABEBenchmark, author = {Prantl, Thomas and Zeck, Timo and Iffländer, Lukas and Beierlieb, Lukas and Dmitrenko, Alexandra and Krupitzer, Christian and Kounev, Samuel}, booktitle = {2022 5th Conference on Cloud and Internet of Things (CIoT)}, organization = {IEEE}, title = {Towards a Cryptography Benchmark: A View on Attribute Based Encryption Schemes}, year = 2022 }

  %0 Conference Paper %1 PrZeIfBeDmKrKo2022-CIoT-ABEBenchmark %A Prantl, Thomas %A Zeck, Timo %A Iffländer, Lukas %A Beierlieb, Lukas %A Dmitrenko, Alexandra %A Krupitzer, Christian %A Kounev, Samuel %B 2022 5th Conference on Cloud and Internet of Things (CIoT) %D 2022 %T Towards a Cryptography Benchmark: A View on Attribute Based Encryption Schemes
• 
  Why Is It Not Solved Yet? Challenges for Production-Ready Autoscaling. Straesser, Martin; Grohmann, Johannes; von Kistowski, J\’{o}akim; Eismann, Simon; Bauer, Andr\’{e}; Kounev, Samuel; in Proceedings of the 2022 ACM/SPEC on International Conference on Performance Engineering 105–115. Association for Computing Machinery, New York, NY, USA.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Autoscaling is a task of major importance in the cloud computing domain as it directly affects both operating costs and customer experience. Although there has been active research in this area for over ten years now, there is still a significant gap between the proposed methods in the literature and the deployed autoscalers in practice. Hence, many research autoscalers do not find their way into production deployments. This paper describes six core challenges that arise in production systems that are still not solved by most research autoscalers. We illustrate these problems through experiments in a realistic cloud environment with a real-world multi-service business application and show that commonly used autoscalers have various shortcomings. In addition, we analyze the behavior of overloaded services and show that these can be problematic for existing autoscalers. Generally, we analyze that these challenges are only insufficiently addressed in the literature and conclude that future scaling approaches should focus on the needs of production systems.

  @inproceedings{10.1145/3489525.3511680, abstract = {Autoscaling is a task of major importance in the cloud computing domain as it directly affects both operating costs and customer experience. Although there has been active research in this area for over ten years now, there is still a significant gap between the proposed methods in the literature and the deployed autoscalers in practice. Hence, many research autoscalers do not find their way into production deployments. This paper describes six core challenges that arise in production systems that are still not solved by most research autoscalers. We illustrate these problems through experiments in a realistic cloud environment with a real-world multi-service business application and show that commonly used autoscalers have various shortcomings. In addition, we analyze the behavior of overloaded services and show that these can be problematic for existing autoscalers. Generally, we analyze that these challenges are only insufficiently addressed in the literature and conclude that future scaling approaches should focus on the needs of production systems.}, address = {New York, NY, USA}, author = {Straesser, Martin and Grohmann, Johannes and von Kistowski, J\'{o}akim and Eismann, Simon and Bauer, Andr\'{e} and Kounev, Samuel}, booktitle = {Proceedings of the 2022 ACM/SPEC on International Conference on Performance Engineering}, pages = {105–115}, publisher = {Association for Computing Machinery}, series = {ICPE '22}, title = {Why Is It Not Solved Yet? Challenges for Production-Ready Autoscaling}, year = 2022 }

  %0 Conference Paper %1 10.1145/3489525.3511680 %A Straesser, Martin %A Grohmann, Johannes %A von Kistowski, J\'{o}akim %A Eismann, Simon %A Bauer, Andr\'{e} %A Kounev, Samuel %B Proceedings of the 2022 ACM/SPEC on International Conference on Performance Engineering %C New York, NY, USA %D 2022 %I Association for Computing Machinery %P 105–115 %T Why Is It Not Solved Yet? Challenges for Production-Ready Autoscaling %U https://doi.org/10.1145/3489525.3511680 %X Autoscaling is a task of major importance in the cloud computing domain as it directly affects both operating costs and customer experience. Although there has been active research in this area for over ten years now, there is still a significant gap between the proposed methods in the literature and the deployed autoscalers in practice. Hence, many research autoscalers do not find their way into production deployments. This paper describes six core challenges that arise in production systems that are still not solved by most research autoscalers. We illustrate these problems through experiments in a realistic cloud environment with a real-world multi-service business application and show that commonly used autoscalers have various shortcomings. In addition, we analyze the behavior of overloaded services and show that these can be problematic for existing autoscalers. Generally, we analyze that these challenges are only insufficiently addressed in the literature and conclude that future scaling approaches should focus on the needs of production systems.

• 
  A Simulation-based Optimization Framework for Online Adaptation of Networks. Herrnleben, Stefan; Grohmann, Johannes; Rygielski, Pitor; Lesch, Veronika; Krupitzer, Christian; Kounev, Samuel; in Proceedings of the 12th EAI International Conference on Simulation Tools and Techniques (SIMUtools), H. Song, D. Jiang (eds.) 513–532. Springer International Publishing, Cham.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Today's data centers face a rapid change of deployed services, growing complexity, and increasing performance requirements. Customers expect not only round-the-clock availability of the hosted services but also high responsiveness. Besides optimizing software architectures and deployments, networks have to be adapted to handle the changing and volatile demands. Approaches from self-adaptive systems can be used for optimizing data center networks to continuously meet Service Level Agreements (SLAs) between data center operators and customers. However, existing approaches focus only on specific objectives like topology design, power optimization, or traffic engineering. In this paper, we present an extensible framework that analyzes networks using different types of simulation and adapts them subject to multiple objectives using various adaptation techniques. Analyzing each suggested adaptation ensures that performance requirements and SLAs are continuously met. We evaluate our framework w.r.t. (i) general requirements and assessments of languages and frameworks for adaptation models, (ii) finding Pareto-optimal solutions considering a multi-dimensional cost model, and (iii) scalability. The evaluation shows that our approach detects the bottlenecks and the violated SLAs correctly, outputs valid and cost-optimal adaptations, and keeps the runtime for the adaptation process constant even with increasing network size and an increasing number of alternative configurations.

  @inproceedings{HeGrRyLeKrKo2020-SIMUtools-Network-Online-Adaptation, abstract = {Today's data centers face a rapid change of deployed services, growing complexity, and increasing performance requirements. Customers expect not only round-the-clock availability of the hosted services but also high responsiveness. Besides optimizing software architectures and deployments, networks have to be adapted to handle the changing and volatile demands. Approaches from self-adaptive systems can be used for optimizing data center networks to continuously meet Service Level Agreements (SLAs) between data center operators and customers. However, existing approaches focus only on specific objectives like topology design, power optimization, or traffic engineering. In this paper, we present an extensible framework that analyzes networks using different types of simulation and adapts them subject to multiple objectives using various adaptation techniques. Analyzing each suggested adaptation ensures that performance requirements and SLAs are continuously met. We evaluate our framework w.r.t. (i) general requirements and assessments of languages and frameworks for adaptation models, (ii) finding Pareto-optimal solutions considering a multi-dimensional cost model, and (iii) scalability. The evaluation shows that our approach detects the bottlenecks and the violated SLAs correctly, outputs valid and cost-optimal adaptations, and keeps the runtime for the adaptation process constant even with increasing network size and an increasing number of alternative configurations.}, address = {Cham}, author = {Herrnleben, Stefan and Grohmann, Johannes and Rygielski, Pitor and Lesch, Veronika and Krupitzer, Christian and Kounev, Samuel}, booktitle = {Proceedings of the 12th EAI International Conference on Simulation Tools and Techniques (SIMUtools)}, editor = {Song, Houbing and Jiang, Dingde}, month = {August}, pages = {513--532}, publisher = {Springer International Publishing}, series = {SIMUtools 2020}, title = {A Simulation-based Optimization Framework for Online Adaptation of Networks}, year = 2021 }

  %0 Conference Paper %1 HeGrRyLeKrKo2020-SIMUtools-Network-Online-Adaptation %A Herrnleben, Stefan %A Grohmann, Johannes %A Rygielski, Pitor %A Lesch, Veronika %A Krupitzer, Christian %A Kounev, Samuel %B Proceedings of the 12th EAI International Conference on Simulation Tools and Techniques (SIMUtools) %C Cham %D 2021 %E Song, Houbing %E Jiang, Dingde %I Springer International Publishing %P 513--532 %T A Simulation-based Optimization Framework for Online Adaptation of Networks %U https://doi.org/10.1007/978-3-030-72792-5_41 %X Today's data centers face a rapid change of deployed services, growing complexity, and increasing performance requirements. Customers expect not only round-the-clock availability of the hosted services but also high responsiveness. Besides optimizing software architectures and deployments, networks have to be adapted to handle the changing and volatile demands. Approaches from self-adaptive systems can be used for optimizing data center networks to continuously meet Service Level Agreements (SLAs) between data center operators and customers. However, existing approaches focus only on specific objectives like topology design, power optimization, or traffic engineering. In this paper, we present an extensible framework that analyzes networks using different types of simulation and adapts them subject to multiple objectives using various adaptation techniques. Analyzing each suggested adaptation ensures that performance requirements and SLAs are continuously met. We evaluate our framework w.r.t. (i) general requirements and assessments of languages and frameworks for adaptation models, (ii) finding Pareto-optimal solutions considering a multi-dimensional cost model, and (iii) scalability. The evaluation shows that our approach detects the bottlenecks and the violated SLAs correctly, outputs valid and cost-optimal adaptations, and keeps the runtime for the adaptation process constant even with increasing network size and an increasing number of alternative configurations.
• 
  ComBench: A Benchmarking Framework for Publish/Subscribe Communication Protocols Under Network Limitations. Herrnleben, Stefan; Leidinger, Maximilian; Lesch, Veronika; Prantl, Thomas; Grohmann, Johannes; Krupitzer, Christian; Kounev, Samuel; in Performance Evaluation Methodologies and Tools, Q. Zhao, L. Xia (eds.) 72–92. Springer International Publishing, Cham.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Efficient and dependable communication is a highly relevant aspect for Internet of Things (IoT) systems in which tiny sensors, actuators, wearables, or other smart devices exchange messages. Various publish/subscribe protocols address the challenges of communication in IoT systems. The selection process of a suitable protocol should consider the communication behavior of the application, the protocol's performance, the resource requirements on the end device, and the network connection quality, as IoT environments often rely on wireless networks. Benchmarking is a common approach to evaluate and compare systems, considering the performance and aspects like dependability or security. In this paper, we present our IoT communication benchmarking framework ComBench for publish/subscribe protocols focusing on constrained networks with varying quality conditions. The benchmarking framework supports system designers, software engineers, and application developers to select and investigate the behavior of communication protocols. Our benchmarking framework contributes to (i) show the impact of fluctuating network quality on communication, (ii) compare multiple protocols, protocol features, and protocol implementations, and (iii) analyze scalability, robustness, and dependability of clients, networks, and brokers in different scenarios. Our case study demonstrates the applicability of our framework to support the decision for the best-suited protocol in various scenarios.

  @inproceedings{10.1007/978-3-030-92511-6_5, abstract = {Efficient and dependable communication is a highly relevant aspect for Internet of Things (IoT) systems in which tiny sensors, actuators, wearables, or other smart devices exchange messages. Various publish/subscribe protocols address the challenges of communication in IoT systems. The selection process of a suitable protocol should consider the communication behavior of the application, the protocol's performance, the resource requirements on the end device, and the network connection quality, as IoT environments often rely on wireless networks. Benchmarking is a common approach to evaluate and compare systems, considering the performance and aspects like dependability or security. In this paper, we present our IoT communication benchmarking framework ComBench for publish/subscribe protocols focusing on constrained networks with varying quality conditions. The benchmarking framework supports system designers, software engineers, and application developers to select and investigate the behavior of communication protocols. Our benchmarking framework contributes to (i) show the impact of fluctuating network quality on communication, (ii) compare multiple protocols, protocol features, and protocol implementations, and (iii) analyze scalability, robustness, and dependability of clients, networks, and brokers in different scenarios. Our case study demonstrates the applicability of our framework to support the decision for the best-suited protocol in various scenarios.}, address = {Cham}, author = {Herrnleben, Stefan and Leidinger, Maximilian and Lesch, Veronika and Prantl, Thomas and Grohmann, Johannes and Krupitzer, Christian and Kounev, Samuel}, booktitle = {Performance Evaluation Methodologies and Tools}, editor = {Zhao, Qianchuan and Xia, Li}, pages = {72--92}, publisher = {Springer International Publishing}, title = {ComBench: A Benchmarking Framework for Publish/Subscribe Communication Protocols Under Network Limitations}, year = 2021 }

  %0 Conference Paper %1 10.1007/978-3-030-92511-6_5 %A Herrnleben, Stefan %A Leidinger, Maximilian %A Lesch, Veronika %A Prantl, Thomas %A Grohmann, Johannes %A Krupitzer, Christian %A Kounev, Samuel %B Performance Evaluation Methodologies and Tools %C Cham %D 2021 %E Zhao, Qianchuan %E Xia, Li %I Springer International Publishing %P 72--92 %T ComBench: A Benchmarking Framework for Publish/Subscribe Communication Protocols Under Network Limitations %U https://link.springer.com/chapter/10.1007%2F978-3-030-92511-6_5 %X Efficient and dependable communication is a highly relevant aspect for Internet of Things (IoT) systems in which tiny sensors, actuators, wearables, or other smart devices exchange messages. Various publish/subscribe protocols address the challenges of communication in IoT systems. The selection process of a suitable protocol should consider the communication behavior of the application, the protocol's performance, the resource requirements on the end device, and the network connection quality, as IoT environments often rely on wireless networks. Benchmarking is a common approach to evaluate and compare systems, considering the performance and aspects like dependability or security. In this paper, we present our IoT communication benchmarking framework ComBench for publish/subscribe protocols focusing on constrained networks with varying quality conditions. The benchmarking framework supports system designers, software engineers, and application developers to select and investigate the behavior of communication protocols. Our benchmarking framework contributes to (i) show the impact of fluctuating network quality on communication, (ii) compare multiple protocols, protocol features, and protocol implementations, and (iii) analyze scalability, robustness, and dependability of clients, networks, and brokers in different scenarios. Our case study demonstrates the applicability of our framework to support the decision for the best-suited protocol in various scenarios.
• 
  Sizeless: Predicting the Optimal Size of Serverless Functions. Eismann, Simon; Bui, Long; Grohmann, Johannes; Abad, Cristina; Herbst, Nikolas; Kounev, Samuel; in Proceedings of the 22nd International MIDDLEWARE Conference 248–259.

  Best Student Paper Award, ACM Artifacts Evaluated — Functional
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Serverless functions are an emerging cloud computing paradigm that is being rapidly adopted by both industry and academia. In this cloud computing model, the provider opaquely handles resource management tasks such as resource provisioning, deployment, and auto-scaling. The only resource management task that developers are still in charge of is selecting how much resources are allocated to each worker instance. However, selecting the optimal size of serverless functions is quite challenging, so developers often neglect it despite its significant cost and performance benefits. Existing approaches aiming to automate serverless functions resource sizing require dedicated performance tests, which are time-consuming to implement and maintain. In this paper, we introduce an approach to predict the optimal resource size of a serverless function using monitoring data from a single resource size. As our approach does not require dedicated performance tests, it enables cloud providers to implement resource sizing on a platform level and automate the last resource management task associated with serverless functions. We evaluate our approach on four different serverless applications on AWS, where it predicts the execution time of the other memory sizes based on monitoring data for a single memory size with an average prediction error of 15.3%. Based on these predictions, it selects the optimal memory size for 79.0% of the serverless functions and the secondbest memory size for 12.3% of the serverless functions, which results in an average speedup of 39.7% while also decreasing average costs by 2.6%.

  @inproceedings{eismann2021sizeless, abstract = {Serverless functions are an emerging cloud computing paradigm that is being rapidly adopted by both industry and academia. In this cloud computing model, the provider opaquely handles resource management tasks such as resource provisioning, deployment, and auto-scaling. The only resource management task that developers are still in charge of is selecting how much resources are allocated to each worker instance. However, selecting the optimal size of serverless functions is quite challenging, so developers often neglect it despite its significant cost and performance benefits. Existing approaches aiming to automate serverless functions resource sizing require dedicated performance tests, which are time-consuming to implement and maintain. In this paper, we introduce an approach to predict the optimal resource size of a serverless function using monitoring data from a single resource size. As our approach does not require dedicated performance tests, it enables cloud providers to implement resource sizing on a platform level and automate the last resource management task associated with serverless functions. We evaluate our approach on four different serverless applications on AWS, where it predicts the execution time of the other memory sizes based on monitoring data for a single memory size with an average prediction error of 15.3%. Based on these predictions, it selects the optimal memory size for 79.0% of the serverless functions and the secondbest memory size for 12.3% of the serverless functions, which results in an average speedup of 39.7% while also decreasing average costs by 2.6%.}, author = {Eismann, Simon and Bui, Long and Grohmann, Johannes and Abad, Cristina and Herbst, Nikolas and Kounev, Samuel}, booktitle = {Proceedings of the 22nd International MIDDLEWARE Conference}, note = {Best Student Paper Award, ACM Artifacts Evaluated — Functional}, pages = {248–259}, title = {Sizeless: Predicting the Optimal Size of Serverless Functions}, year = 2021 }

  %0 Conference Paper %1 eismann2021sizeless %A Eismann, Simon %A Bui, Long %A Grohmann, Johannes %A Abad, Cristina %A Herbst, Nikolas %A Kounev, Samuel %B Proceedings of the 22nd International MIDDLEWARE Conference %D 2021 %P 248–259 %T Sizeless: Predicting the Optimal Size of Serverless Functions %U https://dl.acm.org/doi/abs/10.1145/3464298.3493398 %X Serverless functions are an emerging cloud computing paradigm that is being rapidly adopted by both industry and academia. In this cloud computing model, the provider opaquely handles resource management tasks such as resource provisioning, deployment, and auto-scaling. The only resource management task that developers are still in charge of is selecting how much resources are allocated to each worker instance. However, selecting the optimal size of serverless functions is quite challenging, so developers often neglect it despite its significant cost and performance benefits. Existing approaches aiming to automate serverless functions resource sizing require dedicated performance tests, which are time-consuming to implement and maintain. In this paper, we introduce an approach to predict the optimal resource size of a serverless function using monitoring data from a single resource size. As our approach does not require dedicated performance tests, it enables cloud providers to implement resource sizing on a platform level and automate the last resource management task associated with serverless functions. We evaluate our approach on four different serverless applications on AWS, where it predicts the execution time of the other memory sizes based on monitoring data for a single memory size with an average prediction error of 15.3%. Based on these predictions, it selects the optimal memory size for 79.0% of the serverless functions and the secondbest memory size for 12.3% of the serverless functions, which results in an average speedup of 39.7% while also decreasing average costs by 2.6%.
• 
  Machine Learning Model Update Strategies for Hard Disk Drive Failure Prediction. Z{\"u}fle, Marwin; Erhard, Florian; Kounev, Samuel; in 20th IEEE International Conference on Machine Learning and Applications (ICMLA)

  (to appear)
  • [ BibTeX ]
  • [ EndNote ]
  @inproceedings{zufle2021machine, author = {Z{\"u}fle, Marwin and Erhard, Florian and Kounev, Samuel}, booktitle = {20th IEEE International Conference on Machine Learning and Applications (ICMLA)}, note = {(to appear)}, title = {Machine Learning Model Update Strategies for Hard Disk Drive Failure Prediction}, year = 2021 }

  %0 Conference Paper %1 zufle2021machine %A Z{\"u}fle, Marwin %A Erhard, Florian %A Kounev, Samuel %B 20th IEEE International Conference on Machine Learning and Applications (ICMLA) %D 2021 %T Machine Learning Model Update Strategies for Hard Disk Drive Failure Prediction
• 
  Benchmarking of Pre- and Post-Quantum Group Encryption Schemes with Focus on IoT. Prantl, Thomas; Prantl, Dominik; Bauer, André; Iffländer, Lukas; Dmitrenko, Alexandra; Krupitzer, Christian; Kounev, Samuel; in 2021 IEEE 40th International Performance Computing and Communications Conference (IPCCC)
  • [ BibTeX ]
  • [ EndNote ]
  @inproceedings{PrPrBaIfDmKrKo2021-IPCCC-QuantumGroupEncryption, author = {Prantl, Thomas and Prantl, Dominik and Bauer, André and Iffländer, Lukas and Dmitrenko, Alexandra and Krupitzer, Christian and Kounev, Samuel}, booktitle = {2021 IEEE 40th International Performance Computing and Communications Conference (IPCCC)}, title = {Benchmarking of Pre- and Post-Quantum Group Encryption Schemes with Focus on IoT}, year = 2021 }

  %0 Conference Paper %1 PrPrBaIfDmKrKo2021-IPCCC-QuantumGroupEncryption %A Prantl, Thomas %A Prantl, Dominik %A Bauer, André %A Iffländer, Lukas %A Dmitrenko, Alexandra %A Krupitzer, Christian %A Kounev, Samuel %B 2021 IEEE 40th International Performance Computing and Communications Conference (IPCCC) %D 2021 %T Benchmarking of Pre- and Post-Quantum Group Encryption Schemes with Focus on IoT
• 
  Performance Evaluation of a Post-Quantum Public-Key Cryptosystem. Prantl, Thomas; Prantl, Dominik; Beierlieb, Lukas; Iffländer, Lukas; Dmitrienko, Alexandra; Krupitzer, Christian; Kounev, Samuel; in 2021 IEEE 40th International Performance Computing and Communications Conference (IPCCC)IEEE.
  • [ BibTeX ]
  • [ EndNote ]
  @inproceedings{PrPrBeIfDnKrKo2021-IPCCC-QuantumPublicKeyCryposystem, author = {Prantl, Thomas and Prantl, Dominik and Beierlieb, Lukas and Iffländer, Lukas and Dmitrienko, Alexandra and Krupitzer, Christian and Kounev, Samuel}, booktitle = {2021 IEEE 40th International Performance Computing and Communications Conference (IPCCC)}, organization = {IEEE}, title = {Performance Evaluation of a Post-Quantum Public-Key Cryptosystem}, year = 2021 }

  %0 Conference Paper %1 PrPrBeIfDnKrKo2021-IPCCC-QuantumPublicKeyCryposystem %A Prantl, Thomas %A Prantl, Dominik %A Beierlieb, Lukas %A Iffländer, Lukas %A Dmitrienko, Alexandra %A Krupitzer, Christian %A Kounev, Samuel %B 2021 IEEE 40th International Performance Computing and Communications Conference (IPCCC) %D 2021 %T Performance Evaluation of a Post-Quantum Public-Key Cryptosystem
• 
  Towards Situation-Aware Meta-Optimization of Adaptation Planning Strategies. Lesch, Veronika; Noack, Tanja; Hefter, Johannes; Kounev, Samuel; Krupitzer, Christian; in Proceedings of the 2nd IEEE International Conference on Autonomic Computing and Self-Organizing Systems - ACSOS 2021IEEE.

  Best Paper Award Runner Up, Full Paper Acceptance Ratio: 25%
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @inproceedings{lesch2021towards, author = {Lesch, Veronika and Noack, Tanja and Hefter, Johannes and Kounev, Samuel and Krupitzer, Christian}, booktitle = {Proceedings of the 2nd IEEE International Conference on Autonomic Computing and Self-Organizing Systems - ACSOS 2021}, note = {Best Paper Award Runner Up, Full Paper Acceptance Ratio: 25%}, publisher = {IEEE}, title = {Towards Situation-Aware Meta-Optimization of Adaptation Planning Strategies}, year = 2021 }

  %0 Conference Paper %1 lesch2021towards %A Lesch, Veronika %A Noack, Tanja %A Hefter, Johannes %A Kounev, Samuel %A Krupitzer, Christian %B Proceedings of the 2nd IEEE International Conference on Autonomic Computing and Self-Organizing Systems - ACSOS 2021 %D 2021 %I IEEE %T Towards Situation-Aware Meta-Optimization of Adaptation Planning Strategies
• 
  Recommendations for Data-Driven Degradation Estimation with Case Studies from Manufacturing and Dry-Bulk Shipping. Finke, Nils; Mohr, Marisa; Lontke, Alexander; Z{\"u}fle, Marwin; Kounev, Samuel; M{\"o}ller, Ralf; in Research Challenges in Information Science (RCIS) 189–204. Springer.
  • [ BibTeX ]
  • [ EndNote ]
  @inproceedings{finke2021recommendations, author = {Finke, Nils and Mohr, Marisa and Lontke, Alexander and Z{\"u}fle, Marwin and Kounev, Samuel and M{\"o}ller, Ralf}, booktitle = {Research Challenges in Information Science (RCIS)}, organization = {Springer}, pages = {189--204}, title = {Recommendations for Data-Driven Degradation Estimation with Case Studies from Manufacturing and Dry-Bulk Shipping}, year = 2021 }

  %0 Conference Paper %1 finke2021recommendations %A Finke, Nils %A Mohr, Marisa %A Lontke, Alexander %A Z{\"u}fle, Marwin %A Kounev, Samuel %A M{\"o}ller, Ralf %B Research Challenges in Information Science (RCIS) %D 2021 %P 189--204 %T Recommendations for Data-Driven Degradation Estimation with Case Studies from Manufacturing and Dry-Bulk Shipping
• 
  A Predictive Maintenance Methodology: Predicting the Time-to-Failure of Machines in Industry 4.0. Züfle, Marwin; Agne, Joachim; Grohmann, Johannes; Dörtoluk, Ibrahim; Kounev, Samuel; in Proceedings of the 21st IEEE IES International Conference on Industrial InformaticsIEEE.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Predictive maintenance is an essential aspect of the concept of Industry 4.0. In contrast to previous maintenance strategies, which plan repairs based on periodic schedules or threshold values, predictive maintenance is normally based on estimating the time-to-failure of machines. Thus, predictive maintenance enables a more efficient and effective maintenance approach. Although much research has already been done on time-to-failure prediction, most existing works provide only specialized approaches for specific machines. In most cases, these are either rotary machines (i.e., bearings) or lithium-ion batteries. To bridge the gap to a more general time-to-failure prediction, we propose a generic end-to-end predictive maintenance methodology for the time-to-failure prediction of industrial machines. Our methodology exhibits a number of novel aspects including a universally applicable method for feature extraction based on different types of sensor data, well-known feature transformation and selection techniques, adjustable target class assignment based on fault records with three different labeling strategies, and the training of multiple state-of-the-art machine learning classification models including hyperparameter optimization. We evaluated our time-to-failure prediction methodology in a real-world case study consisting of monitoring data gathered over several years from a large industrial press. The results demonstrated the effectiveness of the proposed methodology for six different time-to-failure pre-diction windows, as well as for the downscaled binary prediction of impending failures. In this case study, the multi-class feed-forward neural network model achieved the overall best results.

  @inproceedings{ZuAgGrDoKo-INDIN2021-PdM, abstract = {Predictive maintenance is an essential aspect of the concept of Industry 4.0. In contrast to previous maintenance strategies, which plan repairs based on periodic schedules or threshold values, predictive maintenance is normally based on estimating the time-to-failure of machines. Thus, predictive maintenance enables a more efficient and effective maintenance approach. Although much research has already been done on time-to-failure prediction, most existing works provide only specialized approaches for specific machines. In most cases, these are either rotary machines (i.e., bearings) or lithium-ion batteries. To bridge the gap to a more general time-to-failure prediction, we propose a generic end-to-end predictive maintenance methodology for the time-to-failure prediction of industrial machines. Our methodology exhibits a number of novel aspects including a universally applicable method for feature extraction based on different types of sensor data, well-known feature transformation and selection techniques, adjustable target class assignment based on fault records with three different labeling strategies, and the training of multiple state-of-the-art machine learning classification models including hyperparameter optimization. We evaluated our time-to-failure prediction methodology in a real-world case study consisting of monitoring data gathered over several years from a large industrial press. The results demonstrated the effectiveness of the proposed methodology for six different time-to-failure pre-diction windows, as well as for the downscaled binary prediction of impending failures. In this case study, the multi-class feed-forward neural network model achieved the overall best results.}, author = {Züfle, Marwin and Agne, Joachim and Grohmann, Johannes and Dörtoluk, Ibrahim and Kounev, Samuel}, booktitle = {Proceedings of the 21st IEEE IES International Conference on Industrial Informatics}, month = {July}, publisher = {IEEE}, series = {INDIN'21}, title = {A Predictive Maintenance Methodology: Predicting the Time-to-Failure of Machines in Industry 4.0}, year = 2021 }

  %0 Conference Paper %1 ZuAgGrDoKo-INDIN2021-PdM %A Züfle, Marwin %A Agne, Joachim %A Grohmann, Johannes %A Dörtoluk, Ibrahim %A Kounev, Samuel %B Proceedings of the 21st IEEE IES International Conference on Industrial Informatics %D 2021 %I IEEE %T A Predictive Maintenance Methodology: Predicting the Time-to-Failure of Machines in Industry 4.0 %U https://ieeexplore.ieee.org/abstract/document/9557387 %X Predictive maintenance is an essential aspect of the concept of Industry 4.0. In contrast to previous maintenance strategies, which plan repairs based on periodic schedules or threshold values, predictive maintenance is normally based on estimating the time-to-failure of machines. Thus, predictive maintenance enables a more efficient and effective maintenance approach. Although much research has already been done on time-to-failure prediction, most existing works provide only specialized approaches for specific machines. In most cases, these are either rotary machines (i.e., bearings) or lithium-ion batteries. To bridge the gap to a more general time-to-failure prediction, we propose a generic end-to-end predictive maintenance methodology for the time-to-failure prediction of industrial machines. Our methodology exhibits a number of novel aspects including a universally applicable method for feature extraction based on different types of sensor data, well-known feature transformation and selection techniques, adjustable target class assignment based on fault records with three different labeling strategies, and the training of multiple state-of-the-art machine learning classification models including hyperparameter optimization. We evaluated our time-to-failure prediction methodology in a real-world case study consisting of monitoring data gathered over several years from a large industrial press. The results demonstrated the effectiveness of the proposed methodology for six different time-to-failure pre-diction windows, as well as for the downscaled binary prediction of impending failures. In this case study, the multi-class feed-forward neural network model achieved the overall best results.
• 
  SuanMing: Explainable Prediction of Performance Degradations in Microservice Applications. Grohmann, Johannes; Straesser, Martin; Chalbani, Avi; Eismann, Simon; Arian, Yair; Herbst, Nikolas; Peretz, Noam; Kounev, Samuel; in Proceedings of the 12th ACM/SPEC International Conference on Performance Engineering (ICPE)ACM, New York, NY, USA.

  Acceptance Rate: 29%
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Application performance management (APM) tools are useful to observe the performance properties of an application during production. However, APM is normally purely reactive, that is, it can only report about current or past performance degradation. Although some approaches capable of predictive application monitoring have been proposed, they can only report a predicted degradation but cannot explain its root-cause, making it hard to prevent the expected degradation. In this paper, we present SuanMing---a framework for predicting performance degradation of microservice applications running in cloud environments. SuanMing is able to predict future root causes for anticipated performance degradations and therefore aims at preventing performance degradations before they actually occur. We evaluate SuanMing on two realistic microservice applications, TeaStore and TrainTicket, and we show that our approach is able to predict and pinpoint performance degradations with an accuracy of over 90\%.

  @inproceedings{GrStChEiArHePeKo2021-ICPE, abstract = {Application performance management (APM) tools are useful to observe the performance properties of an application during production. However, APM is normally purely reactive, that is, it can only report about current or past performance degradation. Although some approaches capable of predictive application monitoring have been proposed, they can only report a predicted degradation but cannot explain its root-cause, making it hard to prevent the expected degradation. In this paper, we present SuanMing---a framework for predicting performance degradation of microservice applications running in cloud environments. SuanMing is able to predict future root causes for anticipated performance degradations and therefore aims at preventing performance degradations before they actually occur. We evaluate SuanMing on two realistic microservice applications, TeaStore and TrainTicket, and we show that our approach is able to predict and pinpoint performance degradations with an accuracy of over 90\%.}, address = {New York, NY, USA}, author = {Grohmann, Johannes and Straesser, Martin and Chalbani, Avi and Eismann, Simon and Arian, Yair and Herbst, Nikolas and Peretz, Noam and Kounev, Samuel}, booktitle = {Proceedings of the 12th ACM/SPEC International Conference on Performance Engineering (ICPE)}, month = {April}, note = {Acceptance Rate: 29%}, organization = {ACM}, title = {SuanMing: Explainable Prediction of Performance Degradations in Microservice Applications}, year = 2021 }

  %0 Conference Paper %1 GrStChEiArHePeKo2021-ICPE %A Grohmann, Johannes %A Straesser, Martin %A Chalbani, Avi %A Eismann, Simon %A Arian, Yair %A Herbst, Nikolas %A Peretz, Noam %A Kounev, Samuel %B Proceedings of the 12th ACM/SPEC International Conference on Performance Engineering (ICPE) %C New York, NY, USA %D 2021 %T SuanMing: Explainable Prediction of Performance Degradations in Microservice Applications %U https://doi.org/10.1145/3427921.3450248 %X Application performance management (APM) tools are useful to observe the performance properties of an application during production. However, APM is normally purely reactive, that is, it can only report about current or past performance degradation. Although some approaches capable of predictive application monitoring have been proposed, they can only report a predicted degradation but cannot explain its root-cause, making it hard to prevent the expected degradation. In this paper, we present SuanMing---a framework for predicting performance degradation of microservice applications running in cloud environments. SuanMing is able to predict future root causes for anticipated performance degradations and therefore aims at preventing performance degradations before they actually occur. We evaluate SuanMing on two realistic microservice applications, TeaStore and TrainTicket, and we show that our approach is able to predict and pinpoint performance degradations with an accuracy of over 90\%.
• 
  Libra: A Benchmark for Time Series Forecasting Methods. Bauer, Andr{\’e}; Z{\"u}fle, Marwin; Eismann, Simon; Grohmann, Johannes; Herbst, Nikolas; Kounev, Samuel; in Proceedings of the 12th ACM/SPEC International Conference on Performance Engineering (ICPE)ACM, New York, NY, USA.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  In many areas of decision making, forecasting is an essential pillar. Consequently, there are many different forecasting methods. According to the "No-Free-Lunch Theorem", there is no single forecasting method that performs best for all time series. In other words, each method has its advantages and disadvantages depending on the specific use case. Therefore, the choice of the forecasting method remains a mandatory expert task. However, expert knowledge cannot be fully automated. To establish a level playing field for evaluating the performance of time series forecasting methods in a broad setting, we propose Libra, a forecasting benchmark that automatically evaluates and ranks forecasting methods based on their performance in a diverse set of evaluation scenarios. The benchmark comprises four different use cases, each covering 100 heterogeneous time series taken from different domains. The data set was assembled from publicly available time series and was designed to exhibit much higher diversity than existing forecasting competitions. Based on this benchmark, we perform a comprehensive evaluation to compare different existing time series forecasting methods.

  @inproceedings{bauer2021benchmark, abstract = {In many areas of decision making, forecasting is an essential pillar. Consequently, there are many different forecasting methods. According to the "No-Free-Lunch Theorem", there is no single forecasting method that performs best for all time series. In other words, each method has its advantages and disadvantages depending on the specific use case. Therefore, the choice of the forecasting method remains a mandatory expert task. However, expert knowledge cannot be fully automated. To establish a level playing field for evaluating the performance of time series forecasting methods in a broad setting, we propose Libra, a forecasting benchmark that automatically evaluates and ranks forecasting methods based on their performance in a diverse set of evaluation scenarios. The benchmark comprises four different use cases, each covering 100 heterogeneous time series taken from different domains. The data set was assembled from publicly available time series and was designed to exhibit much higher diversity than existing forecasting competitions. Based on this benchmark, we perform a comprehensive evaluation to compare different existing time series forecasting methods.}, address = {New York, NY, USA}, author = {Bauer, Andr{\'e} and Z{\"u}fle, Marwin and Eismann, Simon and Grohmann, Johannes and Herbst, Nikolas and Kounev, Samuel}, booktitle = {Proceedings of the 12th ACM/SPEC International Conference on Performance Engineering (ICPE)}, month = {April}, organization = {ACM}, title = {Libra: A Benchmark for Time Series Forecasting Methods}, year = 2021 }

  %0 Conference Paper %1 bauer2021benchmark %A Bauer, Andr{\'e} %A Z{\"u}fle, Marwin %A Eismann, Simon %A Grohmann, Johannes %A Herbst, Nikolas %A Kounev, Samuel %B Proceedings of the 12th ACM/SPEC International Conference on Performance Engineering (ICPE) %C New York, NY, USA %D 2021 %T Libra: A Benchmark for Time Series Forecasting Methods %U https://dl.acm.org/doi/10.1145/3427921.3450241 %X In many areas of decision making, forecasting is an essential pillar. Consequently, there are many different forecasting methods. According to the "No-Free-Lunch Theorem", there is no single forecasting method that performs best for all time series. In other words, each method has its advantages and disadvantages depending on the specific use case. Therefore, the choice of the forecasting method remains a mandatory expert task. However, expert knowledge cannot be fully automated. To establish a level playing field for evaluating the performance of time series forecasting methods in a broad setting, we propose Libra, a forecasting benchmark that automatically evaluates and ranks forecasting methods based on their performance in a diverse set of evaluation scenarios. The benchmark comprises four different use cases, each covering 100 heterogeneous time series taken from different domains. The data set was assembled from publicly available time series and was designed to exhibit much higher diversity than existing forecasting competitions. Based on this benchmark, we perform a comprehensive evaluation to compare different existing time series forecasting methods.

• 
  Predicting the Costs of Serverless Workflows. Eismann, Simon; Grohmann, Johannes; van Eyk, Erwin; Herbst, Nikolas; Kounev, Samuel; in Proceedings of the 2020 ACM/SPEC International Conference on Performance Engineering (ICPE) 265–276. Association for Computing Machinery (ACM), New York, NY, USA.

  {Acceptance Rate: 23.4% (15/64)}
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Function-as-a-Service (FaaS) platforms enable users to run arbitrary functions without being concerned about operational issues, while only paying for the consumed resources. Individual functions are often composed into workflows for complex tasks. However, the pay-per-use model and nontransparent reporting by cloud providers make it challenging to estimate the expected cost of a workflow, which prevents informed business decisions. Existing cost-estimation approaches assume a static response time for the serverless functions, without taking input parameters into account. In this paper, we propose a methodology for the cost prediction of serverless workflows consisting of input-parameter sensitive function models and a monte-carlo simulation of an abstract workflow model. Our approach enables workflow designers to predict, compare, and optimize the expected costs and performance of a planned workflow, which currently requires time-intensive experimentation. In our evaluation, we show that our approach can predict the response time and output parameters of a function based on its input parameters with an accuracy of 96.1%. In a case study with two audio-processing workflows, our approach predicts the costs of the two workflows with an accuracy of 96.2%.

  @inproceedings{EiGrEyHeKo2020-ICPE-ServerlessWorkflows, abstract = {Function-as-a-Service (FaaS) platforms enable users to run arbitrary functions without being concerned about operational issues, while only paying for the consumed resources. Individual functions are often composed into workflows for complex tasks. However, the pay-per-use model and nontransparent reporting by cloud providers make it challenging to estimate the expected cost of a workflow, which prevents informed business decisions. Existing cost-estimation approaches assume a static response time for the serverless functions, without taking input parameters into account. In this paper, we propose a methodology for the cost prediction of serverless workflows consisting of input-parameter sensitive function models and a monte-carlo simulation of an abstract workflow model. Our approach enables workflow designers to predict, compare, and optimize the expected costs and performance of a planned workflow, which currently requires time-intensive experimentation. In our evaluation, we show that our approach can predict the response time and output parameters of a function based on its input parameters with an accuracy of 96.1%. In a case study with two audio-processing workflows, our approach predicts the costs of the two workflows with an accuracy of 96.2%.}, address = {New York, NY, USA}, author = {Eismann, Simon and Grohmann, Johannes and van Eyk, Erwin and Herbst, Nikolas and Kounev, Samuel}, booktitle = {Proceedings of the 2020 ACM/SPEC International Conference on Performance Engineering (ICPE)}, month = {April}, note = {{Acceptance Rate: 23.4% (15/64)}}, pages = {265--276}, publisher = {Association for Computing Machinery (ACM)}, series = {ICPE '20}, title = {Predicting the Costs of Serverless Workflows}, year = 2020 }

  %0 Conference Paper %1 EiGrEyHeKo2020-ICPE-ServerlessWorkflows %A Eismann, Simon %A Grohmann, Johannes %A van Eyk, Erwin %A Herbst, Nikolas %A Kounev, Samuel %B Proceedings of the 2020 ACM/SPEC International Conference on Performance Engineering (ICPE) %C New York, NY, USA %D 2020 %I Association for Computing Machinery (ACM) %P 265--276 %T Predicting the Costs of Serverless Workflows %U https://doi.org/10.1145/3358960.3379133 %X Function-as-a-Service (FaaS) platforms enable users to run arbitrary functions without being concerned about operational issues, while only paying for the consumed resources. Individual functions are often composed into workflows for complex tasks. However, the pay-per-use model and nontransparent reporting by cloud providers make it challenging to estimate the expected cost of a workflow, which prevents informed business decisions. Existing cost-estimation approaches assume a static response time for the serverless functions, without taking input parameters into account. In this paper, we propose a methodology for the cost prediction of serverless workflows consisting of input-parameter sensitive function models and a monte-carlo simulation of an abstract workflow model. Our approach enables workflow designers to predict, compare, and optimize the expected costs and performance of a planned workflow, which currently requires time-intensive experimentation. In our evaluation, we show that our approach can predict the response time and output parameters of a function based on its input parameters with an accuracy of 96.1%. In a case study with two audio-processing workflows, our approach predicts the costs of the two workflows with an accuracy of 96.2%.
• 
  {To Fail Or Not To Fail: Predicting Hard Disk Drive Failure Time Windows}. Z{\"u}fle, Marwin; Krupitzer, Christian; Erhard, Florian; Grohmann, Johannes; Kounev, Samuel; in Proceedings of the 20th International GI/ITG Conference on Measurement, Modelling and Evaluation of Computing Systems 19–36. Springer, Cham.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Due to the increasing size of today's data centers as well as the expectation of 24/7 availability, the complexity in the administration of hardware continuously icreases. Techniques as the Self-Monitoring, Analysis, and Reporting Technology (S.M.A.R.T.) support the monitoring of the hardware. However, those techniques often lack algorithms for intelligent data analytics. Especially, the integration of machine learning to identify potential failures in advance seems to be promising to reduce administration overhead. In this work, we present three machine learning approaches to (i) identify imminent failures, (ii) predict time windows for failures, as well as (iii) predict the exact time-to-failure. In a case study with real data from 369 hard disks, we achieve an F1-score of up to 98.0% and 97.6% for predicting potential failures with two or multiple time windows, respectively, and a hit rate of 84.9% (with a mean absolute error of 4.5 hours) for predicting the time-to-failure.

  @inproceedings{ZuKrErGrKo-MMB2020-ToFailOrNotToFail, abstract = {Due to the increasing size of today's data centers as well as the expectation of 24/7 availability, the complexity in the administration of hardware continuously icreases. Techniques as the Self-Monitoring, Analysis, and Reporting Technology (S.M.A.R.T.) support the monitoring of the hardware. However, those techniques often lack algorithms for intelligent data analytics. Especially, the integration of machine learning to identify potential failures in advance seems to be promising to reduce administration overhead. In this work, we present three machine learning approaches to (i) identify imminent failures, (ii) predict time windows for failures, as well as (iii) predict the exact time-to-failure. In a case study with real data from 369 hard disks, we achieve an F1-score of up to 98.0% and 97.6% for predicting potential failures with two or multiple time windows, respectively, and a hit rate of 84.9% (with a mean absolute error of 4.5 hours) for predicting the time-to-failure.}, address = {Cham}, author = {Z{\"u}fle, Marwin and Krupitzer, Christian and Erhard, Florian and Grohmann, Johannes and Kounev, Samuel}, booktitle = {Proceedings of the 20th International GI/ITG Conference on Measurement, Modelling and Evaluation of Computing Systems}, month = {March}, pages = {19--36}, publisher = {Springer}, series = {MMB 2020}, title = {{To Fail Or Not To Fail: Predicting Hard Disk Drive Failure Time Windows}}, year = 2020 }

  %0 Conference Paper %1 ZuKrErGrKo-MMB2020-ToFailOrNotToFail %A Z{\"u}fle, Marwin %A Krupitzer, Christian %A Erhard, Florian %A Grohmann, Johannes %A Kounev, Samuel %B Proceedings of the 20th International GI/ITG Conference on Measurement, Modelling and Evaluation of Computing Systems %C Cham %D 2020 %I Springer %P 19--36 %T {To Fail Or Not To Fail: Predicting Hard Disk Drive Failure Time Windows} %U https://doi.org/10.1007/978-3-030-43024-5_2 %X Due to the increasing size of today's data centers as well as the expectation of 24/7 availability, the complexity in the administration of hardware continuously icreases. Techniques as the Self-Monitoring, Analysis, and Reporting Technology (S.M.A.R.T.) support the monitoring of the hardware. However, those techniques often lack algorithms for intelligent data analytics. Especially, the integration of machine learning to identify potential failures in advance seems to be promising to reduce administration overhead. In this work, we present three machine learning approaches to (i) identify imminent failures, (ii) predict time windows for failures, as well as (iii) predict the exact time-to-failure. In a case study with real data from 369 hard disks, we achieve an F1-score of up to 98.0% and 97.6% for predicting potential failures with two or multiple time windows, respectively, and a hit rate of 84.9% (with a mean absolute error of 4.5 hours) for predicting the time-to-failure.
• 
  {Model-based Performance Predictions for SDN-based Networks: A Case Study}. Herrnleben, Stefan; Rygielski, Piotr; Grohmann, Johannes; Eismann, Simon; Hossfeld, Tobias; Kounev, Samuel; in Proceedings of the 20th International GI/ITG Conference on Measurement, Modelling and Evaluation of Computing SystemsSpringer, Cham.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Emerging paradigms for network virtualization like Software-Defined Networking (SDN) and Network Functions Virtualization (NFV) form new challenges for accurate performance modeling and analysis tools. Therefore, performance modeling and prediction approaches that support SDN or NFV technologies help system operators to analyze the performance of a data center and its corresponding network. The Descartes Network Infrastructures (DNI) offers a high-level descriptive language to model SDN-based networks, which can be transformed into various predictive modeling formalisms. However, these modeling concepts have not yet been evaluated in a realistic scenario. In this paper, we present an extensive case study evaluating the DNI modeling capabilities, the transformations to predictive models, and the performance prediction using the OMNeT++ and SimQPN simulation frameworks. We present five realistic scenarios of a content distribution network (CDN), compare the performance predictions with real-world measurements, and discuss modeling gaps and calibration issues causing mispredictions in some scenarios.

  @inproceedings{HeRyGrEiHoKo-MMB2020-Model-based-SDN-Performance, abstract = {Emerging paradigms for network virtualization like Software-Defined Networking (SDN) and Network Functions Virtualization (NFV) form new challenges for accurate performance modeling and analysis tools. Therefore, performance modeling and prediction approaches that support SDN or NFV technologies help system operators to analyze the performance of a data center and its corresponding network. The Descartes Network Infrastructures (DNI) offers a high-level descriptive language to model SDN-based networks, which can be transformed into various predictive modeling formalisms. However, these modeling concepts have not yet been evaluated in a realistic scenario. In this paper, we present an extensive case study evaluating the DNI modeling capabilities, the transformations to predictive models, and the performance prediction using the OMNeT++ and SimQPN simulation frameworks. We present five realistic scenarios of a content distribution network (CDN), compare the performance predictions with real-world measurements, and discuss modeling gaps and calibration issues causing mispredictions in some scenarios.}, address = {Cham}, author = {Herrnleben, Stefan and Rygielski, Piotr and Grohmann, Johannes and Eismann, Simon and Hossfeld, Tobias and Kounev, Samuel}, booktitle = {Proceedings of the 20th International GI/ITG Conference on Measurement, Modelling and Evaluation of Computing Systems}, month = {March}, publisher = {Springer}, series = {MMB 2020}, title = {{Model-based Performance Predictions for SDN-based Networks: A Case Study}}, year = 2020 }

  %0 Conference Paper %1 HeRyGrEiHoKo-MMB2020-Model-based-SDN-Performance %A Herrnleben, Stefan %A Rygielski, Piotr %A Grohmann, Johannes %A Eismann, Simon %A Hossfeld, Tobias %A Kounev, Samuel %B Proceedings of the 20th International GI/ITG Conference on Measurement, Modelling and Evaluation of Computing Systems %C Cham %D 2020 %I Springer %T {Model-based Performance Predictions for SDN-based Networks: A Case Study} %U https://doi.org/10.1007/978-3-030-43024-5_6 %X Emerging paradigms for network virtualization like Software-Defined Networking (SDN) and Network Functions Virtualization (NFV) form new challenges for accurate performance modeling and analysis tools. Therefore, performance modeling and prediction approaches that support SDN or NFV technologies help system operators to analyze the performance of a data center and its corresponding network. The Descartes Network Infrastructures (DNI) offers a high-level descriptive language to model SDN-based networks, which can be transformed into various predictive modeling formalisms. However, these modeling concepts have not yet been evaluated in a realistic scenario. In this paper, we present an extensive case study evaluating the DNI modeling capabilities, the transformations to predictive models, and the performance prediction using the OMNeT++ and SimQPN simulation frameworks. We present five realistic scenarios of a content distribution network (CDN), compare the performance predictions with real-world measurements, and discuss modeling gaps and calibration issues causing mispredictions in some scenarios.
• 
  An IoT Network Emulator for Analyzing the Influence of Varying Network Quality. Herrnleben, Stefan; Ailabouni, Rudy; Grohmann, Johannes; Prantl, Thomas; Krupitzer, Christian; Kounev, Samuel; in Proceedings of the 12th EAI International Conference on Simulation Tools and Techniques (SIMUtools)
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  IoT devices often communicate over wireless or cellular networks with varying connection quality. These fluctuations are caused, among others, by the free-space path loss (FSPL), buildings, topological obstacles, weather, and mobility of the receiver. Varying signal quality affects bandwidth, transmission delays, packet loss, and jitter. Mobile IoT applications exposed to varying connection characteristics have to handle such variations and take them into account during development and testing. However, tests in real mobile networks are complex and challenging to reproduce. Therefore, network emulators can be used to simulate the behavior of real-world networks by adding artificial disturbance. However, existing network emulators often require a lot of technical knowledge and complex setup. Integrating such emulators into automated software testing pipelines could be a challenging task. In this paper, we propose a framework for emulating IoT networks with varying quality characteristics. An existing base emulator is used and integrated into our framework enabling the user to utilize it without extensive network expertise and configuration effort. The evaluation proves that our framework can simulate a variety of different network quality characteristics as well as emulating real-world network traces.

  @inproceedings{HeAiGrPrKrKo2020-SIMUtools-IoT-Network-Emulator, abstract = {IoT devices often communicate over wireless or cellular networks with varying connection quality. These fluctuations are caused, among others, by the free-space path loss (FSPL), buildings, topological obstacles, weather, and mobility of the receiver. Varying signal quality affects bandwidth, transmission delays, packet loss, and jitter. Mobile IoT applications exposed to varying connection characteristics have to handle such variations and take them into account during development and testing. However, tests in real mobile networks are complex and challenging to reproduce. Therefore, network emulators can be used to simulate the behavior of real-world networks by adding artificial disturbance. However, existing network emulators often require a lot of technical knowledge and complex setup. Integrating such emulators into automated software testing pipelines could be a challenging task. In this paper, we propose a framework for emulating IoT networks with varying quality characteristics. An existing base emulator is used and integrated into our framework enabling the user to utilize it without extensive network expertise and configuration effort. The evaluation proves that our framework can simulate a variety of different network quality characteristics as well as emulating real-world network traces.}, author = {Herrnleben, Stefan and Ailabouni, Rudy and Grohmann, Johannes and Prantl, Thomas and Krupitzer, Christian and Kounev, Samuel}, booktitle = {Proceedings of the 12th EAI International Conference on Simulation Tools and Techniques (SIMUtools)}, month = {August}, series = {SIMUtools 2020}, title = {An IoT Network Emulator for Analyzing the Influence of Varying Network Quality}, year = 2020 }

  %0 Conference Paper %1 HeAiGrPrKrKo2020-SIMUtools-IoT-Network-Emulator %A Herrnleben, Stefan %A Ailabouni, Rudy %A Grohmann, Johannes %A Prantl, Thomas %A Krupitzer, Christian %A Kounev, Samuel %B Proceedings of the 12th EAI International Conference on Simulation Tools and Techniques (SIMUtools) %D 2020 %T An IoT Network Emulator for Analyzing the Influence of Varying Network Quality %U https://doi.org/10.1007/978-3-030-72795-6_47 %X IoT devices often communicate over wireless or cellular networks with varying connection quality. These fluctuations are caused, among others, by the free-space path loss (FSPL), buildings, topological obstacles, weather, and mobility of the receiver. Varying signal quality affects bandwidth, transmission delays, packet loss, and jitter. Mobile IoT applications exposed to varying connection characteristics have to handle such variations and take them into account during development and testing. However, tests in real mobile networks are complex and challenging to reproduce. Therefore, network emulators can be used to simulate the behavior of real-world networks by adding artificial disturbance. However, existing network emulators often require a lot of technical knowledge and complex setup. Integrating such emulators into automated software testing pipelines could be a challenging task. In this paper, we propose a framework for emulating IoT networks with varying quality characteristics. An existing base emulator is used and integrated into our framework enabling the user to utilize it without extensive network expertise and configuration effort. The evaluation proves that our framework can simulate a variety of different network quality characteristics as well as emulating real-world network traces.
• 
  Evaluating the Performance of a State-of-the-Art Group-oriented Encryption Scheme for Dynamic Groups in an IoT Scenario. Prantl, Thomas; Ten, Peter; Iffländer, Lukas; Dmitrenko, Alexandra; Kounev, Samuel; Krupitzer, Christian; in 2020 IEEE 28th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS)

  Acceptance Rate: 27\%
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  New emerging technologies, such as autonomous driving, intelligent buildings, and smart cities, are promising to revolutionize user experience and offer new services. The world has to undergo large scale deployment of billions of things — cost-efficient intelligent sensors that will be interconnected into extensive networks and will collect and supply data to intelligent algorithms — to make it happen. To date, however, it is challenging to secure such an infrastructure for many-fold reasons, such as resource constraints of things, large scale deployment, manyto-many communication patterns, and dynamically changing communication groups. All these factors rule out most of the state-of-the-art encryption and key-management techniques. Group encryption algorithms are well-suitable for many-tomany communication patterns typical for IoT networks, and many of them can deal with dynamic groups. There are, however, very few constructions that could potentially fulfill the computational and storage constraints of IoT devices while providing sufficient scalability for large networks. The promising candidates, such as construction by Nishat et al. [1], were not evaluated using IoT platforms and under constraints typical for IoT networks. In this paper, we aim to fill this gap and present the evaluation of a state-of-the-art group-oriented encryption scheme by Nishat et al. to identify its applicability to IoT systems. In detail, we provide a measurement workflow, a revised version of the approach, and describe a reproducible hardware testbed. Using this evaluation environment, we analyze the performance of the encryption scheme in a typical IoT scenario from a group member perspective. The results show that all calculation times can be assumed to be constant and are always below 2 seconds. The memory requirement for permanent parameters can also be considered to be constant and are below 8.5 kbit in each case. However, the information that has to be stored temporarily for group updates has turned out to be the bottleneck of the scheme, since their memory requirements increase linearly with the group size.

  @inproceedings{PrTeIfDmKo2020-MASCOTS-GroupEncryption, abstract = {New emerging technologies, such as autonomous driving, intelligent buildings, and smart cities, are promising to revolutionize user experience and offer new services. The world has to undergo large scale deployment of billions of things — cost-efficient intelligent sensors that will be interconnected into extensive networks and will collect and supply data to intelligent algorithms — to make it happen. To date, however, it is challenging to secure such an infrastructure for many-fold reasons, such as resource constraints of things, large scale deployment, manyto-many communication patterns, and dynamically changing communication groups. All these factors rule out most of the state-of-the-art encryption and key-management techniques. Group encryption algorithms are well-suitable for many-tomany communication patterns typical for IoT networks, and many of them can deal with dynamic groups. There are, however, very few constructions that could potentially fulfill the computational and storage constraints of IoT devices while providing sufficient scalability for large networks. The promising candidates, such as construction by Nishat et al. [1], were not evaluated using IoT platforms and under constraints typical for IoT networks. In this paper, we aim to fill this gap and present the evaluation of a state-of-the-art group-oriented encryption scheme by Nishat et al. to identify its applicability to IoT systems. In detail, we provide a measurement workflow, a revised version of the approach, and describe a reproducible hardware testbed. Using this evaluation environment, we analyze the performance of the encryption scheme in a typical IoT scenario from a group member perspective. The results show that all calculation times can be assumed to be constant and are always below 2 seconds. The memory requirement for permanent parameters can also be considered to be constant and are below 8.5 kbit in each case. However, the information that has to be stored temporarily for group updates has turned out to be the bottleneck of the scheme, since their memory requirements increase linearly with the group size.}, author = {Prantl, Thomas and Ten, Peter and Iffländer, Lukas and Dmitrenko, Alexandra and Kounev, Samuel and Krupitzer, Christian}, booktitle = {2020 IEEE 28th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS)}, month = {November}, note = {Acceptance Rate: 27\%}, series = {MASCOTS '20}, title = {Evaluating the Performance of a State-of-the-Art Group-oriented Encryption Scheme for Dynamic Groups in an IoT Scenario}, year = 2020 }

  %0 Conference Paper %1 PrTeIfDmKo2020-MASCOTS-GroupEncryption %A Prantl, Thomas %A Ten, Peter %A Iffländer, Lukas %A Dmitrenko, Alexandra %A Kounev, Samuel %A Krupitzer, Christian %B 2020 IEEE 28th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS) %D 2020 %T Evaluating the Performance of a State-of-the-Art Group-oriented Encryption Scheme for Dynamic Groups in an IoT Scenario %X New emerging technologies, such as autonomous driving, intelligent buildings, and smart cities, are promising to revolutionize user experience and offer new services. The world has to undergo large scale deployment of billions of things — cost-efficient intelligent sensors that will be interconnected into extensive networks and will collect and supply data to intelligent algorithms — to make it happen. To date, however, it is challenging to secure such an infrastructure for many-fold reasons, such as resource constraints of things, large scale deployment, manyto-many communication patterns, and dynamically changing communication groups. All these factors rule out most of the state-of-the-art encryption and key-management techniques. Group encryption algorithms are well-suitable for many-tomany communication patterns typical for IoT networks, and many of them can deal with dynamic groups. There are, however, very few constructions that could potentially fulfill the computational and storage constraints of IoT devices while providing sufficient scalability for large networks. The promising candidates, such as construction by Nishat et al. [1], were not evaluated using IoT platforms and under constraints typical for IoT networks. In this paper, we aim to fill this gap and present the evaluation of a state-of-the-art group-oriented encryption scheme by Nishat et al. to identify its applicability to IoT systems. In detail, we provide a measurement workflow, a revised version of the approach, and describe a reproducible hardware testbed. Using this evaluation environment, we analyze the performance of the encryption scheme in a typical IoT scenario from a group member perspective. The results show that all calculation times can be assumed to be constant and are always below 2 seconds. The memory requirement for permanent parameters can also be considered to be constant and are below 8.5 kbit in each case. However, the information that has to be stored temporarily for group updates has turned out to be the bottleneck of the scheme, since their memory requirements increase linearly with the group size.
• 
  Baloo: Measuring and Modeling the Performance Configurations of Distributed DBMS. Grohmann, Johannes; Seybold, Daniel; Eismann, Simon; Leznik, Mark; Kounev, Samuel; Domaschka, Jörg; in 2020 28th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS) 1–8. IEEE.

  Acceptance Rate: 27%
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Correctly configuring a distributed database management system (DBMS) deployed in a cloud environment for maximizing performance poses many challenges to operators. Even if the entire configuration spectrum could be measured directly, which is often infeasible due to the multitude of parameters, single measurements are subject to random variations and need to be repeated multiple times. In this work, we propose Baloo, a framework for systematically measuring and modeling different performance-relevant configurations of distributed DBMS in cloud environments. Baloo dynamically estimates the required number of configurations, as well as the number of required measurement repetitions per configuration based on a desired target accuracy. We evaluate Baloo based on a data set consisting of 900 DBMS configuration measurements conducted in our private cloud setup. Our evaluation shows that the highly configurable framework is able to achieve a prediction error of up to 12% while saving 80% of the measurement effort. We also publish all code and the acquired data set to foster future research.

  @inproceedings{GrSeEiLeKoDo2020-MASCOTS-DBMSPerformance, abstract = {Correctly configuring a distributed database management system (DBMS) deployed in a cloud environment for maximizing performance poses many challenges to operators. Even if the entire configuration spectrum could be measured directly, which is often infeasible due to the multitude of parameters, single measurements are subject to random variations and need to be repeated multiple times. In this work, we propose Baloo, a framework for systematically measuring and modeling different performance-relevant configurations of distributed DBMS in cloud environments. Baloo dynamically estimates the required number of configurations, as well as the number of required measurement repetitions per configuration based on a desired target accuracy. We evaluate Baloo based on a data set consisting of 900 DBMS configuration measurements conducted in our private cloud setup. Our evaluation shows that the highly configurable framework is able to achieve a prediction error of up to 12% while saving 80% of the measurement effort. We also publish all code and the acquired data set to foster future research.}, author = {Grohmann, Johannes and Seybold, Daniel and Eismann, Simon and Leznik, Mark and Kounev, Samuel and Domaschka, Jörg}, booktitle = {2020 28th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS)}, month = {November}, note = {Acceptance Rate: 27%}, pages = {1--8}, publisher = {IEEE}, series = {MASCOTS '20}, title = {Baloo: Measuring and Modeling the Performance Configurations of Distributed DBMS}, year = 2020 }

  %0 Conference Paper %1 GrSeEiLeKoDo2020-MASCOTS-DBMSPerformance %A Grohmann, Johannes %A Seybold, Daniel %A Eismann, Simon %A Leznik, Mark %A Kounev, Samuel %A Domaschka, Jörg %B 2020 28th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS) %D 2020 %I IEEE %P 1--8 %T Baloo: Measuring and Modeling the Performance Configurations of Distributed DBMS %U https://doi.org/10.1109/MASCOTS50786.2020.9285960 %X Correctly configuring a distributed database management system (DBMS) deployed in a cloud environment for maximizing performance poses many challenges to operators. Even if the entire configuration spectrum could be measured directly, which is often infeasible due to the multitude of parameters, single measurements are subject to random variations and need to be repeated multiple times. In this work, we propose Baloo, a framework for systematically measuring and modeling different performance-relevant configurations of distributed DBMS in cloud environments. Baloo dynamically estimates the required number of configurations, as well as the number of required measurement repetitions per configuration based on a desired target accuracy. We evaluate Baloo based on a data set consisting of 900 DBMS configuration measurements conducted in our private cloud setup. Our evaluation shows that the highly configurable framework is able to achieve a prediction error of up to 12% while saving 80% of the measurement effort. We also publish all code and the acquired data set to foster future research.
• 
  Quantifying measurement quality and load distribution in Tor. Greubel, Andre; Pohl, Steffen; Kounev, Samuel; in Proceedings of the 36th Annual Computer Security Applications Conference (ACSAC 2020)
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Tor is a widely used anonymization network. Traffic is routed over different relay nodes to conceal the communication partners. However, if a single relay handles too much traffic, de-anonymization attacks are possible. The Tor Load Balancing Mechanism (TLBM) is responsible for balanced and secure load distribution. It must verify that relays cannot attract more traffic than they should by lying about their self-reported bandwidth. This work shows that the current bandwidth measurement method used for bandwidth verification is not suitable to verify the bandwidth of many relays. Most importantly, multiple measurements of high-bandwidth relays are uncorrelated to each other. Furthermore, we analyze the current load distribution in Tor. We show that the current load distribution reduces the resources necessary for several large-scale de-anonymization attacks by more than 80\%. Additionally, as Tor favors fast relays during path selection, verifiable relays only handle a small fraction of Tor’s traffic. More precisely, we show that only 7.21\% of all paths consist of entry and exit relays verifiable by measurements. We discuss these results’ security implications and argue that future TLBM research should focus at least as much on secure load distribution as on high traffic performance.

  @article{greubel2020quantifying, abstract = {Tor is a widely used anonymization network. Traffic is routed over different relay nodes to conceal the communication partners. However, if a single relay handles too much traffic, de-anonymization attacks are possible. The Tor Load Balancing Mechanism (TLBM) is responsible for balanced and secure load distribution. It must verify that relays cannot attract more traffic than they should by lying about their self-reported bandwidth. This work shows that the current bandwidth measurement method used for bandwidth verification is not suitable to verify the bandwidth of many relays. Most importantly, multiple measurements of high-bandwidth relays are uncorrelated to each other. Furthermore, we analyze the current load distribution in Tor. We show that the current load distribution reduces the resources necessary for several large-scale de-anonymization attacks by more than 80\%. Additionally, as Tor favors fast relays during path selection, verifiable relays only handle a small fraction of Tor’s traffic. More precisely, we show that only 7.21\% of all paths consist of entry and exit relays verifiable by measurements. We discuss these results’ security implications and argue that future TLBM research should focus at least as much on secure load distribution as on high traffic performance.}, author = {Greubel, Andre and Pohl, Steffen and Kounev, Samuel}, journal = {Proceedings of the 36th Annual Computer Security Applications Conference (ACSAC 2020)}, title = {Quantifying measurement quality and load distribution in Tor}, year = 2020 }

  %0 Journal Article %1 greubel2020quantifying %A Greubel, Andre %A Pohl, Steffen %A Kounev, Samuel %D 2020 %J Proceedings of the 36th Annual Computer Security Applications Conference (ACSAC 2020) %T Quantifying measurement quality and load distribution in Tor %X Tor is a widely used anonymization network. Traffic is routed over different relay nodes to conceal the communication partners. However, if a single relay handles too much traffic, de-anonymization attacks are possible. The Tor Load Balancing Mechanism (TLBM) is responsible for balanced and secure load distribution. It must verify that relays cannot attract more traffic than they should by lying about their self-reported bandwidth. This work shows that the current bandwidth measurement method used for bandwidth verification is not suitable to verify the bandwidth of many relays. Most importantly, multiple measurements of high-bandwidth relays are uncorrelated to each other. Furthermore, we analyze the current load distribution in Tor. We show that the current load distribution reduces the resources necessary for several large-scale de-anonymization attacks by more than 80\%. Additionally, as Tor favors fast relays during path selection, verifiable relays only handle a small fraction of Tor’s traffic. More precisely, we show that only 7.21\% of all paths consist of entry and exit relays verifiable by measurements. We discuss these results’ security implications and argue that future TLBM research should focus at least as much on secure load distribution as on high traffic performance.

• 
  Monitorless: Predicting Performance Degradation in Cloud Applications with Machine Learning. Grohmann, Johannes; Nicholson, Patrick K.; Iglesias, Jesus Omana; Kounev, Samuel; Lugones, Diego; in Proceedings of the 20th International Middleware Conference 149–162. Association for Computing Machinery (ACM), New York, NY, USA.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Today, software operation engineers rely on application key performance indicators (KPIs) for sizing and orchestrating cloud resources dynamically. KPIs are monitored to assess the achievable performance and to configure various cloud-specific parameters such as flavors of instances and autoscaling rules, among others. Usually, keeping KPIs within acceptable levels requires application expertise which is expensive and can slow down the continuous delivery of software. Expertise is required because KPIs are normally based on application-specific quality-of-service metrics, like service response time and processing rate, instead of generic platform metrics, like those typical across various environments (e.g., CPU and memory utilization, I/O rate, etc.)In this paper, we investigate the feasibility of outsourcing the management of application performance from developers to cloud operators. In the same way that the serverless paradigm allows the execution environment to be fully managed by a third party, we discuss a monitorless model to streamline application deployment by delegating performance management. We show that training a machine learning model with platform-level data, collected from the execution of representative containerized services, allows inferring application KPI degradation. This is an opportunity to simplify operations as engineers can rely solely on platform metrics -- while still fulfilling application KPIs -- to configure portable and application agnostic rules and other cloud-specific parameters to automatically trigger actions such as autoscaling, instance migration, network slicing, etc.Results show that monitorless infers KPI degradation with an accuracy of 97% and, notably, it performs similarly to typical autoscaling solutions, even when autoscaling rules are optimally tuned with knowledge of the expected workload.

  @inproceedings{GrNiIgKoLu-MIDDLEWARE2019-Monitorless, abstract = {Today, software operation engineers rely on application key performance indicators (KPIs) for sizing and orchestrating cloud resources dynamically. KPIs are monitored to assess the achievable performance and to configure various cloud-specific parameters such as flavors of instances and autoscaling rules, among others. Usually, keeping KPIs within acceptable levels requires application expertise which is expensive and can slow down the continuous delivery of software. Expertise is required because KPIs are normally based on application-specific quality-of-service metrics, like service response time and processing rate, instead of generic platform metrics, like those typical across various environments (e.g., CPU and memory utilization, I/O rate, etc.)In this paper, we investigate the feasibility of outsourcing the management of application performance from developers to cloud operators. In the same way that the serverless paradigm allows the execution environment to be fully managed by a third party, we discuss a monitorless model to streamline application deployment by delegating performance management. We show that training a machine learning model with platform-level data, collected from the execution of representative containerized services, allows inferring application KPI degradation. This is an opportunity to simplify operations as engineers can rely solely on platform metrics -- while still fulfilling application KPIs -- to configure portable and application agnostic rules and other cloud-specific parameters to automatically trigger actions such as autoscaling, instance migration, network slicing, etc.Results show that monitorless infers KPI degradation with an accuracy of 97% and, notably, it performs similarly to typical autoscaling solutions, even when autoscaling rules are optimally tuned with knowledge of the expected workload.}, address = {New York, NY, USA}, author = {Grohmann, Johannes and Nicholson, Patrick K. and Iglesias, Jesus Omana and Kounev, Samuel and Lugones, Diego}, booktitle = {Proceedings of the 20th International Middleware Conference}, pages = {149--162}, publisher = {Association for Computing Machinery (ACM)}, series = {Middleware '19}, title = {Monitorless: Predicting Performance Degradation in Cloud Applications with Machine Learning}, year = 2019 }

  %0 Conference Paper %1 GrNiIgKoLu-MIDDLEWARE2019-Monitorless %A Grohmann, Johannes %A Nicholson, Patrick K. %A Iglesias, Jesus Omana %A Kounev, Samuel %A Lugones, Diego %B Proceedings of the 20th International Middleware Conference %C New York, NY, USA %D 2019 %I Association for Computing Machinery (ACM) %P 149--162 %T Monitorless: Predicting Performance Degradation in Cloud Applications with Machine Learning %U https://doi.org/10.1145/3361525.3361543 %X Today, software operation engineers rely on application key performance indicators (KPIs) for sizing and orchestrating cloud resources dynamically. KPIs are monitored to assess the achievable performance and to configure various cloud-specific parameters such as flavors of instances and autoscaling rules, among others. Usually, keeping KPIs within acceptable levels requires application expertise which is expensive and can slow down the continuous delivery of software. Expertise is required because KPIs are normally based on application-specific quality-of-service metrics, like service response time and processing rate, instead of generic platform metrics, like those typical across various environments (e.g., CPU and memory utilization, I/O rate, etc.)In this paper, we investigate the feasibility of outsourcing the management of application performance from developers to cloud operators. In the same way that the serverless paradigm allows the execution environment to be fully managed by a third party, we discuss a monitorless model to streamline application deployment by delegating performance management. We show that training a machine learning model with platform-level data, collected from the execution of representative containerized services, allows inferring application KPI degradation. This is an opportunity to simplify operations as engineers can rely solely on platform metrics -- while still fulfilling application KPIs -- to configure portable and application agnostic rules and other cloud-specific parameters to automatically trigger actions such as autoscaling, instance migration, network slicing, etc.Results show that monitorless infers KPI degradation with an accuracy of 97% and, notably, it performs similarly to typical autoscaling solutions, even when autoscaling rules are optimally tuned with knowledge of the expected workload.
• 
  {Detecting Parametric Dependencies for Performance Models Using Feature Selection Techniques}. Grohmann, Johannes; Eismann, Simon; Elflein, Sven; Mazkatli, Manar; von Kistowski, J{\’o}akim; Kounev, Samuel; in 2019 IEEE 27th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS) 309–322. IEEE Computer Society.

  {Acceptance Rate: 23.8% (29/122)}
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Architectural performance models are a common approach to predict the performance properties of a software system. Parametric dependencies, which describe the relation between the input parameters of a component and its performance properties, significantly increase the prediction accuracy of architectural performance models. However, manually modeling parametric dependencies is time-intensive and requires expert knowledge. Existing automated extraction approaches require dedicated performance tests, which are often infeasible. In this paper, we introduce an approach to automatically identify parametric dependencies from monitoring data using feature selection techniques from the area of machine learning. We evaluate the applicability of three techniques selected from each of the three groups of feature selection methods: a filter method, an embedded method, and a wrapper method. Our evaluation shows that the filter technique outperforms the other approaches. Based on these results, we apply this technique to a distributed micro-service web-shop, where it correctly identifies 11 performance-relevant dependencies, achieving a precision of 91.7% based on a manually labeled gold-standard.

  @inproceedings{GrEiElMaKiKo2019-MASCOTS-DependencyIdentification, abstract = {Architectural performance models are a common approach to predict the performance properties of a software system. Parametric dependencies, which describe the relation between the input parameters of a component and its performance properties, significantly increase the prediction accuracy of architectural performance models. However, manually modeling parametric dependencies is time-intensive and requires expert knowledge. Existing automated extraction approaches require dedicated performance tests, which are often infeasible. In this paper, we introduce an approach to automatically identify parametric dependencies from monitoring data using feature selection techniques from the area of machine learning. We evaluate the applicability of three techniques selected from each of the three groups of feature selection methods: a filter method, an embedded method, and a wrapper method. Our evaluation shows that the filter technique outperforms the other approaches. Based on these results, we apply this technique to a distributed micro-service web-shop, where it correctly identifies 11 performance-relevant dependencies, achieving a precision of 91.7% based on a manually labeled gold-standard.}, author = {Grohmann, Johannes and Eismann, Simon and Elflein, Sven and Mazkatli, Manar and von Kistowski, J{\'o}akim and Kounev, Samuel}, booktitle = {2019 IEEE 27th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS)}, month = {October}, note = {{Acceptance Rate: 23.8% (29/122)}}, pages = {309--322}, publisher = {IEEE Computer Society}, series = {MASCOTS '19}, title = {{Detecting Parametric Dependencies for Performance Models Using Feature Selection Techniques}}, year = 2019 }

  %0 Conference Paper %1 GrEiElMaKiKo2019-MASCOTS-DependencyIdentification %A Grohmann, Johannes %A Eismann, Simon %A Elflein, Sven %A Mazkatli, Manar %A von Kistowski, J{\'o}akim %A Kounev, Samuel %B 2019 IEEE 27th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS) %D 2019 %I IEEE Computer Society %P 309--322 %T {Detecting Parametric Dependencies for Performance Models Using Feature Selection Techniques} %U https://doi.org/10.1109/MASCOTS.2019.00042 %X Architectural performance models are a common approach to predict the performance properties of a software system. Parametric dependencies, which describe the relation between the input parameters of a component and its performance properties, significantly increase the prediction accuracy of architectural performance models. However, manually modeling parametric dependencies is time-intensive and requires expert knowledge. Existing automated extraction approaches require dedicated performance tests, which are often infeasible. In this paper, we introduce an approach to automatically identify parametric dependencies from monitoring data using feature selection techniques from the area of machine learning. We evaluate the applicability of three techniques selected from each of the three groups of feature selection methods: a filter method, an embedded method, and a wrapper method. Our evaluation shows that the filter technique outperforms the other approaches. Based on these results, we apply this technique to a distributed micro-service web-shop, where it correctly identifies 11 performance-relevant dependencies, achieving a precision of 91.7% based on a manually labeled gold-standard.
• 
  Online Power Consumption Estimation for Functions in Cloud Applications. Schmitt, Norbert; Iffländer, Lukas; Bauer, André; Kounev, Samuel; in Proceedings of the 16th IEEE International Conference on Autonomic Computing (ICAC)IEEE.
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @inproceedings{ScIfBaKo2019-ICAC-OnlinePowerEstimation, author = {Schmitt, Norbert and Iffländer, Lukas and Bauer, André and Kounev, Samuel}, booktitle = {Proceedings of the 16th IEEE International Conference on Autonomic Computing (ICAC)}, month = {June}, publisher = {IEEE}, title = {Online Power Consumption Estimation for Functions in Cloud Applications}, year = 2019 }

  %0 Conference Paper %1 ScIfBaKo2019-ICAC-OnlinePowerEstimation %A Schmitt, Norbert %A Iffländer, Lukas %A Bauer, André %A Kounev, Samuel %B Proceedings of the 16th IEEE International Conference on Autonomic Computing (ICAC) %D 2019 %I IEEE %T Online Power Consumption Estimation for Functions in Cloud Applications
• 
  {Autonomic Forecasting Method Selection: Examination and Ways Ahead}. Z{\"u}fle, Marwin; Bauer, Andr{\’e}; Lesch, Veronika; Krupitzer, Christian; Herbst, Nikolas; Kounev, Samuel; Curtef, Valentin; in Proceedings of the 16th {IEEE} International Conference on Autonomic Computing (ICAC)IEEE.
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @inproceedings{ZuBaLeKrHeKoCu2019-ICAC-Autonomic, author = {Z{\"u}fle, Marwin and Bauer, Andr{\'e} and Lesch, Veronika and Krupitzer, Christian and Herbst, Nikolas and Kounev, Samuel and Curtef, Valentin}, booktitle = {Proceedings of the 16th {IEEE} International Conference on Autonomic Computing (ICAC)}, month = {June}, publisher = {IEEE}, title = {{Autonomic Forecasting Method Selection: Examination and Ways Ahead}}, year = 2019 }

  %0 Conference Paper %1 ZuBaLeKrHeKoCu2019-ICAC-Autonomic %A Z{\"u}fle, Marwin %A Bauer, Andr{\'e} %A Lesch, Veronika %A Krupitzer, Christian %A Herbst, Nikolas %A Kounev, Samuel %A Curtef, Valentin %B Proceedings of the 16th {IEEE} International Conference on Autonomic Computing (ICAC) %D 2019 %I IEEE %T {Autonomic Forecasting Method Selection: Examination and Ways Ahead}
• 
  Chamulteon: Coordinated Auto-Scaling of Micro-Services. Bauer, Andr{\’e}; Lesch, Veronika; Versluis, Laurens; Ilyushkin, Alexey; Herbst, Nikolas; Kounev, Samuel; in Proceedings of the 39th IEEE International Conference on Distributed Computing Systems (ICDCS)
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @inproceedings{BaLeVeIlHeKo2019-ICDCS-Chamulteon, author = {Bauer, Andr{\'e} and Lesch, Veronika and Versluis, Laurens and Ilyushkin, Alexey and Herbst, Nikolas and Kounev, Samuel}, booktitle = {Proceedings of the 39th IEEE International Conference on Distributed Computing Systems (ICDCS)}, month = {July}, title = {Chamulteon: Coordinated Auto-Scaling of Micro-Services}, year = 2019 }

  %0 Conference Paper %1 BaLeVeIlHeKo2019-ICDCS-Chamulteon %A Bauer, Andr{\'e} %A Lesch, Veronika %A Versluis, Laurens %A Ilyushkin, Alexey %A Herbst, Nikolas %A Kounev, Samuel %B Proceedings of the 39th IEEE International Conference on Distributed Computing Systems (ICDCS) %D 2019 %T Chamulteon: Coordinated Auto-Scaling of Micro-Services
• 
  {Integrating Statistical Response Time Models in Architectural Performance Models}. Eismann, Simon; Grohmann, Johannes; Walter, J{\"u}rgen; von Kistowski, J{\’o}akim; Kounev, Samuel; in Proceedings of the 2019 IEEE International Conference on Software Architecture (ICSA) 71–80. IEEE.

  Acceptance Rate: 21,9\% (21/96)
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Performance predictions enable software architects to optimize the performance of a software system early in the development cycle. Architectural performance models and statistical response time models are commonly used to derive these performance predictions. However, both methods have significant downsides: Statistical response time models can only predict scenarios for which training data is available, making the prediction of previously unseen system configurations infeasible. In contrast, the time required to simulate an architectural performance model increases exponentially with both system size and level of modeling detail, making the analysis of large, detailed models challenging. Existing approaches use statistical response time models in architectural performance models to avoid modeling subsystems that are difficult or time-consuming to model, yet they do not consider simulation time. In this paper, we propose to model software systems using classical queuing theory and statistical response time models in parallel. This approach allows users to tailor the model for each analysis run, based on the performed adaptations and the requested performance metrics. Our approach enables faster model solution compared to traditional performance models while retaining their ability to predict previously unseen scenarios. In our experiments we observed speedups of up to 94.8%, making the analysis of much larger and more detailed systems feasible.

  @inproceedings{EiGrWaKiKo2019-ICSA-Integrating, abstract = {Performance predictions enable software architects to optimize the performance of a software system early in the development cycle. Architectural performance models and statistical response time models are commonly used to derive these performance predictions. However, both methods have significant downsides: Statistical response time models can only predict scenarios for which training data is available, making the prediction of previously unseen system configurations infeasible. In contrast, the time required to simulate an architectural performance model increases exponentially with both system size and level of modeling detail, making the analysis of large, detailed models challenging. Existing approaches use statistical response time models in architectural performance models to avoid modeling subsystems that are difficult or time-consuming to model, yet they do not consider simulation time. In this paper, we propose to model software systems using classical queuing theory and statistical response time models in parallel. This approach allows users to tailor the model for each analysis run, based on the performed adaptations and the requested performance metrics. Our approach enables faster model solution compared to traditional performance models while retaining their ability to predict previously unseen scenarios. In our experiments we observed speedups of up to 94.8%, making the analysis of much larger and more detailed systems feasible.}, author = {Eismann, Simon and Grohmann, Johannes and Walter, J{\"u}rgen and von Kistowski, J{\'o}akim and Kounev, Samuel}, booktitle = {Proceedings of the 2019 IEEE International Conference on Software Architecture (ICSA)}, month = {March}, note = {Acceptance Rate: 21,9\% (21/96)}, pages = {71--80}, publisher = {IEEE}, title = {{Integrating Statistical Response Time Models in Architectural Performance Models}}, year = 2019 }

  %0 Conference Paper %1 EiGrWaKiKo2019-ICSA-Integrating %A Eismann, Simon %A Grohmann, Johannes %A Walter, J{\"u}rgen %A von Kistowski, J{\'o}akim %A Kounev, Samuel %B Proceedings of the 2019 IEEE International Conference on Software Architecture (ICSA) %D 2019 %I IEEE %P 71--80 %T {Integrating Statistical Response Time Models in Architectural Performance Models} %U https://doi.org/10.1109/ICSA.2019.00016 %X Performance predictions enable software architects to optimize the performance of a software system early in the development cycle. Architectural performance models and statistical response time models are commonly used to derive these performance predictions. However, both methods have significant downsides: Statistical response time models can only predict scenarios for which training data is available, making the prediction of previously unseen system configurations infeasible. In contrast, the time required to simulate an architectural performance model increases exponentially with both system size and level of modeling detail, making the analysis of large, detailed models challenging. Existing approaches use statistical response time models in architectural performance models to avoid modeling subsystems that are difficult or time-consuming to model, yet they do not consider simulation time. In this paper, we propose to model software systems using classical queuing theory and statistical response time models in parallel. This approach allows users to tailor the model for each analysis run, based on the performed adaptations and the requested performance metrics. Our approach enables faster model solution compared to traditional performance models while retaining their ability to predict previously unseen scenarios. In our experiments we observed speedups of up to 94.8%, making the analysis of much larger and more detailed systems feasible.
• 
  {Measuring the Energy Efficiency of Transactional Loads on GPGPU}. von Kistowski, J{\’o}akim; Pais, Johann; Wahl, Tobias; Lange, Klaus-Dieter; Block, Hansfried; Beckett, John; Kounev, Samuel; in Proceedings of the 19th ACM/SPEC International Conference on Performance EngineeringACM, New York, NY, USA.

  {Best Industry Paper Award}
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  {General Purpose Graphics Processing Units (GPGPUs) are becoming more and more common in current servers and data centers, which in turn consume a significant amount of electrical power. Measuring and benchmarking this power consumption is important as it helps with optimization and selection of these servers. However, benchmarking and comparing the energy efficiency of GPGPU workloads is challenging as standardized workloads are rare and standardized power and efficiency measurement methods and metrics do not exist. In addition, not all GPGPU systems run at maximum load all the time. Systems that are utilized in transactional, request driven workloads, for example, can run at lower utilization levels. Existing benchmarks for GPGPU systems primarily consider performance and are intended only to run at maximum load. They do not measure performance or energy efficiency at other loads. In turn, server energy-efficiency benchmarks that consider multiple load levels do not address GPGPUs. This paper introduces a measurement methodology for servers with GPGPU accelerators that considers multiple load levels for transactional workloads. The methodology also addresses verifiability of results in order to achieve comparability of different device solutions. We analyze our methodology on three different systems with solutions from two different accelerator vendors. We investigate the efficacy of different methods of load levels scaling and our methodology's reproducibility. We show that the methodology is able to produce consistent and reproducible results with a maximum coefficient of variation of 1.4\% regarding power consumption.}

  @inproceedings{KiPaWaLaBlBeKo2019-ICPE-GPGPUPower, abstract = {{General Purpose Graphics Processing Units (GPGPUs) are becoming more and more common in current servers and data centers, which in turn consume a significant amount of electrical power. Measuring and benchmarking this power consumption is important as it helps with optimization and selection of these servers. However, benchmarking and comparing the energy efficiency of GPGPU workloads is challenging as standardized workloads are rare and standardized power and efficiency measurement methods and metrics do not exist. In addition, not all GPGPU systems run at maximum load all the time. Systems that are utilized in transactional, request driven workloads, for example, can run at lower utilization levels. Existing benchmarks for GPGPU systems primarily consider performance and are intended only to run at maximum load. They do not measure performance or energy efficiency at other loads. In turn, server energy-efficiency benchmarks that consider multiple load levels do not address GPGPUs. This paper introduces a measurement methodology for servers with GPGPU accelerators that considers multiple load levels for transactional workloads. The methodology also addresses verifiability of results in order to achieve comparability of different device solutions. We analyze our methodology on three different systems with solutions from two different accelerator vendors. We investigate the efficacy of different methods of load levels scaling and our methodology's reproducibility. We show that the methodology is able to produce consistent and reproducible results with a maximum coefficient of variation of 1.4\% regarding power consumption.}}, address = {New York, NY, USA}, author = {von Kistowski, J{\'o}akim and Pais, Johann and Wahl, Tobias and Lange, Klaus-Dieter and Block, Hansfried and Beckett, John and Kounev, Samuel}, booktitle = {Proceedings of the 19th ACM/SPEC International Conference on Performance Engineering}, note = {{<b>Best Industry Paper Award</b>}}, publisher = {ACM}, series = {ICPE '19}, title = {{Measuring the Energy Efficiency of Transactional Loads on GPGPU}}, year = 2019 }

  %0 Conference Paper %1 KiPaWaLaBlBeKo2019-ICPE-GPGPUPower %A von Kistowski, J{\'o}akim %A Pais, Johann %A Wahl, Tobias %A Lange, Klaus-Dieter %A Block, Hansfried %A Beckett, John %A Kounev, Samuel %B Proceedings of the 19th ACM/SPEC International Conference on Performance Engineering %C New York, NY, USA %D 2019 %I ACM %T {Measuring the Energy Efficiency of Transactional Loads on GPGPU} %X {General Purpose Graphics Processing Units (GPGPUs) are becoming more and more common in current servers and data centers, which in turn consume a significant amount of electrical power. Measuring and benchmarking this power consumption is important as it helps with optimization and selection of these servers. However, benchmarking and comparing the energy efficiency of GPGPU workloads is challenging as standardized workloads are rare and standardized power and efficiency measurement methods and metrics do not exist. In addition, not all GPGPU systems run at maximum load all the time. Systems that are utilized in transactional, request driven workloads, for example, can run at lower utilization levels. Existing benchmarks for GPGPU systems primarily consider performance and are intended only to run at maximum load. They do not measure performance or energy efficiency at other loads. In turn, server energy-efficiency benchmarks that consider multiple load levels do not address GPGPUs. This paper introduces a measurement methodology for servers with GPGPU accelerators that considers multiple load levels for transactional workloads. The methodology also addresses verifiability of results in order to achieve comparability of different device solutions. We analyze our methodology on three different systems with solutions from two different accelerator vendors. We investigate the efficacy of different methods of load levels scaling and our methodology's reproducibility. We show that the methodology is able to produce consistent and reproducible results with a maximum coefficient of variation of 1.4\% regarding power consumption.}
• 
  {Predicting Server Power Consumption from Standard Rating Results}. von Kistowski, J{\’o}akim; Grohmann, Johannes; Schmitt, Norbert; Kounev, Samuel; in Proceedings of the 19th ACM/SPEC International Conference on Performance Engineering 301–312. Association for Computing Machinery (ACM), New York, NY, USA.

  {Full Paper Acceptance Rate: 18.6\% (13/70)}
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Data center providers and server operators try to reduce the power consumption of their servers. Finding an energy efficient server for a specific target application is a first step in this regard. Estimating the power consumption of an application on an unavailable server is difficult, as nameplate power values are generally overestimations. Offline power models are able to predict the consumption accurately, but are usually intended for system design, requiring very specific and detailed knowledge about the system under consideration. In this paper, we introduce an offline power prediction method that uses the results of standard power rating tools. The method predicts the power consumption of a specific application for multiple load levels on a target server that is otherwise unavailable for testing. We evaluate our approach by predicting the power consumption of three applications on different physical servers. Our method is able to achieve an average prediction error of 9.49% for three workloads running on real-world, physical servers.

  @inproceedings{KiGrScKo2019-ICPE-PowerPrediction, abstract = {Data center providers and server operators try to reduce the power consumption of their servers. Finding an energy efficient server for a specific target application is a first step in this regard. Estimating the power consumption of an application on an unavailable server is difficult, as nameplate power values are generally overestimations. Offline power models are able to predict the consumption accurately, but are usually intended for system design, requiring very specific and detailed knowledge about the system under consideration. In this paper, we introduce an offline power prediction method that uses the results of standard power rating tools. The method predicts the power consumption of a specific application for multiple load levels on a target server that is otherwise unavailable for testing. We evaluate our approach by predicting the power consumption of three applications on different physical servers. Our method is able to achieve an average prediction error of 9.49% for three workloads running on real-world, physical servers.}, address = {New York, NY, USA}, author = {von Kistowski, J{\'o}akim and Grohmann, Johannes and Schmitt, Norbert and Kounev, Samuel}, booktitle = {Proceedings of the 19th ACM/SPEC International Conference on Performance Engineering}, note = {{Full Paper Acceptance Rate: 18.6\% (13/70)}}, pages = {301--312}, publisher = {Association for Computing Machinery (ACM)}, series = {ICPE '19}, title = {{Predicting Server Power Consumption from Standard Rating Results}}, year = 2019 }

  %0 Conference Paper %1 KiGrScKo2019-ICPE-PowerPrediction %A von Kistowski, J{\'o}akim %A Grohmann, Johannes %A Schmitt, Norbert %A Kounev, Samuel %B Proceedings of the 19th ACM/SPEC International Conference on Performance Engineering %C New York, NY, USA %D 2019 %I Association for Computing Machinery (ACM) %P 301--312 %T {Predicting Server Power Consumption from Standard Rating Results} %U https://doi.org/10.1145/3297663.3310298 %X Data center providers and server operators try to reduce the power consumption of their servers. Finding an energy efficient server for a specific target application is a first step in this regard. Estimating the power consumption of an application on an unavailable server is difficult, as nameplate power values are generally overestimations. Offline power models are able to predict the consumption accurately, but are usually intended for system design, requiring very specific and detailed knowledge about the system under consideration. In this paper, we introduce an offline power prediction method that uses the results of standard power rating tools. The method predicts the power consumption of a specific application for multiple load levels on a target server that is otherwise unavailable for testing. We evaluate our approach by predicting the power consumption of three applications on different physical servers. Our method is able to achieve an average prediction error of 9.49% for three workloads running on real-world, physical servers.

• 
  {TeaStore: A Micro-Service Reference Application for Benchmarking, Modeling and Resource Management Research}. von Kistowski, J{\’o}akim; Eismann, Simon; Schmitt, Norbert; Bauer, Andr{\’e}; Grohmann, Johannes; Kounev, Samuel; in Proceedings of the 26th IEEE International Symposium on the Modelling, Analysis, and Simulation of Computer and Telecommunication Systems 223–236. IEEE Computer Society.

  {Acceptance Rate: 29.5\% (23/78)}
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Modern distributed applications offer complex performance behavior and many degrees of freedom regarding deployment and configuration. Researchers employ various methods of analysis, modeling, and management that leverage these degrees of freedom to predict or improve non-functional properties of the software under consideration. In order to demonstrate and evaluate their applicability in the real world, methods resulting from such research areas require test and reference applications that offer a range of different behaviors, as well as the necessary degrees of freedom. Existing production software is often inaccessible for researchers or closed off to instrumentation. Existing testing and benchmarking frameworks, on the other hand, are either designed for specific testing scenarios, or they do not offer the necessary degrees of freedom. Further, most test applications are difficult to deploy and run, or are outdated. In this paper, we introduce the TeaStore, a state-of-the-art micro-service-based test and reference application. TeaStore offers services with different performance characteristics and many degrees of freedom regarding deployment and configuration to be used as a benchmarking framework for researchers. The TeaStore allows evaluating performance modeling and resource management techniques; it also offers instrumented variants to enable extensive run-time analysis. We demonstrate TeaStore's use in three contexts: performance modeling, cloud resource management, and energy efficiency analysis. Our experiments show that TeaStore can be used for evaluating novel approaches in these contexts and also motivates further research in the areas of performance modeling and resource management.

  @inproceedings{KiEiScBaGrKo2018-MASCOTS-TeaStore, abstract = {Modern distributed applications offer complex performance behavior and many degrees of freedom regarding deployment and configuration. Researchers employ various methods of analysis, modeling, and management that leverage these degrees of freedom to predict or improve non-functional properties of the software under consideration. In order to demonstrate and evaluate their applicability in the real world, methods resulting from such research areas require test and reference applications that offer a range of different behaviors, as well as the necessary degrees of freedom. Existing production software is often inaccessible for researchers or closed off to instrumentation. Existing testing and benchmarking frameworks, on the other hand, are either designed for specific testing scenarios, or they do not offer the necessary degrees of freedom. Further, most test applications are difficult to deploy and run, or are outdated. In this paper, we introduce the TeaStore, a state-of-the-art micro-service-based test and reference application. TeaStore offers services with different performance characteristics and many degrees of freedom regarding deployment and configuration to be used as a benchmarking framework for researchers. The TeaStore allows evaluating performance modeling and resource management techniques; it also offers instrumented variants to enable extensive run-time analysis. We demonstrate TeaStore's use in three contexts: performance modeling, cloud resource management, and energy efficiency analysis. Our experiments show that TeaStore can be used for evaluating novel approaches in these contexts and also motivates further research in the areas of performance modeling and resource management.}, author = {von Kistowski, J{\'o}akim and Eismann, Simon and Schmitt, Norbert and Bauer, Andr{\'e} and Grohmann, Johannes and Kounev, Samuel}, booktitle = {Proceedings of the 26th IEEE International Symposium on the Modelling, Analysis, and Simulation of Computer and Telecommunication Systems}, month = {September}, note = {{Acceptance Rate: 29.5\% (23/78)}}, pages = {223--236}, publisher = {IEEE Computer Society}, series = {MASCOTS '18}, title = {{TeaStore: A Micro-Service Reference Application for Benchmarking, Modeling and Resource Management Research}}, year = 2018 }

  %0 Conference Paper %1 KiEiScBaGrKo2018-MASCOTS-TeaStore %A von Kistowski, J{\'o}akim %A Eismann, Simon %A Schmitt, Norbert %A Bauer, Andr{\'e} %A Grohmann, Johannes %A Kounev, Samuel %B Proceedings of the 26th IEEE International Symposium on the Modelling, Analysis, and Simulation of Computer and Telecommunication Systems %D 2018 %I IEEE Computer Society %P 223--236 %T {TeaStore: A Micro-Service Reference Application for Benchmarking, Modeling and Resource Management Research} %U https://doi.org/10.1109/MASCOTS.2018.00030 %X Modern distributed applications offer complex performance behavior and many degrees of freedom regarding deployment and configuration. Researchers employ various methods of analysis, modeling, and management that leverage these degrees of freedom to predict or improve non-functional properties of the software under consideration. In order to demonstrate and evaluate their applicability in the real world, methods resulting from such research areas require test and reference applications that offer a range of different behaviors, as well as the necessary degrees of freedom. Existing production software is often inaccessible for researchers or closed off to instrumentation. Existing testing and benchmarking frameworks, on the other hand, are either designed for specific testing scenarios, or they do not offer the necessary degrees of freedom. Further, most test applications are difficult to deploy and run, or are outdated. In this paper, we introduce the TeaStore, a state-of-the-art micro-service-based test and reference application. TeaStore offers services with different performance characteristics and many degrees of freedom regarding deployment and configuration to be used as a benchmarking framework for researchers. The TeaStore allows evaluating performance modeling and resource management techniques; it also offers instrumented variants to enable extensive run-time analysis. We demonstrate TeaStore's use in three contexts: performance modeling, cloud resource management, and energy efficiency analysis. Our experiments show that TeaStore can be used for evaluating novel approaches in these contexts and also motivates further research in the areas of performance modeling and resource management.
• 
  {SmarTor: Smarter Tor with Smart Contracts: Improving resilience of topology distribution in the Tor network}. Greubel, Andre; Dmitrienko, Alexandra; Kounev, Samuel; in Proceedings of the 34th Annual Computer Security Applications ConferenceACM, New York, NY, USA.
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @inproceedings{GrDmKo2018-ACSAC-SmarTor, address = {New York, NY, USA}, author = {Greubel, Andre and Dmitrienko, Alexandra and Kounev, Samuel}, booktitle = {Proceedings of the 34th Annual Computer Security Applications Conference}, month = {December}, organization = {ACM}, series = {ACSAC '18}, title = {{SmarTor: Smarter Tor with Smart Contracts: Improving resilience of topology distribution in the Tor network}}, year = 2018 }

  %0 Conference Paper %1 GrDmKo2018-ACSAC-SmarTor %A Greubel, Andre %A Dmitrienko, Alexandra %A Kounev, Samuel %B Proceedings of the 34th Annual Computer Security Applications Conference %C New York, NY, USA %D 2018 %T {SmarTor: Smarter Tor with Smart Contracts: Improving resilience of topology distribution in the Tor network}
• 
  {Modeling of Parametric Dependencies for Performance Prediction of Component-based Software Systems at Run-time}. Eismann, Simon; Walter, J{\"u}rgen; von Kistowski, J{\’o}akim; Kounev, Samuel; in 2018 IEEE International Conference on Software Architecture (ICSA) 135–144.

  Acceptance Rate: 25,6\% (22/86)
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Model-based performance analysis can be leveraged to explore performance properties of software systems. To capture the behavior of varying workload mixes, configurations, and deployments of a software system requires formal modeling of the impact of configuration parameters and user input on the system behavior. Such influences are represented as parametric dependencies in software performance models. Existing modeling approaches focus on modeling parametric dependencies at design-time. This paper identifies runtime specific parametric dependency features, which are not supported by existing work. Therefore, this paper proposes a novel modeling methodology for parametric dependencies and a corresponding graph-based resolution algorithm. This algorithm enables the solution of models containing component instance-level dependencies, variables with multiple descriptions in parallel, and correlations modeled as parametric dependencies. We integrate our work into the Descartes Modeling Language (DML), allowing for accurate and efficient modeling and analysis of parametric dependencies. These performance predictions are valuable for various purposes such as capacity planning, bottleneck analysis, configuration optimization and proactive auto-scaling. Our evaluation analyzes a video store application. The prediction for varying language mixes and video sizes shows a mean error below 5% for utilization and below 10% for response time.

  @inproceedings{EiWaKiKo2018-ParametricDependencies, abstract = {Model-based performance analysis can be leveraged to explore performance properties of software systems. To capture the behavior of varying workload mixes, configurations, and deployments of a software system requires formal modeling of the impact of configuration parameters and user input on the system behavior. Such influences are represented as parametric dependencies in software performance models. Existing modeling approaches focus on modeling parametric dependencies at design-time. This paper identifies runtime specific parametric dependency features, which are not supported by existing work. Therefore, this paper proposes a novel modeling methodology for parametric dependencies and a corresponding graph-based resolution algorithm. This algorithm enables the solution of models containing component instance-level dependencies, variables with multiple descriptions in parallel, and correlations modeled as parametric dependencies. We integrate our work into the Descartes Modeling Language (DML), allowing for accurate and efficient modeling and analysis of parametric dependencies. These performance predictions are valuable for various purposes such as capacity planning, bottleneck analysis, configuration optimization and proactive auto-scaling. Our evaluation analyzes a video store application. The prediction for varying language mixes and video sizes shows a mean error below 5% for utilization and below 10% for response time.}, author = {Eismann, Simon and Walter, J{\"u}rgen and von Kistowski, J{\'o}akim and Kounev, Samuel}, booktitle = {2018 IEEE International Conference on Software Architecture (ICSA)}, month = {April}, note = {Acceptance Rate: 25,6\% (22/86)}, pages = {135-144}, title = {{Modeling of Parametric Dependencies for Performance Prediction of Component-based Software Systems at Run-time}}, year = 2018 }

  %0 Conference Paper %1 EiWaKiKo2018-ParametricDependencies %A Eismann, Simon %A Walter, J{\"u}rgen %A von Kistowski, J{\'o}akim %A Kounev, Samuel %B 2018 IEEE International Conference on Software Architecture (ICSA) %D 2018 %P 135-144 %T {Modeling of Parametric Dependencies for Performance Prediction of Component-based Software Systems at Run-time} %X Model-based performance analysis can be leveraged to explore performance properties of software systems. To capture the behavior of varying workload mixes, configurations, and deployments of a software system requires formal modeling of the impact of configuration parameters and user input on the system behavior. Such influences are represented as parametric dependencies in software performance models. Existing modeling approaches focus on modeling parametric dependencies at design-time. This paper identifies runtime specific parametric dependency features, which are not supported by existing work. Therefore, this paper proposes a novel modeling methodology for parametric dependencies and a corresponding graph-based resolution algorithm. This algorithm enables the solution of models containing component instance-level dependencies, variables with multiple descriptions in parallel, and correlations modeled as parametric dependencies. We integrate our work into the Descartes Modeling Language (DML), allowing for accurate and efficient modeling and analysis of parametric dependencies. These performance predictions are valuable for various purposes such as capacity planning, bottleneck analysis, configuration optimization and proactive auto-scaling. Our evaluation analyzes a video store application. The prediction for varying language mixes and video sizes shows a mean error below 5% for utilization and below 10% for response time.
• 
  {On the Value of Service Demand Estimation for Auto-Scaling}. Bauer, Andr{\’e}; Grohmann, Johannes; Herbst, Nikolas; Kounev, Samuel; in Proceedings of 19th International GI/ITG Conference on Measurement, Modelling and Evaluation of Computing Systems (MMB 2018) (Vol. 10740) 142–156. Springer, Cham.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  In the context of performance models, service demands are key model parameters capturing the average time individual requests of different workload classes are actively processed. In a system under load, due to measurement interference, service demands normally cannot be measured directly, however, a number of estimation approaches exist based on high-level performance metrics. In this paper, we show that service demands provide significant benefits for implementing modern auto-scalers. Auto-scaling describes the process of dynamically adjusting the number of allocated virtual resources (e.g., virtual machines) in a data center according to the incoming workload. We demonstrate that even a simple auto-scaler that leverages information about service demands significantly outperforms auto-scalers solely based on CPU utilization measurements. This is shown by testing two approaches in three different scenarios. Our results show that the service demand-based auto-scaler outperforms the CPU utilization-based one in all scenarios. Our results encourage further research on the application of service demand estimates for resource management in data centers.

  @inproceedings{BaGrHeKo2018-MMB-ServiceDemand, abstract = {In the context of performance models, service demands are key model parameters capturing the average time individual requests of different workload classes are actively processed. In a system under load, due to measurement interference, service demands normally cannot be measured directly, however, a number of estimation approaches exist based on high-level performance metrics. In this paper, we show that service demands provide significant benefits for implementing modern auto-scalers. Auto-scaling describes the process of dynamically adjusting the number of allocated virtual resources (e.g., virtual machines) in a data center according to the incoming workload. We demonstrate that even a simple auto-scaler that leverages information about service demands significantly outperforms auto-scalers solely based on CPU utilization measurements. This is shown by testing two approaches in three different scenarios. Our results show that the service demand-based auto-scaler outperforms the CPU utilization-based one in all scenarios. Our results encourage further research on the application of service demand estimates for resource management in data centers.}, address = {Cham}, author = {Bauer, Andr{\'e} and Grohmann, Johannes and Herbst, Nikolas and Kounev, Samuel}, booktitle = {Proceedings of 19th International GI/ITG Conference on Measurement, Modelling and Evaluation of Computing Systems (MMB 2018)}, month = {February}, pages = {142--156}, publisher = {Springer}, series = {Lecture Notes in Computer Science}, title = {{On the Value of Service Demand Estimation for Auto-Scaling}}, volume = 10740, year = 2018 }

  %0 Conference Paper %1 BaGrHeKo2018-MMB-ServiceDemand %A Bauer, Andr{\'e} %A Grohmann, Johannes %A Herbst, Nikolas %A Kounev, Samuel %B Proceedings of 19th International GI/ITG Conference on Measurement, Modelling and Evaluation of Computing Systems (MMB 2018) %C Cham %D 2018 %I Springer %P 142--156 %T {On the Value of Service Demand Estimation for Auto-Scaling} %U https://doi.org/10.1007/978-3-319-74947-1_10 %V 10740 %X In the context of performance models, service demands are key model parameters capturing the average time individual requests of different workload classes are actively processed. In a system under load, due to measurement interference, service demands normally cannot be measured directly, however, a number of estimation approaches exist based on high-level performance metrics. In this paper, we show that service demands provide significant benefits for implementing modern auto-scalers. Auto-scaling describes the process of dynamically adjusting the number of allocated virtual resources (e.g., virtual machines) in a data center according to the incoming workload. We demonstrate that even a simple auto-scaler that leverages information about service demands significantly outperforms auto-scalers solely based on CPU utilization measurements. This is shown by testing two approaches in three different scenarios. Our results show that the service demand-based auto-scaler outperforms the CPU utilization-based one in all scenarios. Our results encourage further research on the application of service demand estimates for resource management in data centers.
• 
  {FOX: Cost-Awareness for Autonomic Resource Management in Public Clouds}. Lesch, Veronika; Bauer, Andr{\’e}; Herbst, Nikolas; Kounev, Samuel; in Proceedings of the 9th ACM/SPEC International Conference on Performance Engineering (ICPE 2018)ACM, New York, NY, USA.

  Full Paper Acceptance Ratio: 23,7%
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  @inproceedings{LeBaHeKo-ICPE18-FOX-AS-COST, address = {New York, NY, USA}, author = {Lesch, Veronika and Bauer, Andr{\'e} and Herbst, Nikolas and Kounev, Samuel}, booktitle = {Proceedings of the 9th ACM/SPEC International Conference on Performance Engineering (ICPE 2018)}, month = {April}, note = {Full Paper Acceptance Ratio: 23,7%}, publisher = {ACM}, title = {{FOX: Cost-Awareness for Autonomic Resource Management in Public Clouds}}, year = 2018 }

  %0 Conference Paper %1 LeBaHeKo-ICPE18-FOX-AS-COST %A Lesch, Veronika %A Bauer, Andr{\'e} %A Herbst, Nikolas %A Kounev, Samuel %B Proceedings of the 9th ACM/SPEC International Conference on Performance Engineering (ICPE 2018) %C New York, NY, USA %D 2018 %I ACM %T {FOX: Cost-Awareness for Autonomic Resource Management in Public Clouds} %U https://doi.org/10.1145/3184407.3184415

• 
  Performance Assessment of Cloud Migrations from Network and Application Point of View. Iffländer, Lukas; Wamser, Florian; Metter, Christopher; Tran-Gia, Phuoc; Kounev, Samuel; in {Proceedings of 9th EAI International Conference on Mobile Networks and Management (MONAMI 2018)}
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @inproceedings{IfWaMeTrKo-Monami-Migration-Performance, author = {Iffländer, Lukas and Wamser, Florian and Metter, Christopher and Tran-Gia, Phuoc and Kounev, Samuel}, booktitle = {{Proceedings of 9th EAI International Conference on Mobile Networks and Management (MONAMI 2018)}}, month = {December}, title = {Performance Assessment of Cloud Migrations from Network and Application Point of View}, year = 2017 }

  %0 Conference Paper %1 IfWaMeTrKo-Monami-Migration-Performance %A Iffländer, Lukas %A Wamser, Florian %A Metter, Christopher %A Tran-Gia, Phuoc %A Kounev, Samuel %B {Proceedings of 9th EAI International Conference on Mobile Networks and Management (MONAMI 2018)} %D 2017 %T Performance Assessment of Cloud Migrations from Network and Application Point of View
• 
  {Automated and Adaptable Decision Support for Software Performance Engineering}. Walter, J{\"u}rgen; van Hoorn, Andre; Kounev, Samuel; in Proceedings of the 11th EAI International Conference on Performance Evaluation Methodologies and Tools
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Software performance engineering (SPE) provides a plethora of methods and tooling for measuring, modeling, and evaluating performance properties of software systems. The solution approaches come with different strengths and limitations concerning, for example, accuracy, time-to-result, or system overhead. While approaches allow for interchangeability, the choice of an appropriate approach and tooling to solve a given performance concern still relies on expert knowledge. Currently, there is no automated and extensible approach for decision support. In this paper, we present a methodology for the automated selection of performance engineering approaches tailored to user concerns. We decouple the complexity of selecting an SPE approach for a given scenario providing a decision engine and solution approach capability models. This separation allows to easily append additional solution approaches and rating criteria. We demonstrate the applicability by presenting decision engines that compare measurement- and model-based analysis approaches.

  @inproceedings{WalterHoKo2017-DecisionSupport, abstract = {Software performance engineering (SPE) provides a plethora of methods and tooling for measuring, modeling, and evaluating performance properties of software systems. The solution approaches come with different strengths and limitations concerning, for example, accuracy, time-to-result, or system overhead. While approaches allow for interchangeability, the choice of an appropriate approach and tooling to solve a given performance concern still relies on expert knowledge. Currently, there is no automated and extensible approach for decision support. In this paper, we present a methodology for the automated selection of performance engineering approaches tailored to user concerns. We decouple the complexity of selecting an SPE approach for a given scenario providing a decision engine and solution approach capability models. This separation allows to easily append additional solution approaches and rating criteria. We demonstrate the applicability by presenting decision engines that compare measurement- and model-based analysis approaches.}, author = {Walter, J{\"u}rgen and van Hoorn, Andre and Kounev, Samuel}, booktitle = {Proceedings of the 11th EAI International Conference on Performance Evaluation Methodologies and Tools}, month = 12, title = {{Automated and Adaptable Decision Support for Software Performance Engineering}}, year = 2017 }

  %0 Conference Paper %1 WalterHoKo2017-DecisionSupport %A Walter, J{\"u}rgen %A van Hoorn, Andre %A Kounev, Samuel %B Proceedings of the 11th EAI International Conference on Performance Evaluation Methodologies and Tools %D 2017 %T {Automated and Adaptable Decision Support for Software Performance Engineering} %X Software performance engineering (SPE) provides a plethora of methods and tooling for measuring, modeling, and evaluating performance properties of software systems. The solution approaches come with different strengths and limitations concerning, for example, accuracy, time-to-result, or system overhead. While approaches allow for interchangeability, the choice of an appropriate approach and tooling to solve a given performance concern still relies on expert knowledge. Currently, there is no automated and extensible approach for decision support. In this paper, we present a methodology for the automated selection of performance engineering approaches tailored to user concerns. We decouple the complexity of selecting an SPE approach for a given scenario providing a decision engine and solution approach capability models. This separation allows to easily append additional solution approaches and rating criteria. We demonstrate the applicability by presenting decision engines that compare measurement- and model-based analysis approaches.

• 
  {Automated Extraction of Network Traffic Models Suitable for Performance Simulation}. Rygielski, Piotr; Simko, Viliam; Sittner, Felix; Aschenbrenner, Doris; Kounev, Samuel; Schilling, Klaus; in Proceedings of the 7th ACM/SPEC International Conference on Performance Engineering (ICPE 2016) 27–35. ACM.

  {Acceptance rate (Full Paper): 19/57 = 33\%}
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  {Data centers are increasingly becoming larger and dynamic due to virtualization. In order to leverage the performance modeling and prediction techniques, such as Palladio Component Model or Descartes Modeling Language, in such a dynamic environments, it is necessary to automate the model extraction. Building and maintaining such models manually is not feasible anymore due to their size and the level of details. This paper is focused on traffic models that are an essential part of network infrastructure. Our goal is to decompose real traffic dumps into models suitable for performance prediction using Descartes Network Infrastructure modeling approach. The main challenge was to efficiently encode an arbitrary signal in the form of simple traffic generators while maintaining the shape of the original signal. We show that a typical 15 minute long tcpdump trace can be compressed to 0.4--15% of its original size whereas the relative median of extraction error is 0% for the most of the 69 examined traces.}

  @inproceedings{RySiSiAschKoSch2016-dni-traffic-extraction-icpe2016, abstract = {{Data centers are increasingly becoming larger and dynamic due to virtualization. In order to leverage the performance modeling and prediction techniques, such as Palladio Component Model or Descartes Modeling Language, in such a dynamic environments, it is necessary to automate the model extraction. Building and maintaining such models manually is not feasible anymore due to their size and the level of details. This paper is focused on traffic models that are an essential part of network infrastructure. Our goal is to decompose real traffic dumps into models suitable for performance prediction using Descartes Network Infrastructure modeling approach. The main challenge was to efficiently encode an arbitrary signal in the form of simple traffic generators while maintaining the shape of the original signal. We show that a typical 15 minute long tcpdump trace can be compressed to 0.4--15% of its original size whereas the relative median of extraction error is 0% for the most of the 69 examined traces.}}, author = {Rygielski, Piotr and Simko, Viliam and Sittner, Felix and Aschenbrenner, Doris and Kounev, Samuel and Schilling, Klaus}, booktitle = {Proceedings of the 7th ACM/SPEC International Conference on Performance Engineering (ICPE 2016)}, month = {March}, note = {{Acceptance rate (Full Paper): 19/57 = 33\%}}, pages = {27--35}, publisher = {ACM}, title = {{Automated Extraction of Network Traffic Models Suitable for Performance Simulation}}, year = 2016 }

  %0 Conference Paper %1 RySiSiAschKoSch2016-dni-traffic-extraction-icpe2016 %A Rygielski, Piotr %A Simko, Viliam %A Sittner, Felix %A Aschenbrenner, Doris %A Kounev, Samuel %A Schilling, Klaus %B Proceedings of the 7th ACM/SPEC International Conference on Performance Engineering (ICPE 2016) %D 2016 %I ACM %P 27--35 %T {Automated Extraction of Network Traffic Models Suitable for Performance Simulation} %U http://dl.acm.org/citation.cfm?doid=2851553.2851570 %X {Data centers are increasingly becoming larger and dynamic due to virtualization. In order to leverage the performance modeling and prediction techniques, such as Palladio Component Model or Descartes Modeling Language, in such a dynamic environments, it is necessary to automate the model extraction. Building and maintaining such models manually is not feasible anymore due to their size and the level of details. This paper is focused on traffic models that are an essential part of network infrastructure. Our goal is to decompose real traffic dumps into models suitable for performance prediction using Descartes Network Infrastructure modeling approach. The main challenge was to efficiently encode an arbitrary signal in the form of simple traffic generators while maintaining the shape of the original signal. We show that a typical 15 minute long tcpdump trace can be compressed to 0.4--15% of its original size whereas the relative median of extraction error is 0% for the most of the 69 examined traces.}
• 
  {Modeling and Prediction of Software-Defined Networks Performance using Queueing Petri Nets}. Rygielski, Piotr; Seliuchenko, Marian; Kounev, Samuel; in Proceedings of the Ninth International Conference on Simulation Tools and Techniques (SIMUTools 2016) 66–75.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  {Using various modeling and simulation approaches for predicting network performance requires extensive experience and involves a number of time consuming manual steps regarding each of the modeling formalisms. Descartes Network Infrastructure (DNI) is a data center network performance modeling approach that addresses this challenge by offering multiple performance models but requiring to use only a single modeling language. In this paper, we thoroughly extend DNI to support new networking paradigms like, among others, Software-Defined Networking (SDN) and Network-Function Virtualization (NFV). Additionally, we demonstrate how SDN-based networks can be modeled using DNI and how are they transformed later into Queueing Petri Nets (QPN) using a model-to-model transformation. In the analysis of the performance prediction accuracy, we show that automatically generated QPN models represent the performance of heterogeneous SDN hardware with maximal prediction accuracy error of 12 percent.}

  @inproceedings{RySeKo2016-SIMUTools-SDN-switches-QPN, abstract = {{Using various modeling and simulation approaches for predicting network performance requires extensive experience and involves a number of time consuming manual steps regarding each of the modeling formalisms. Descartes Network Infrastructure (DNI) is a data center network performance modeling approach that addresses this challenge by offering multiple performance models but requiring to use only a single modeling language. In this paper, we thoroughly extend DNI to support new networking paradigms like, among others, Software-Defined Networking (SDN) and Network-Function Virtualization (NFV). Additionally, we demonstrate how SDN-based networks can be modeled using DNI and how are they transformed later into Queueing Petri Nets (QPN) using a model-to-model transformation. In the analysis of the performance prediction accuracy, we show that automatically generated QPN models represent the performance of heterogeneous SDN hardware with maximal prediction accuracy error of 12 percent.}}, author = {Rygielski, Piotr and Seliuchenko, Marian and Kounev, Samuel}, booktitle = {Proceedings of the Ninth International Conference on Simulation Tools and Techniques (SIMUTools 2016)}, month = {August}, pages = {66--75}, title = {{Modeling and Prediction of Software-Defined Networks Performance using Queueing Petri Nets}}, year = 2016 }

  %0 Conference Paper %1 RySeKo2016-SIMUTools-SDN-switches-QPN %A Rygielski, Piotr %A Seliuchenko, Marian %A Kounev, Samuel %B Proceedings of the Ninth International Conference on Simulation Tools and Techniques (SIMUTools 2016) %D 2016 %P 66--75 %T {Modeling and Prediction of Software-Defined Networks Performance using Queueing Petri Nets} %U http://dl.acm.org/citation.cfm?id=3021426.3021437 %X {Using various modeling and simulation approaches for predicting network performance requires extensive experience and involves a number of time consuming manual steps regarding each of the modeling formalisms. Descartes Network Infrastructure (DNI) is a data center network performance modeling approach that addresses this challenge by offering multiple performance models but requiring to use only a single modeling language. In this paper, we thoroughly extend DNI to support new networking paradigms like, among others, Software-Defined Networking (SDN) and Network-Function Virtualization (NFV). Additionally, we demonstrate how SDN-based networks can be modeled using DNI and how are they transformed later into Queueing Petri Nets (QPN) using a model-to-model transformation. In the analysis of the performance prediction accuracy, we show that automatically generated QPN models represent the performance of heterogeneous SDN hardware with maximal prediction accuracy error of 12 percent.}
• 
  {Variations in CPU Power Consumption}. von Kistowski, J{\’o}akim; Block, Hansfried; Beckett, John; Spradling, Cloyce; Lange, Klaus-Dieter; Kounev, Samuel; in Proceedings of the 7th ACM/SPEC International Conference on Performance Engineering (ICPE 2016)ACM, New York, NY, USA.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  {Experimental analysis of computer systems' power consumption has become an integral part of system performance evaluation, efficiency management, and model-based analysis. As with all measurements, repeatability and reproducibility of power measurements is a major challenge. Nominally identical systems can have different power consumption running the same workload under otherwise identical conditions. This behavior can also be observed for individual system components. Specifically, CPU power consumption can vary amongst different samples of nominally identical CPUs. This in turn has a significant impact on the overall system power, considering that a system's processor is the largest and most dynamic power consumer of the overall system. The concrete impact of CPU sample power variations is unknown, as comprehensive studies about differences in power consumption for nominally identical systems are currently missing. We address this lack of studies by conducting measurements on four different processor types from two different architectures. For each of these types, we compare up to 30 physical processor samples with a total sum of 90 samples over all processor types. We analyze the variations in power consumption for the different samples using six different workloads over five load levels. Additionally, we analyze how these variations change for different processor core counts and architectures. The results of this paper show that selection of a processor sample can have a statistically significant impact on power consumption. With no correlation to performance, power consumption for nominally identical processors can differ as much as 29.6\% in idle and 19.5\% at full load. We also show that these variations change over different architectures and processor types.}

  @inproceedings{KiBlBeSpLaKo2016-ICPE-PowerVariation, abstract = {{Experimental analysis of computer systems' power consumption has become an integral part of system performance evaluation, efficiency management, and model-based analysis. As with all measurements, repeatability and reproducibility of power measurements is a major challenge. Nominally identical systems can have different power consumption running the same workload under otherwise identical conditions. This behavior can also be observed for individual system components. Specifically, CPU power consumption can vary amongst different samples of nominally identical CPUs. This in turn has a significant impact on the overall system power, considering that a system's processor is the largest and most dynamic power consumer of the overall system. The concrete impact of CPU sample power variations is unknown, as comprehensive studies about differences in power consumption for nominally identical systems are currently missing. We address this lack of studies by conducting measurements on four different processor types from two different architectures. For each of these types, we compare up to 30 physical processor samples with a total sum of 90 samples over all processor types. We analyze the variations in power consumption for the different samples using six different workloads over five load levels. Additionally, we analyze how these variations change for different processor core counts and architectures. The results of this paper show that selection of a processor sample can have a statistically significant impact on power consumption. With no correlation to performance, power consumption for nominally identical processors can differ as much as 29.6\% in idle and 19.5\% at full load. We also show that these variations change over different architectures and processor types.}}, address = {New York, NY, USA}, author = {von Kistowski, J{\'o}akim and Block, Hansfried and Beckett, John and Spradling, Cloyce and Lange, Klaus-Dieter and Kounev, Samuel}, booktitle = {Proceedings of the 7th ACM/SPEC International Conference on Performance Engineering (ICPE 2016)}, month = {March}, publisher = {ACM}, title = {{Variations in CPU Power Consumption}}, year = 2016 }

  %0 Conference Paper %1 KiBlBeSpLaKo2016-ICPE-PowerVariation %A von Kistowski, J{\'o}akim %A Block, Hansfried %A Beckett, John %A Spradling, Cloyce %A Lange, Klaus-Dieter %A Kounev, Samuel %B Proceedings of the 7th ACM/SPEC International Conference on Performance Engineering (ICPE 2016) %C New York, NY, USA %D 2016 %I ACM %T {Variations in CPU Power Consumption} %X {Experimental analysis of computer systems' power consumption has become an integral part of system performance evaluation, efficiency management, and model-based analysis. As with all measurements, repeatability and reproducibility of power measurements is a major challenge. Nominally identical systems can have different power consumption running the same workload under otherwise identical conditions. This behavior can also be observed for individual system components. Specifically, CPU power consumption can vary amongst different samples of nominally identical CPUs. This in turn has a significant impact on the overall system power, considering that a system's processor is the largest and most dynamic power consumer of the overall system. The concrete impact of CPU sample power variations is unknown, as comprehensive studies about differences in power consumption for nominally identical systems are currently missing. We address this lack of studies by conducting measurements on four different processor types from two different architectures. For each of these types, we compare up to 30 physical processor samples with a total sum of 90 samples over all processor types. We analyze the variations in power consumption for the different samples using six different workloads over five load levels. Additionally, we analyze how these variations change for different processor core counts and architectures. The results of this paper show that selection of a processor sample can have a statistically significant impact on power consumption. With no correlation to performance, power consumption for nominally identical processors can differ as much as 29.6\% in idle and 19.5\% at full load. We also show that these variations change over different architectures and processor types.}
• 
  {Quantifying the Attack Detection Accuracy of Intrusion Detection Systems in Virtualized Environments}. Milenkoski, Aleksandar; Jayaram, K. R.; Antunes, Nuno; Vieira, Marco; Kounev, Samuel; in {Proceedings of The 27th IEEE International Symposium on Software Reliability Engineering (ISSRE 2016)}IEEE Computer Society, {Washington DC, USA}.

  {Acceptance rate (Full Paper): 45/130 = 34\%}
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  With the widespread adoption of virtualization, intrusion detection systems (IDSes) are increasingly being deployed in virtualized environments. When securing an environment, IT security officers are often faced with the question of how accurate deployed IDSes are at detecting attacks. To this end, metrics for assessing the attack detection accuracy of IDSes have been developed. However, these metrics are defined with respect to a fixed set of hardware resources available to the tested IDS. Therefore, IDSes deployed in virtualized environments featuring elasticity (i.e., on-demand allocation or deallocation of virtualized hardware resources during system operation) cannot be evaluated in an accurate manner using existing metrics. In this paper, we demonstrate the impact of elasticity on IDS attack detection accuracy. In addition, we propose a novel metric and measurement methodology for accurately quantifying the accuracy of IDSes deployed in virtualized environments featuring elasticity. We demonstrate their practical use through case studies involving commonly used IDSes.

  @inproceedings{MiJaAnViKo2016-ISSRE-Quantifying, abstract = {With the widespread adoption of virtualization, intrusion detection systems (IDSes) are increasingly being deployed in virtualized environments. When securing an environment, IT security officers are often faced with the question of how accurate deployed IDSes are at detecting attacks. To this end, metrics for assessing the attack detection accuracy of IDSes have been developed. However, these metrics are defined with respect to a fixed set of hardware resources available to the tested IDS. Therefore, IDSes deployed in virtualized environments featuring elasticity (i.e., on-demand allocation or deallocation of virtualized hardware resources during system operation) cannot be evaluated in an accurate manner using existing metrics. In this paper, we demonstrate the impact of elasticity on IDS attack detection accuracy. In addition, we propose a novel metric and measurement methodology for accurately quantifying the accuracy of IDSes deployed in virtualized environments featuring elasticity. We demonstrate their practical use through case studies involving commonly used IDSes.}, address = {{Washington DC, USA}}, author = {Milenkoski, Aleksandar and Jayaram, K. R. and Antunes, Nuno and Vieira, Marco and Kounev, Samuel}, booktitle = {{Proceedings of The 27th IEEE International Symposium on Software Reliability Engineering (ISSRE 2016)}}, month = {October}, note = {{Acceptance rate (Full Paper): 45/130 = 34\%}}, organization = {IEEE}, publisher = {IEEE Computer Society}, title = {{Quantifying the Attack Detection Accuracy of Intrusion Detection Systems in Virtualized Environments}}, year = 2016 }

  %0 Conference Paper %1 MiJaAnViKo2016-ISSRE-Quantifying %A Milenkoski, Aleksandar %A Jayaram, K. R. %A Antunes, Nuno %A Vieira, Marco %A Kounev, Samuel %B {Proceedings of The 27th IEEE International Symposium on Software Reliability Engineering (ISSRE 2016)} %C {Washington DC, USA} %D 2016 %I IEEE Computer Society %T {Quantifying the Attack Detection Accuracy of Intrusion Detection Systems in Virtualized Environments} %U http://ieeexplore.ieee.org/document/7774527/?reload=true %X With the widespread adoption of virtualization, intrusion detection systems (IDSes) are increasingly being deployed in virtualized environments. When securing an environment, IT security officers are often faced with the question of how accurate deployed IDSes are at detecting attacks. To this end, metrics for assessing the attack detection accuracy of IDSes have been developed. However, these metrics are defined with respect to a fixed set of hardware resources available to the tested IDS. Therefore, IDSes deployed in virtualized environments featuring elasticity (i.e., on-demand allocation or deallocation of virtualized hardware resources during system operation) cannot be evaluated in an accurate manner using existing metrics. In this paper, we demonstrate the impact of elasticity on IDS attack detection accuracy. In addition, we propose a novel metric and measurement methodology for accurately quantifying the accuracy of IDSes deployed in virtualized environments featuring elasticity. We demonstrate their practical use through case studies involving commonly used IDSes.
• 
  {Performance Analysis of SDN Switches with Hardware and Software Flow Tables}. Rygielski, Piotr; Seliuchenko, Marian; Kounev, Samuel; Klymash, Mykhailo; in Proceedings of the 10th EAI International Conference on Performance Evaluation Methodologies and Tools (ValueTools 2016)
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  {Nowadays data centers are increasingly becoming larger and dynamic due to virtualization. Software-Defined Networking is the leading approach to network virtualization and flexible management. The wide variety of hardware implementations have brought strong heterogeneity to the market of networking devices which are different in terms of OpenFlow features and performance. In this paper we address the issue of heterogeneity of four hardware OpenFlow switches by characterizing selected performance relevant parameters for the hardware and software flow tables. We characterize maximum size of hardware flow tables for each switch including the behavior of a rule promotion engine that moves the rules between tables. We show that in the worst case forwarding packets using software table decreases the throughput by two orders of magnitude (from 940 to 14 Mbit/s). Our results can help the developers of SDN applications to account performance limitations of hardware and software processing as well as limited hardware support for a specific rule types. }

  @inproceedings{RySeKoKl2016-valuetools2016-sdn-switches-hw-sw-flow-tables, abstract = {{Nowadays data centers are increasingly becoming larger and dynamic due to virtualization. Software-Defined Networking is the leading approach to network virtualization and flexible management. The wide variety of hardware implementations have brought strong heterogeneity to the market of networking devices which are different in terms of OpenFlow features and performance. In this paper we address the issue of heterogeneity of four hardware OpenFlow switches by characterizing selected performance relevant parameters for the hardware and software flow tables. We characterize maximum size of hardware flow tables for each switch including the behavior of a rule promotion engine that moves the rules between tables. We show that in the worst case forwarding packets using software table decreases the throughput by two orders of magnitude (from 940 to 14 Mbit/s). Our results can help the developers of SDN applications to account performance limitations of hardware and software processing as well as limited hardware support for a specific rule types. }}, author = {Rygielski, Piotr and Seliuchenko, Marian and Kounev, Samuel and Klymash, Mykhailo}, booktitle = {Proceedings of the 10th EAI International Conference on Performance Evaluation Methodologies and Tools (ValueTools 2016)}, month = {October}, title = {{Performance Analysis of SDN Switches with Hardware and Software Flow Tables}}, year = 2016 }

  %0 Conference Paper %1 RySeKoKl2016-valuetools2016-sdn-switches-hw-sw-flow-tables %A Rygielski, Piotr %A Seliuchenko, Marian %A Kounev, Samuel %A Klymash, Mykhailo %B Proceedings of the 10th EAI International Conference on Performance Evaluation Methodologies and Tools (ValueTools 2016) %D 2016 %T {Performance Analysis of SDN Switches with Hardware and Software Flow Tables} %X {Nowadays data centers are increasingly becoming larger and dynamic due to virtualization. Software-Defined Networking is the leading approach to network virtualization and flexible management. The wide variety of hardware implementations have brought strong heterogeneity to the market of networking devices which are different in terms of OpenFlow features and performance. In this paper we address the issue of heterogeneity of four hardware OpenFlow switches by characterizing selected performance relevant parameters for the hardware and software flow tables. We characterize maximum size of hardware flow tables for each switch including the behavior of a rule promotion engine that moves the rules between tables. We show that in the worst case forwarding packets using software table decreases the throughput by two orders of magnitude (from 940 to 14 Mbit/s). Our results can help the developers of SDN applications to account performance limitations of hardware and software processing as well as limited hardware support for a specific rule types. }

• 
  {Flexible Performance Prediction of Data Center Networks using Automatically Generated Simulation Models}. Rygielski, Piotr; Kounev, Samuel; Tran-Gia, Phuoc; in Proceedings of the Eighth International Conference on Simulation Tools and Techniques (SIMUTools 2015) 119–128.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  {Using different modeling and simulation approaches for predicting network performance requires extensive experience and involves a number of time consuming manual steps regarding each of the modeling formalisms. In this paper, we propose a generic approach to modeling the performance of data center networks. The approach offers multiple performance models but requires to use only a single modeling language. We propose a two-step modeling methodology, in which a high-level descriptive model of the network is built in the first step, and in the second step model-to-model transformations are used to automatically transform the descriptive model to different network simulation models. We automatically generate three performance models defined at different levels of abstraction to analyze network throughput. By offering multiple simulation models in parallel, we provide flexibility in trading-off between the modeling accuracy and the simulation overhead. We analyze the simulation models by comparing the prediction accuracy with respect to the simulation duration. We observe, that in the investigated scenarios the solution duration of coarser simulation models is up to 300 times shorter, whereas the average prediction accuracy decreases only by 4 percent.}

  @inproceedings{RyKoTr2015-SIMUTools-FlexPerformancePrediction, abstract = {{Using different modeling and simulation approaches for predicting network performance requires extensive experience and involves a number of time consuming manual steps regarding each of the modeling formalisms. In this paper, we propose a generic approach to modeling the performance of data center networks. The approach offers multiple performance models but requires to use only a single modeling language. We propose a two-step modeling methodology, in which a high-level descriptive model of the network is built in the first step, and in the second step model-to-model transformations are used to automatically transform the descriptive model to different network simulation models. We automatically generate three performance models defined at different levels of abstraction to analyze network throughput. By offering multiple simulation models in parallel, we provide flexibility in trading-off between the modeling accuracy and the simulation overhead. We analyze the simulation models by comparing the prediction accuracy with respect to the simulation duration. We observe, that in the investigated scenarios the solution duration of coarser simulation models is up to 300 times shorter, whereas the average prediction accuracy decreases only by 4 percent.}}, author = {Rygielski, Piotr and Kounev, Samuel and Tran-Gia, Phuoc}, booktitle = {Proceedings of the Eighth International Conference on Simulation Tools and Techniques (SIMUTools 2015)}, month = {August}, pages = {119--128}, title = {{Flexible Performance Prediction of Data Center Networks using Automatically Generated Simulation Models}}, year = 2015 }

  %0 Conference Paper %1 RyKoTr2015-SIMUTools-FlexPerformancePrediction %A Rygielski, Piotr %A Kounev, Samuel %A Tran-Gia, Phuoc %B Proceedings of the Eighth International Conference on Simulation Tools and Techniques (SIMUTools 2015) %D 2015 %P 119--128 %T {Flexible Performance Prediction of Data Center Networks using Automatically Generated Simulation Models} %X {Using different modeling and simulation approaches for predicting network performance requires extensive experience and involves a number of time consuming manual steps regarding each of the modeling formalisms. In this paper, we propose a generic approach to modeling the performance of data center networks. The approach offers multiple performance models but requires to use only a single modeling language. We propose a two-step modeling methodology, in which a high-level descriptive model of the network is built in the first step, and in the second step model-to-model transformations are used to automatically transform the descriptive model to different network simulation models. We automatically generate three performance models defined at different levels of abstraction to analyze network throughput. By offering multiple simulation models in parallel, we provide flexibility in trading-off between the modeling accuracy and the simulation overhead. We analyze the simulation models by comparing the prediction accuracy with respect to the simulation duration. We observe, that in the investigated scenarios the solution duration of coarser simulation models is up to 300 times shorter, whereas the average prediction accuracy decreases only by 4 percent.}
• 
  {Proactive Memory Scaling of Virtualized Applications}. Spinner, Simon; Herbst, Nikolas; Kounev, Samuel; Zhu, Xiaoyun; Lu, Lei; Uysal, Mustafa; Griffith, Rean; in Proceedings of the 2015 IEEE 8th International Conference on Cloud Computing (IEEE CLOUD 2015) 277–284. IEEE.

  Acceptance Rate: 15%
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  {Enterprise applications in virtualized environments are often subject to time-varying workloads with multiple seasonal patterns and trends. In order to ensure quality of service for such applications while avoiding over-provisioning, resources need to be dynamically adapted to accommodate the current workload demands. Many memory-intensive applications are not suitable for the traditional horizontal scaling approach often used for runtime performance management, as it relies on complex and expensive state replication. On the other hand, vertical scaling of memory often requires a restart of the application. In this paper, we propose a proactive approach to memory scaling for virtualized applications. It uses statistical forecasting to predict the future workload and reconfigure the memory size of the virtual machine of an application automatically. To this end, we propose an extended forecasting technique that leverages meta-knowledge, such as calendar information, to improve the forecast accuracy. In addition, we develop an application controller to adjust settings associated with application memory management during memory reconfiguration. Our evaluation using real-world traces shows that the forecast accuracy quantified with the MASE error metric can be improved by 11-59%. Furthermore, we demonstrate that the proactive approach can reduce the impact of reconfiguration on application availability by over 80% and significantly improve performance relative to a reactive controller.}

  @inproceedings{SpHeKoZhLuUyGr2015-CLOUD-ProactiveMemoryScaling, abstract = {{Enterprise applications in virtualized environments are often subject to time-varying workloads with multiple seasonal patterns and trends. In order to ensure quality of service for such applications while avoiding over-provisioning, resources need to be dynamically adapted to accommodate the current workload demands. Many memory-intensive applications are not suitable for the traditional horizontal scaling approach often used for runtime performance management, as it relies on complex and expensive state replication. On the other hand, vertical scaling of memory often requires a restart of the application. In this paper, we propose a proactive approach to memory scaling for virtualized applications. It uses statistical forecasting to predict the future workload and reconfigure the memory size of the virtual machine of an application automatically. To this end, we propose an extended forecasting technique that leverages meta-knowledge, such as calendar information, to improve the forecast accuracy. In addition, we develop an application controller to adjust settings associated with application memory management during memory reconfiguration. Our evaluation using real-world traces shows that the forecast accuracy quantified with the MASE error metric can be improved by 11-59%. Furthermore, we demonstrate that the proactive approach can reduce the impact of reconfiguration on application availability by over 80% and significantly improve performance relative to a reactive controller.}}, author = {Spinner, Simon and Herbst, Nikolas and Kounev, Samuel and Zhu, Xiaoyun and Lu, Lei and Uysal, Mustafa and Griffith, Rean}, booktitle = {Proceedings of the 2015 IEEE 8th International Conference on Cloud Computing (IEEE CLOUD 2015)}, month = {June}, note = {Acceptance Rate: 15%}, pages = {277--284}, publisher = {IEEE}, title = {{Proactive Memory Scaling of Virtualized Applications}}, year = 2015 }

  %0 Conference Paper %1 SpHeKoZhLuUyGr2015-CLOUD-ProactiveMemoryScaling %A Spinner, Simon %A Herbst, Nikolas %A Kounev, Samuel %A Zhu, Xiaoyun %A Lu, Lei %A Uysal, Mustafa %A Griffith, Rean %B Proceedings of the 2015 IEEE 8th International Conference on Cloud Computing (IEEE CLOUD 2015) %D 2015 %I IEEE %P 277--284 %T {Proactive Memory Scaling of Virtualized Applications} %X {Enterprise applications in virtualized environments are often subject to time-varying workloads with multiple seasonal patterns and trends. In order to ensure quality of service for such applications while avoiding over-provisioning, resources need to be dynamically adapted to accommodate the current workload demands. Many memory-intensive applications are not suitable for the traditional horizontal scaling approach often used for runtime performance management, as it relies on complex and expensive state replication. On the other hand, vertical scaling of memory often requires a restart of the application. In this paper, we propose a proactive approach to memory scaling for virtualized applications. It uses statistical forecasting to predict the future workload and reconfigure the memory size of the virtual machine of an application automatically. To this end, we propose an extended forecasting technique that leverages meta-knowledge, such as calendar information, to improve the forecast accuracy. In addition, we develop an application controller to adjust settings associated with application memory management during memory reconfiguration. Our evaluation using real-world traces shows that the forecast accuracy quantified with the MASE error metric can be improved by 11-59%. Furthermore, we demonstrate that the proactive approach can reduce the impact of reconfiguration on application availability by over 80% and significantly improve performance relative to a reactive controller.}
• 
  {Analysis of the Influences on Server Power Consumption and Energy Efficiency for CPU-Intensive Workloads}. von Kistowski, J{\’o}akim; Block, Hansfried; Beckett, John; Lange, Klaus-Dieter; Arnold, Jeremy A.; Kounev, Samuel; in Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering (ICPE 2015)ACM, New York, NY, USA.

  acceptance rate: 27\%
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  { Energy efficiency of servers has become a significant research topic over the last years, as server energy consumption varies depending on multiple factors, such as server utilization and workload type. Server energy analysis and estimation must take all relevant factors into account to ensure reliable estimates and conclusions. Thorough system analysis requires benchmarks capable of testing different system resources at different load levels using multiple workload types. Server energy estimation approaches, on the other hand, require knowledge about the interactions of these factors for the creation of accurate power models. Common approaches to energy-aware workload classification classify workloads depending on the resource types used by the different workloads. However, they rarely take into account differences in workloads targeting the same resources. Industrial energy-efficiency benchmarks typically do not evaluate the system's energy consumption at different resource load levels, and they only provide data for system analysis at maximum system load. In this paper, we benchmark multiple server configurations using the CPU worklets included in SPEC's Server Efficiency Rating Tool (SERT). We evaluate the impact of load levels and different CPU workloads on power consumption and energy efficiency. We analyze how functions approximating the measured power consumption differ over multiple server configurations and architectures. We show that workloads targeting the same resource can differ significantly in their power draw and energy efficiency. The power consumption of a given workload type varies depending on utilization, hardware and software configuration. The power consumption of CPU-intensive workloads does not scale uniformly with increased load, nor do hardware or software configuration changes affect it in a uniform manner.}

  @inproceedings{KiBlBeLaArKo2015-ICPE-SERT, abstract = {{ Energy efficiency of servers has become a significant research topic over the last years, as server energy consumption varies depending on multiple factors, such as server utilization and workload type. Server energy analysis and estimation must take all relevant factors into account to ensure reliable estimates and conclusions. Thorough system analysis requires benchmarks capable of testing different system resources at different load levels using multiple workload types. Server energy estimation approaches, on the other hand, require knowledge about the interactions of these factors for the creation of accurate power models. Common approaches to energy-aware workload classification classify workloads depending on the resource types used by the different workloads. However, they rarely take into account differences in workloads targeting the same resources. Industrial energy-efficiency benchmarks typically do not evaluate the system's energy consumption at different resource load levels, and they only provide data for system analysis at maximum system load. In this paper, we benchmark multiple server configurations using the CPU worklets included in SPEC's Server Efficiency Rating Tool (SERT). We evaluate the impact of load levels and different CPU workloads on power consumption and energy efficiency. We analyze how functions approximating the measured power consumption differ over multiple server configurations and architectures. We show that workloads targeting the same resource can differ significantly in their power draw and energy efficiency. The power consumption of a given workload type varies depending on utilization, hardware and software configuration. The power consumption of CPU-intensive workloads does not scale uniformly with increased load, nor do hardware or software configuration changes affect it in a uniform manner.}}, address = {New York, NY, USA}, author = {von Kistowski, J{\'o}akim and Block, Hansfried and Beckett, John and Lange, Klaus-Dieter and Arnold, Jeremy A. and Kounev, Samuel}, booktitle = {Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering (ICPE 2015)}, month = {February}, note = {acceptance rate: 27\%}, publisher = {ACM}, series = {ICPE '15}, title = {{Analysis of the Influences on Server Power Consumption and Energy Efficiency for CPU-Intensive Workloads}}, year = 2015 }

  %0 Conference Paper %1 KiBlBeLaArKo2015-ICPE-SERT %A von Kistowski, J{\'o}akim %A Block, Hansfried %A Beckett, John %A Lange, Klaus-Dieter %A Arnold, Jeremy A. %A Kounev, Samuel %B Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering (ICPE 2015) %C New York, NY, USA %D 2015 %I ACM %T {Analysis of the Influences on Server Power Consumption and Energy Efficiency for CPU-Intensive Workloads} %X { Energy efficiency of servers has become a significant research topic over the last years, as server energy consumption varies depending on multiple factors, such as server utilization and workload type. Server energy analysis and estimation must take all relevant factors into account to ensure reliable estimates and conclusions. Thorough system analysis requires benchmarks capable of testing different system resources at different load levels using multiple workload types. Server energy estimation approaches, on the other hand, require knowledge about the interactions of these factors for the creation of accurate power models. Common approaches to energy-aware workload classification classify workloads depending on the resource types used by the different workloads. However, they rarely take into account differences in workloads targeting the same resources. Industrial energy-efficiency benchmarks typically do not evaluate the system's energy consumption at different resource load levels, and they only provide data for system analysis at maximum system load. In this paper, we benchmark multiple server configurations using the CPU worklets included in SPEC's Server Efficiency Rating Tool (SERT). We evaluate the impact of load levels and different CPU workloads on power consumption and energy efficiency. We analyze how functions approximating the measured power consumption differ over multiple server configurations and architectures. We show that workloads targeting the same resource can differ significantly in their power draw and energy efficiency. The power consumption of a given workload type varies depending on utilization, hardware and software configuration. The power consumption of CPU-intensive workloads does not scale uniformly with increased load, nor do hardware or software configuration changes affect it in a uniform manner.}
• 
  {Automated Workload Characterization for I/O Performance Analysis in Virtualized Environments}. Busch, Axel; Noorshams, Qais; Kounev, Samuel; Koziolek, Anne; Reussner, Ralf; Amrehn, Erich; in Proceedings of the ACM/SPEC International Conference on Performance EngineeringACM, New York, NY, USA.

  Acceptance Rate (Full Paper): 15/56 = 27\%.
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  @inproceedings{BuNoKoKoReAm2015-ICPE-WorkloadCharacterization, address = {New York, NY, USA}, author = {Busch, Axel and Noorshams, Qais and Kounev, Samuel and Koziolek, Anne and Reussner, Ralf and Amrehn, Erich}, booktitle = {Proceedings of the ACM/SPEC International Conference on Performance Engineering}, month = {February}, note = {Acceptance Rate (Full Paper): 15/56 = 27\%.}, publisher = {ACM}, series = {ICPE '15}, title = {{Automated Workload Characterization for I/O Performance Analysis in Virtualized Environments}}, year = 2015 }

  %0 Conference Paper %1 BuNoKoKoReAm2015-ICPE-WorkloadCharacterization %A Busch, Axel %A Noorshams, Qais %A Kounev, Samuel %A Koziolek, Anne %A Reussner, Ralf %A Amrehn, Erich %B Proceedings of the ACM/SPEC International Conference on Performance Engineering %C New York, NY, USA %D 2015 %I ACM %T {Automated Workload Characterization for I/O Performance Analysis in Virtualized Environments} %U http://dx.doi.org/10.1145/2668930.2688050
• 
  {BUNGEE: An Elasticity Benchmark for Self-Adaptive IaaS Cloud Environments}. Herbst, Nikolas Roman; Kounev, Samuel; Weber, Andreas; Groenda, Henning; in Proceedings of the 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS 2015)

  Acceptance rate: 29\%
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Today's infrastructure clouds provide resource elasticity (i.e. auto-scaling) mechanisms enabling self-adaptive resource provisioning to reflect variations in the load intensity over time. These mechanisms impact on the application performance, however, their effect in specific situations is hard to quantify and compare. To evaluate the quality of elasticity mechanisms provided by different platforms and configurations, respective metrics and benchmarks are required. Existing metrics for elasticity only consider the time required to provision and deprovision resources or the costs impact of adaptations. Existing benchmarks lack the capability to handle open workloads with realistic load intensity profiles and do not explicitly distinguish between the performance exhibited by the provisioned underlying resources, on the one hand, and the quality of the elasticity mechanisms themselves, on the other hand. In this paper, we propose reliable metrics for quantifying the timing aspects and accuracy of elasticity. Based on these metrics, we propose a novel approach for benchmarking the elasticity of Infrastructure-as-a-Service (IaaS) cloud platforms independent of the performance exhibited by the provisioned underlying resources. We show that the proposed metrics provide consistent ranking of elastic platforms on an ordinal scale. Finally, we present an extensive case study of real-world complexity demonstrating that the proposed approach is applicable in realistic scenarios and can cope with different levels of resource efficiency.

  @inproceedings{HeKoWeGr_2015_SEAMS_BUNGEE, abstract = {Today's infrastructure clouds provide resource elasticity (i.e. auto-scaling) mechanisms enabling self-adaptive resource provisioning to reflect variations in the load intensity over time. These mechanisms impact on the application performance, however, their effect in specific situations is hard to quantify and compare. To evaluate the quality of elasticity mechanisms provided by different platforms and configurations, respective metrics and benchmarks are required. Existing metrics for elasticity only consider the time required to provision and deprovision resources or the costs impact of adaptations. Existing benchmarks lack the capability to handle open workloads with realistic load intensity profiles and do not explicitly distinguish between the performance exhibited by the provisioned underlying resources, on the one hand, and the quality of the elasticity mechanisms themselves, on the other hand. In this paper, we propose reliable metrics for quantifying the timing aspects and accuracy of elasticity. Based on these metrics, we propose a novel approach for benchmarking the elasticity of Infrastructure-as-a-Service (IaaS) cloud platforms independent of the performance exhibited by the provisioned underlying resources. We show that the proposed metrics provide consistent ranking of elastic platforms on an ordinal scale. Finally, we present an extensive case study of real-world complexity demonstrating that the proposed approach is applicable in realistic scenarios and can cope with different levels of resource efficiency.}, author = {Herbst, Nikolas Roman and Kounev, Samuel and Weber, Andreas and Groenda, Henning}, booktitle = {Proceedings of the 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS 2015)}, month = {May}, note = {Acceptance rate: 29\%}, title = {{BUNGEE: An Elasticity Benchmark for Self-Adaptive IaaS Cloud Environments}}, year = 2015 }

  %0 Conference Paper %1 HeKoWeGr_2015_SEAMS_BUNGEE %A Herbst, Nikolas Roman %A Kounev, Samuel %A Weber, Andreas %A Groenda, Henning %B Proceedings of the 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS 2015) %D 2015 %T {BUNGEE: An Elasticity Benchmark for Self-Adaptive IaaS Cloud Environments} %X Today's infrastructure clouds provide resource elasticity (i.e. auto-scaling) mechanisms enabling self-adaptive resource provisioning to reflect variations in the load intensity over time. These mechanisms impact on the application performance, however, their effect in specific situations is hard to quantify and compare. To evaluate the quality of elasticity mechanisms provided by different platforms and configurations, respective metrics and benchmarks are required. Existing metrics for elasticity only consider the time required to provision and deprovision resources or the costs impact of adaptations. Existing benchmarks lack the capability to handle open workloads with realistic load intensity profiles and do not explicitly distinguish between the performance exhibited by the provisioned underlying resources, on the one hand, and the quality of the elasticity mechanisms themselves, on the other hand. In this paper, we propose reliable metrics for quantifying the timing aspects and accuracy of elasticity. Based on these metrics, we propose a novel approach for benchmarking the elasticity of Infrastructure-as-a-Service (IaaS) cloud platforms independent of the performance exhibited by the provisioned underlying resources. We show that the proposed metrics provide consistent ranking of elastic platforms on an ordinal scale. Finally, we present an extensive case study of real-world complexity demonstrating that the proposed approach is applicable in realistic scenarios and can cope with different levels of resource efficiency.
• 
  {Modeling and Extracting Load Intensity Profiles}. von Kistowski, J{\’o}akim; Herbst, Nikolas Roman; Zoller, Daniel; Kounev, Samuel; Hotho, Andreas; in Proceedings of the 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS 2015)

  Acceptance rate: 29\%
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Today's system developers and operators face the challenge of creating software systems that make efficient use of dynamically allocated resources under highly variable and dynamic load profiles, while at the same time delivering reliable performance. Benchmarking of systems under these constraints is difficult, as state-of-the-art benchmarking frameworks provide only limited support for emulating such dynamic and highly variable load profiles for the creation of realistic workload scenarios. Industrial benchmarks typically confine themselves to workloads with constant or stepwise increasing loads. Alternatively, they support replaying of recorded load traces. Statistical load intensity descriptions also do not sufficiently capture concrete pattern load profile variations over time. To address these issues, we present the Descartes Load Intensity Model (DLIM). DLIM provides a modeling formalism for describing load intensity variations over time. A DLIM instance can be used as a compact representation of a recorded load intensity trace, providing a powerful tool for benchmarking and performance analysis. As manually obtaining DLIM instances can be time consuming, we present three different automated extraction methods, which also help to enable autonomous system analysis for self-adaptive systems. Model expressiveness is validated using the presented extraction methods. Extracted DLIM instances exhibit a median modeling error of 12.4% on average over nine different real-world traces covering between two weeks and seven months. Additionally, extraction methods perform orders of magnitude faster than existing time series decomposition approaches.

  @inproceedings{KiHeZoKoHo2015_SEAMS_DLIMExtraction, abstract = {Today's system developers and operators face the challenge of creating software systems that make efficient use of dynamically allocated resources under highly variable and dynamic load profiles, while at the same time delivering reliable performance. Benchmarking of systems under these constraints is difficult, as state-of-the-art benchmarking frameworks provide only limited support for emulating such dynamic and highly variable load profiles for the creation of realistic workload scenarios. Industrial benchmarks typically confine themselves to workloads with constant or stepwise increasing loads. Alternatively, they support replaying of recorded load traces. Statistical load intensity descriptions also do not sufficiently capture concrete pattern load profile variations over time. To address these issues, we present the Descartes Load Intensity Model (DLIM). DLIM provides a modeling formalism for describing load intensity variations over time. A DLIM instance can be used as a compact representation of a recorded load intensity trace, providing a powerful tool for benchmarking and performance analysis. As manually obtaining DLIM instances can be time consuming, we present three different automated extraction methods, which also help to enable autonomous system analysis for self-adaptive systems. Model expressiveness is validated using the presented extraction methods. Extracted DLIM instances exhibit a median modeling error of 12.4% on average over nine different real-world traces covering between two weeks and seven months. Additionally, extraction methods perform orders of magnitude faster than existing time series decomposition approaches.}, author = {von Kistowski, J{\'o}akim and Herbst, Nikolas Roman and Zoller, Daniel and Kounev, Samuel and Hotho, Andreas}, booktitle = {Proceedings of the 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS 2015)}, month = {may}, note = {Acceptance rate: 29\%}, title = {{Modeling and Extracting Load Intensity Profiles}}, year = 2015 }

  %0 Conference Paper %1 KiHeZoKoHo2015_SEAMS_DLIMExtraction %A von Kistowski, J{\'o}akim %A Herbst, Nikolas Roman %A Zoller, Daniel %A Kounev, Samuel %A Hotho, Andreas %B Proceedings of the 10th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS 2015) %D 2015 %T {Modeling and Extracting Load Intensity Profiles} %X Today's system developers and operators face the challenge of creating software systems that make efficient use of dynamically allocated resources under highly variable and dynamic load profiles, while at the same time delivering reliable performance. Benchmarking of systems under these constraints is difficult, as state-of-the-art benchmarking frameworks provide only limited support for emulating such dynamic and highly variable load profiles for the creation of realistic workload scenarios. Industrial benchmarks typically confine themselves to workloads with constant or stepwise increasing loads. Alternatively, they support replaying of recorded load traces. Statistical load intensity descriptions also do not sufficiently capture concrete pattern load profile variations over time. To address these issues, we present the Descartes Load Intensity Model (DLIM). DLIM provides a modeling formalism for describing load intensity variations over time. A DLIM instance can be used as a compact representation of a recorded load intensity trace, providing a powerful tool for benchmarking and performance analysis. As manually obtaining DLIM instances can be time consuming, we present three different automated extraction methods, which also help to enable autonomous system analysis for self-adaptive systems. Model expressiveness is validated using the presented extraction methods. Extracted DLIM instances exhibit a median modeling error of 12.4% on average over nine different real-world traces covering between two weeks and seven months. Additionally, extraction methods perform orders of magnitude faster than existing time series decomposition approaches.
• 
  {Parallel Simulation of Queueing Petri Nets}. Walter, J{\"u}rgen; Spinner, Simon; Kounev, Samuel; in Proceedings of the Eighth EAI International Conference on Simulation Tools and Techniques (SIMUTools 2015)
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Queueing Petri nets show high accurracy to model and analyze cloud scenarios. To apply these models in online scenarios a speedup of simulation is desirable. One way to speedup simulation is parallelization. Decades of intensive research showed no general solution for parallel discrete event simulation, which provides reasonable speedups. We research the application of parallel simulation for cloud scenarios. We develop a parallel simulation engine that shows speedup for certain scenarios common in cloud models. We present a parallel simualtion engine that employs application level and event level parallelism. We parallelize the execution of a single run by the use of a conservative barrier-based synchronization algorithm. The speedup for a single run depends on the capability of the model. Hence, a number of experiments on different net characteristics were conducted showing that for certain models a superlinear speedup is possible.

  @inproceedings{WaSpKo-SIMUTools-QPNParallelSimulation, abstract = {Queueing Petri nets show high accurracy to model and analyze cloud scenarios. To apply these models in online scenarios a speedup of simulation is desirable. One way to speedup simulation is parallelization. Decades of intensive research showed no general solution for parallel discrete event simulation, which provides reasonable speedups. We research the application of parallel simulation for cloud scenarios. We develop a parallel simulation engine that shows speedup for certain scenarios common in cloud models. We present a parallel simualtion engine that employs application level and event level parallelism. We parallelize the execution of a single run by the use of a conservative barrier-based synchronization algorithm. The speedup for a single run depends on the capability of the model. Hence, a number of experiments on different net characteristics were conducted showing that for certain models a superlinear speedup is possible.}, author = {Walter, J{\"u}rgen and Spinner, Simon and Kounev, Samuel}, booktitle = {Proceedings of the Eighth EAI International Conference on Simulation Tools and Techniques (SIMUTools 2015)}, month = {August}, title = {{Parallel Simulation of Queueing Petri Nets}}, year = 2015 }

  %0 Conference Paper %1 WaSpKo-SIMUTools-QPNParallelSimulation %A Walter, J{\"u}rgen %A Spinner, Simon %A Kounev, Samuel %B Proceedings of the Eighth EAI International Conference on Simulation Tools and Techniques (SIMUTools 2015) %D 2015 %T {Parallel Simulation of Queueing Petri Nets} %X Queueing Petri nets show high accurracy to model and analyze cloud scenarios. To apply these models in online scenarios a speedup of simulation is desirable. One way to speedup simulation is parallelization. Decades of intensive research showed no general solution for parallel discrete event simulation, which provides reasonable speedups. We research the application of parallel simulation for cloud scenarios. We develop a parallel simulation engine that shows speedup for certain scenarios common in cloud models. We present a parallel simualtion engine that employs application level and event level parallelism. We parallelize the execution of a single run by the use of a conservative barrier-based synchronization algorithm. The speedup for a single run depends on the capability of the model. Hence, a number of experiments on different net characteristics were conducted showing that for certain models a superlinear speedup is possible.
• 
  {Specialized Heuristics for the Controller Placement Problem in Large Scale SDN Networks}. Lange, Stanislav; Gebert, Steffen; Spoerhase, Joachim; Rygielski, Piotr; Zinner, Thomas; Kounev, Samuel; Tran-Gia, Phuoc; in 27th International Teletraffic Congress ({ITC 2015}) 210–218. Ghent, Belgium.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  {The Software Defined Networking~(SDN) concept introduces a paradigm shift in the networking world towards an externalized control plane which is logically centralized. When designing an SDN-based WAN architecture, it is of vital importance to find a feasible solution to the controller placement problem, i.e., to decide where to position a limited amount of resources within the network. In addition to time-independent constraints regarding aspects like scalability, resilience, and control plane communication delays, dynamically changing network conditions like traffic patterns or bandwidth demands need to be considered as well. Consequently, such dynamic environments call for a regular and fast recalculation of placements in order to adapt to the current situation in a timely manner. While an exhaustive evaluation of all possible solutions can be performed within a practically feasible time frame for small and medium-sized networks, such an approach is out of scope for large problem instances which have significantly higher time and memory requirements. Therefore, this work investigates a specialized heuristic, which takes into account a particular set of optimization objectives and returns solutions representing the possible trade-offs between them. Due to its low computation time and acceptable margin of error, this heuristic can be employed by automatic decision systems operating in dynamic environments.}

  @inproceedings{LaGeSpRyZiKoTG-SDN-Controller-Placement-ITC2015, abstract = {{The Software Defined Networking~(SDN) concept introduces a paradigm shift in the networking world towards an externalized control plane which is logically centralized. When designing an SDN-based WAN architecture, it is of vital importance to find a feasible solution to the controller placement problem, i.e., to decide where to position a limited amount of resources within the network. In addition to time-independent constraints regarding aspects like scalability, resilience, and control plane communication delays, dynamically changing network conditions like traffic patterns or bandwidth demands need to be considered as well. Consequently, such dynamic environments call for a regular and fast recalculation of placements in order to adapt to the current situation in a timely manner. While an exhaustive evaluation of all possible solutions can be performed within a practically feasible time frame for small and medium-sized networks, such an approach is out of scope for large problem instances which have significantly higher time and memory requirements. Therefore, this work investigates a specialized heuristic, which takes into account a particular set of optimization objectives and returns solutions representing the possible trade-offs between them. Due to its low computation time and acceptable margin of error, this heuristic can be employed by automatic decision systems operating in dynamic environments.}}, address = {Ghent, Belgium}, author = {Lange, Stanislav and Gebert, Steffen and Spoerhase, Joachim and Rygielski, Piotr and Zinner, Thomas and Kounev, Samuel and Tran-Gia, Phuoc}, booktitle = {27th International Teletraffic Congress ({ITC 2015})}, month = {September}, pages = {210--218}, title = {{Specialized Heuristics for the Controller Placement Problem in Large Scale SDN Networks}}, year = 2015 }

  %0 Conference Paper %1 LaGeSpRyZiKoTG-SDN-Controller-Placement-ITC2015 %A Lange, Stanislav %A Gebert, Steffen %A Spoerhase, Joachim %A Rygielski, Piotr %A Zinner, Thomas %A Kounev, Samuel %A Tran-Gia, Phuoc %B 27th International Teletraffic Congress ({ITC 2015}) %C Ghent, Belgium %D 2015 %P 210--218 %T {Specialized Heuristics for the Controller Placement Problem in Large Scale SDN Networks} %U http://dx.doi.org/10.1109/ITC.2015.32 %X {The Software Defined Networking~(SDN) concept introduces a paradigm shift in the networking world towards an externalized control plane which is logically centralized. When designing an SDN-based WAN architecture, it is of vital importance to find a feasible solution to the controller placement problem, i.e., to decide where to position a limited amount of resources within the network. In addition to time-independent constraints regarding aspects like scalability, resilience, and control plane communication delays, dynamically changing network conditions like traffic patterns or bandwidth demands need to be considered as well. Consequently, such dynamic environments call for a regular and fast recalculation of placements in order to adapt to the current situation in a timely manner. While an exhaustive evaluation of all possible solutions can be performed within a practically feasible time frame for small and medium-sized networks, such an approach is out of scope for large problem instances which have significantly higher time and memory requirements. Therefore, this work investigates a specialized heuristic, which takes into account a particular set of optimization objectives and returns solutions representing the possible trade-offs between them. Due to its low computation time and acceptable margin of error, this heuristic can be employed by automatic decision systems operating in dynamic environments.}
• 
  {Energy Efficiency of Hierarchical Server Load Distribution Strategies}. von Kistowski, J{\’o}akim; Beckett, John; Lange, Klaus-Dieter; Block, Hansfried; Arnold, Jeremy A.; Kounev, Samuel; in Proceedings of the IEEE 23nd International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS 2015)IEEE.

  Full paper acceptance rate: 19\%
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  {Energy efficiency of servers has become a significant issue over the last years. Load distribution plays a crucial role in the improvement of energy efficiency as (un-)balancing strategies can be leveraged to distribute load over one or multiple systems in a way in which resources are utilized at high performance, yet low overall power consumption. This can be achieved on multiple levels, from load distribution on single CPU cores to machine level load balancing on distributed systems. With modern day server architectures providing load balancing opportunities at several layers, answering the question of optimal load distribution has become non-trivial. Work has to be distributed hierarchically in a fashion that enables maximum energy efficiency at each level. Current approaches balance load based on generalized assumptions about the energy efficiency of servers. These assumptions are based either on very machine-specific or highly generalized observations that may or may not hold true over a variety of systems and configurations. In this paper, we use a modified version of the SPEC SERT suite to measure the energy efficiency of a variety of hierarchical load distribution strategies on single and multi-node systems. We introduce a new strategy and evaluate energy efficiency for homogeneous and heterogeneous workloads over different hardware configurations. Our results show that the selection of a load distribution strategy depends heavily on workload, system utilization, as well as hardware. Used in conjunction with existing strategies, our new load distribution strategy can reduce a single system's power consumption by up to 10.7%.}

  @inproceedings{KiBeLaBlArKo2015-MASCOTS, abstract = {{Energy efficiency of servers has become a significant issue over the last years. Load distribution plays a crucial role in the improvement of energy efficiency as (un-)balancing strategies can be leveraged to distribute load over one or multiple systems in a way in which resources are utilized at high performance, yet low overall power consumption. This can be achieved on multiple levels, from load distribution on single CPU cores to machine level load balancing on distributed systems. With modern day server architectures providing load balancing opportunities at several layers, answering the question of optimal load distribution has become non-trivial. Work has to be distributed hierarchically in a fashion that enables maximum energy efficiency at each level. Current approaches balance load based on generalized assumptions about the energy efficiency of servers. These assumptions are based either on very machine-specific or highly generalized observations that may or may not hold true over a variety of systems and configurations. In this paper, we use a modified version of the SPEC SERT suite to measure the energy efficiency of a variety of hierarchical load distribution strategies on single and multi-node systems. We introduce a new strategy and evaluate energy efficiency for homogeneous and heterogeneous workloads over different hardware configurations. Our results show that the selection of a load distribution strategy depends heavily on workload, system utilization, as well as hardware. Used in conjunction with existing strategies, our new load distribution strategy can reduce a single system's power consumption by up to 10.7%.}}, author = {von Kistowski, J{\'o}akim and Beckett, John and Lange, Klaus-Dieter and Block, Hansfried and Arnold, Jeremy A. and Kounev, Samuel}, booktitle = {Proceedings of the IEEE 23nd International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS 2015)}, month = {October}, note = {Full paper acceptance rate: 19\%}, publisher = {IEEE}, title = {{Energy Efficiency of Hierarchical Server Load Distribution Strategies}}, year = 2015 }

  %0 Conference Paper %1 KiBeLaBlArKo2015-MASCOTS %A von Kistowski, J{\'o}akim %A Beckett, John %A Lange, Klaus-Dieter %A Block, Hansfried %A Arnold, Jeremy A. %A Kounev, Samuel %B Proceedings of the IEEE 23nd International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS 2015) %D 2015 %I IEEE %T {Energy Efficiency of Hierarchical Server Load Distribution Strategies} %X {Energy efficiency of servers has become a significant issue over the last years. Load distribution plays a crucial role in the improvement of energy efficiency as (un-)balancing strategies can be leveraged to distribute load over one or multiple systems in a way in which resources are utilized at high performance, yet low overall power consumption. This can be achieved on multiple levels, from load distribution on single CPU cores to machine level load balancing on distributed systems. With modern day server architectures providing load balancing opportunities at several layers, answering the question of optimal load distribution has become non-trivial. Work has to be distributed hierarchically in a fashion that enables maximum energy efficiency at each level. Current approaches balance load based on generalized assumptions about the energy efficiency of servers. These assumptions are based either on very machine-specific or highly generalized observations that may or may not hold true over a variety of systems and configurations. In this paper, we use a modified version of the SPEC SERT suite to measure the energy efficiency of a variety of hierarchical load distribution strategies on single and multi-node systems. We introduce a new strategy and evaluate energy efficiency for homogeneous and heterogeneous workloads over different hardware configurations. Our results show that the selection of a load distribution strategy depends heavily on workload, system utilization, as well as hardware. Used in conjunction with existing strategies, our new load distribution strategy can reduce a single system's power consumption by up to 10.7%.}
• 
  {Model-based Autonomic and Performance-aware System Adaptation in Heterogeneous Resource Environments: A Case Study}. Huber, Nikolaus; Walter, J{\"u}rgen; B{\"a}hr, Manuel; Kounev, Samuel; in Proceedings of the 2015 IEEE International Conference on Cloud and Autonomic Computing (ICCAC)IEEE.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Recent trends like cloud computing show that service providers increasingly adopt to modern self-adaptive system architectures promising higher resource efficiency and lower operating costs. Blue Yonder is such a service provider in the field of predictive analytics, looking for novel approaches to automatically adapt their service infrastructure to changing customer workloads and service-level agreements. In this paper, we present a realistic case study in which we apply a holistic modelbased approach to build a self-adaptive system that automatically maintains performance requirements and resource efficiency in the heterogeneous resource environment of Blue Yonder. As modeling formalism, we employ the Descartes Modeling Language (DML), a domain-specific language for modeling the performance behavior and run-time adaptation processes of modern dynamic IT systems. This case study is an exemplar for applying a holistic model-based approach to engineering performance-aware system adaptation. More specifically, the results evaluate the applicability and effectiveness of our approach and demonstrate that DML provides suitable modeling abstractions that can be used as a basis for self-adaptive performance and resource management.

  @inproceedings{HuWaBaKo2015-ICCAC-BlueYonder, abstract = {Recent trends like cloud computing show that service providers increasingly adopt to modern self-adaptive system architectures promising higher resource efficiency and lower operating costs. Blue Yonder is such a service provider in the field of predictive analytics, looking for novel approaches to automatically adapt their service infrastructure to changing customer workloads and service-level agreements. In this paper, we present a realistic case study in which we apply a holistic modelbased approach to build a self-adaptive system that automatically maintains performance requirements and resource efficiency in the heterogeneous resource environment of Blue Yonder. As modeling formalism, we employ the Descartes Modeling Language (DML), a domain-specific language for modeling the performance behavior and run-time adaptation processes of modern dynamic IT systems. This case study is an exemplar for applying a holistic model-based approach to engineering performance-aware system adaptation. More specifically, the results evaluate the applicability and effectiveness of our approach and demonstrate that DML provides suitable modeling abstractions that can be used as a basis for self-adaptive performance and resource management.}, author = {Huber, Nikolaus and Walter, J{\"u}rgen and B{\"a}hr, Manuel and Kounev, Samuel}, booktitle = {Proceedings of the 2015 IEEE International Conference on Cloud and Autonomic Computing (ICCAC)}, month = {September}, publisher = {IEEE}, title = {{Model-based Autonomic and Performance-aware System Adaptation in Heterogeneous Resource Environments: A Case Study}}, year = 2015 }

  %0 Conference Paper %1 HuWaBaKo2015-ICCAC-BlueYonder %A Huber, Nikolaus %A Walter, J{\"u}rgen %A B{\"a}hr, Manuel %A Kounev, Samuel %B Proceedings of the 2015 IEEE International Conference on Cloud and Autonomic Computing (ICCAC) %D 2015 %I IEEE %T {Model-based Autonomic and Performance-aware System Adaptation in Heterogeneous Resource Environments: A Case Study} %X Recent trends like cloud computing show that service providers increasingly adopt to modern self-adaptive system architectures promising higher resource efficiency and lower operating costs. Blue Yonder is such a service provider in the field of predictive analytics, looking for novel approaches to automatically adapt their service infrastructure to changing customer workloads and service-level agreements. In this paper, we present a realistic case study in which we apply a holistic modelbased approach to build a self-adaptive system that automatically maintains performance requirements and resource efficiency in the heterogeneous resource environment of Blue Yonder. As modeling formalism, we employ the Descartes Modeling Language (DML), a domain-specific language for modeling the performance behavior and run-time adaptation processes of modern dynamic IT systems. This case study is an exemplar for applying a holistic model-based approach to engineering performance-aware system adaptation. More specifically, the results evaluate the applicability and effectiveness of our approach and demonstrate that DML provides suitable modeling abstractions that can be used as a basis for self-adaptive performance and resource management.
• 
  {Evaluation of Intrusion Detection Systems in Virtualized Environments Using Attack Injection}. Milenkoski, Aleksandar; Payne, Bryan D.; Antunes, Nuno; Vieira, Marco; Kounev, Samuel; Avritzer, Alberto; Luft, Matthias; in The 18th International Symposium on Research in Attacks, Intrusions, and Defenses (RAID 2015){Springer}.

  {Acceptance Rate: 23%.}
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  {The evaluation of intrusion detection systems (IDSes) is an active research area with many open challenges, one of which is the generation of representative workloads that contain attacks. In this paper, we propose a novel approach for the rigorous evaluation of IDSes in virtualized environments, with a focus on IDSes designed to detect attacks leveraging or targeting the hypervisor via its hypercall interface. We present hInjector, a tool for generating IDS evaluation workloads by injecting such attacks during regular operation of a virtualized environment. We demonstrate the application of our approach and show its practical usefulness by evaluating a representative IDS designed to operate in virtualized environments. The virtualized environment of the industry-standard benchmark SPECvirt_sc2013 is used as a testbed, whose drivers generate workloads representative of workloads seen in production environments. This work enables for the first time the injection of attacks in virtualized environments for the purpose of generating representative IDS evaluation workloads.}

  @inproceedings{MiPaAnViKoAvLu2015-RAID-Challenges, abstract = {{The evaluation of intrusion detection systems (IDSes) is an active research area with many open challenges, one of which is the generation of representative workloads that contain attacks. In this paper, we propose a novel approach for the rigorous evaluation of IDSes in virtualized environments, with a focus on IDSes designed to detect attacks leveraging or targeting the hypervisor via its hypercall interface. We present hInjector, a tool for generating IDS evaluation workloads by injecting such attacks during regular operation of a virtualized environment. We demonstrate the application of our approach and show its practical usefulness by evaluating a representative IDS designed to operate in virtualized environments. The virtualized environment of the industry-standard benchmark SPECvirt_sc2013 is used as a testbed, whose drivers generate workloads representative of workloads seen in production environments. This work enables for the first time the injection of attacks in virtualized environments for the purpose of generating representative IDS evaluation workloads.}}, author = {Milenkoski, Aleksandar and Payne, Bryan D. and Antunes, Nuno and Vieira, Marco and Kounev, Samuel and Avritzer, Alberto and Luft, Matthias}, booktitle = {The 18th International Symposium on Research in Attacks, Intrusions, and Defenses (RAID 2015)}, month = {{November}}, note = {{Acceptance Rate: 23%.}}, publisher = {{Springer}}, title = {{Evaluation of Intrusion Detection Systems in Virtualized Environments Using Attack Injection}}, year = 2015 }

  %0 Conference Paper %1 MiPaAnViKoAvLu2015-RAID-Challenges %A Milenkoski, Aleksandar %A Payne, Bryan D. %A Antunes, Nuno %A Vieira, Marco %A Kounev, Samuel %A Avritzer, Alberto %A Luft, Matthias %B The 18th International Symposium on Research in Attacks, Intrusions, and Defenses (RAID 2015) %D 2015 %I {Springer} %T {Evaluation of Intrusion Detection Systems in Virtualized Environments Using Attack Injection} %U http://link.springer.com/chapter/10.1007/978-3-319-26362-5_22 %X {The evaluation of intrusion detection systems (IDSes) is an active research area with many open challenges, one of which is the generation of representative workloads that contain attacks. In this paper, we propose a novel approach for the rigorous evaluation of IDSes in virtualized environments, with a focus on IDSes designed to detect attacks leveraging or targeting the hypervisor via its hypercall interface. We present hInjector, a tool for generating IDS evaluation workloads by injecting such attacks during regular operation of a virtualized environment. We demonstrate the application of our approach and show its practical usefulness by evaluating a representative IDS designed to operate in virtualized environments. The virtualized environment of the industry-standard benchmark SPECvirt_sc2013 is used as a testbed, whose drivers generate workloads representative of workloads seen in production environments. This work enables for the first time the injection of attacks in virtualized environments for the purpose of generating representative IDS evaluation workloads.}

• 
  {Platform-as-a-Service Architecture for Performance Isolated Multi-Tenant Applications}. Krebs, Rouven; Loesch, Manuel; Kounev, Samuel; in Proceedings of the 7th IEEE International Conference on Cloud Computing (IEEE Cloud 2014)IEEE.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  {Software-as-a-Service (SaaS) often shares one single application instance among different tenants to reduce costs. However, sharing potentially leads to undesired influence from one tenant onto the performance observed by the others. This is a significant problem as performance is one of the major obstacles for cloud customers. The application does intentionally not manage hardware resources, and the operating system is not aware of application level entities like tenants which makes the performance control a challenge. In case the SaaS is hosted on a Platform-as-a-Service (PaaS), the SaaS developer usually wants to control performance-related issues according to individual needs, and available information is even more limited. Thus, it is difficult to control the performance of different tenants to keep them isolated. Existing work focuses on concrete methods to provide performance isolation in systems where the whole stack is under control. In this paper we present a concrete PaaS enhancement which enables application developers to realize isolation methods for their hosted SaaS application. In a case study we evaluated the applicability and effectiveness of the enhancement in different environments.}

  @inproceedings{KrLoKo2014_CLOUD_Framework, abstract = {{Software-as-a-Service (SaaS) often shares one single application instance among different tenants to reduce costs. However, sharing potentially leads to undesired influence from one tenant onto the performance observed by the others. This is a significant problem as performance is one of the major obstacles for cloud customers. The application does intentionally not manage hardware resources, and the operating system is not aware of application level entities like tenants which makes the performance control a challenge. In case the SaaS is hosted on a Platform-as-a-Service (PaaS), the SaaS developer usually wants to control performance-related issues according to individual needs, and available information is even more limited. Thus, it is difficult to control the performance of different tenants to keep them isolated. Existing work focuses on concrete methods to provide performance isolation in systems where the whole stack is under control. In this paper we present a concrete PaaS enhancement which enables application developers to realize isolation methods for their hosted SaaS application. In a case study we evaluated the applicability and effectiveness of the enhancement in different environments.}}, author = {Krebs, Rouven and Loesch, Manuel and Kounev, Samuel}, booktitle = {Proceedings of the 7th IEEE International Conference on Cloud Computing (IEEE Cloud 2014)}, month = {July}, publisher = {IEEE}, title = {{Platform-as-a-Service Architecture for Performance Isolated Multi-Tenant Applications}}, year = 2014 }

  %0 Conference Paper %1 KrLoKo2014_CLOUD_Framework %A Krebs, Rouven %A Loesch, Manuel %A Kounev, Samuel %B Proceedings of the 7th IEEE International Conference on Cloud Computing (IEEE Cloud 2014) %D 2014 %I IEEE %T {Platform-as-a-Service Architecture for Performance Isolated Multi-Tenant Applications} %X {Software-as-a-Service (SaaS) often shares one single application instance among different tenants to reduce costs. However, sharing potentially leads to undesired influence from one tenant onto the performance observed by the others. This is a significant problem as performance is one of the major obstacles for cloud customers. The application does intentionally not manage hardware resources, and the operating system is not aware of application level entities like tenants which makes the performance control a challenge. In case the SaaS is hosted on a Platform-as-a-Service (PaaS), the SaaS developer usually wants to control performance-related issues according to individual needs, and available information is even more limited. Thus, it is difficult to control the performance of different tenants to keep them isolated. Existing work focuses on concrete methods to provide performance isolation in systems where the whole stack is under control. In this paper we present a concrete PaaS enhancement which enables application developers to realize isolation methods for their hosted SaaS application. In a case study we evaluated the applicability and effectiveness of the enhancement in different environments.}
• 
  {Experience Report: An Analysis of Hypercall Handler Vulnerabilities}. Milenkoski, Aleksandar; Payne, Bryan D.; Antunes, Nuno; Vieira, Marco; Kounev, Samuel; in {Proceedings of The 25th IEEE International Symposium on Software Reliability Engineering (ISSRE 2014) --- Research Track}IEEE Computer Society, {Washington DC, USA}.

  Acceptance Rate: 25\%, Best Paper Award Nomination
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  {Hypervisors are becoming increasingly ubiquitous with the growing proliferation of virtualized data centers. As a result, attackers are exploring vectors to attack hypervisors, against which an attack may be executed via several attack vectors such as device drivers, virtual machine exit events, or hypercalls. Hypercalls enable intrusions in hypervisors through their hypercall interfaces. Despite the importance, there is very limited publicly available information on vulnerabilities of hypercall handlers and attacks triggering them, which significantly hinders advances towards monitoring and securing these interfaces. In this paper, we characterize the hypercall attack surface based on analyzing a set of vulnerabilities of hypercall handlers. We systematize and discuss the errors that caused the considered vulnerabilities, and activities for executing attacks triggering them. We also demonstrate attacks triggering the considered vulnerabilities and analyze their effects. Finally, we suggest an action plan for improving the security of hypercall interfaces.}

  @inproceedings{MiPaAnViKo2014-ISSRE-AnAnalHypHanVulns, abstract = {{Hypervisors are becoming increasingly ubiquitous with the growing proliferation of virtualized data centers. As a result, attackers are exploring vectors to attack hypervisors, against which an attack may be executed via several attack vectors such as device drivers, virtual machine exit events, or hypercalls. Hypercalls enable intrusions in hypervisors through their hypercall interfaces. Despite the importance, there is very limited publicly available information on vulnerabilities of hypercall handlers and attacks triggering them, which significantly hinders advances towards monitoring and securing these interfaces. In this paper, we characterize the hypercall attack surface based on analyzing a set of vulnerabilities of hypercall handlers. We systematize and discuss the errors that caused the considered vulnerabilities, and activities for executing attacks triggering them. We also demonstrate attacks triggering the considered vulnerabilities and analyze their effects. Finally, we suggest an action plan for improving the security of hypercall interfaces.}}, address = {{Washington DC, USA}}, author = {Milenkoski, Aleksandar and Payne, Bryan D. and Antunes, Nuno and Vieira, Marco and Kounev, Samuel}, booktitle = {{Proceedings of The 25th IEEE International Symposium on Software Reliability Engineering (ISSRE 2014) --- Research Track}}, month = {November}, note = {Acceptance Rate: 25\%, <b>Best Paper Award Nomination</b>}, organization = {IEEE}, publisher = {IEEE Computer Society}, title = {{Experience Report: An Analysis of Hypercall Handler Vulnerabilities}}, year = 2014 }

  %0 Conference Paper %1 MiPaAnViKo2014-ISSRE-AnAnalHypHanVulns %A Milenkoski, Aleksandar %A Payne, Bryan D. %A Antunes, Nuno %A Vieira, Marco %A Kounev, Samuel %B {Proceedings of The 25th IEEE International Symposium on Software Reliability Engineering (ISSRE 2014) --- Research Track} %C {Washington DC, USA} %D 2014 %I IEEE Computer Society %T {Experience Report: An Analysis of Hypercall Handler Vulnerabilities} %U http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6982618 %X {Hypervisors are becoming increasingly ubiquitous with the growing proliferation of virtualized data centers. As a result, attackers are exploring vectors to attack hypervisors, against which an attack may be executed via several attack vectors such as device drivers, virtual machine exit events, or hypercalls. Hypercalls enable intrusions in hypervisors through their hypercall interfaces. Despite the importance, there is very limited publicly available information on vulnerabilities of hypercall handlers and attacks triggering them, which significantly hinders advances towards monitoring and securing these interfaces. In this paper, we characterize the hypercall attack surface based on analyzing a set of vulnerabilities of hypercall handlers. We systematize and discuss the errors that caused the considered vulnerabilities, and activities for executing attacks triggering them. We also demonstrate attacks triggering the considered vulnerabilities and analyze their effects. Finally, we suggest an action plan for improving the security of hypercall interfaces.}
• 
  {Runtime Vertical Scaling of Virtualized Applications via Online Model Estimation}. Spinner, Simon; Kounev, Samuel; Zhu, Xiaoyun; Lu, Lei; Uysal, Mustafa; Holler, Anne; Griffith, Rean; in Proceedings of the 2014 IEEE Eighth International Conference on Self-Adaptive and Self-Organizing Systems (SASO) 157–166. IEEE.

  Acceptance Rate (Full Papers): 26%
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  Applications in virtualized data centers are often subject to Service Level Objectives (SLOs) regarding their performance (e.g., latency or throughput). In order to fulfill these SLOs, it is necessary to allocate sufficient resources of different types (CPU, memory, I/O, etc.) to an application. However, the relationship between the application performance and the resource allocations is complex and depends on multiple factors including application architecture, system configuration, and workload demands. In this paper, we present a model-based approach to ensure that the application performance meets the user-defined SLO efficiently by runtime "vertical scaling" (i.e., adding or removing resources) of individual virtual machines (VMs) running the application. A layered performance model describing the relationship between the resource allocations and the observed application performance is automatically extracted and updated online using resource demand estimation techniques. Such a model is then used in a feedback controller to dynamically adapt the number of virtual CPUs of individual VMs. We have implemented the controller on top of the VMware vSphere platform and evaluated it in a case study using a real-world email and groupware server. The experimental results show that our approach allows the managed application to achieve SLO satisfaction in spite of workload demand variation while avoiding oscillations commonly observed with state-of-the-art threshold-based controllers.

  @inproceedings{SpKoZhLuUyHoGr2014-SASO, abstract = {Applications in virtualized data centers are often subject to Service Level Objectives (SLOs) regarding their performance (e.g., latency or throughput). In order to fulfill these SLOs, it is necessary to allocate sufficient resources of different types (CPU, memory, I/O, etc.) to an application. However, the relationship between the application performance and the resource allocations is complex and depends on multiple factors including application architecture, system configuration, and workload demands. In this paper, we present a model-based approach to ensure that the application performance meets the user-defined SLO efficiently by runtime "vertical scaling" (i.e., adding or removing resources) of individual virtual machines (VMs) running the application. A layered performance model describing the relationship between the resource allocations and the observed application performance is automatically extracted and updated online using resource demand estimation techniques. Such a model is then used in a feedback controller to dynamically adapt the number of virtual CPUs of individual VMs. We have implemented the controller on top of the VMware vSphere platform and evaluated it in a case study using a real-world email and groupware server. The experimental results show that our approach allows the managed application to achieve SLO satisfaction in spite of workload demand variation while avoiding oscillations commonly observed with state-of-the-art threshold-based controllers.}, author = {Spinner, Simon and Kounev, Samuel and Zhu, Xiaoyun and Lu, Lei and Uysal, Mustafa and Holler, Anne and Griffith, Rean}, booktitle = {Proceedings of the 2014 IEEE Eighth International Conference on Self-Adaptive and Self-Organizing Systems (SASO)}, month = {September}, note = {Acceptance Rate (Full Papers): 26%}, pages = {157--166}, publisher = {IEEE}, title = {{Runtime Vertical Scaling of Virtualized Applications via Online Model Estimation}}, year = 2014 }

  %0 Conference Paper %1 SpKoZhLuUyHoGr2014-SASO %A Spinner, Simon %A Kounev, Samuel %A Zhu, Xiaoyun %A Lu, Lei %A Uysal, Mustafa %A Holler, Anne %A Griffith, Rean %B Proceedings of the 2014 IEEE Eighth International Conference on Self-Adaptive and Self-Organizing Systems (SASO) %D 2014 %I IEEE %P 157--166 %T {Runtime Vertical Scaling of Virtualized Applications via Online Model Estimation} %X Applications in virtualized data centers are often subject to Service Level Objectives (SLOs) regarding their performance (e.g., latency or throughput). In order to fulfill these SLOs, it is necessary to allocate sufficient resources of different types (CPU, memory, I/O, etc.) to an application. However, the relationship between the application performance and the resource allocations is complex and depends on multiple factors including application architecture, system configuration, and workload demands. In this paper, we present a model-based approach to ensure that the application performance meets the user-defined SLO efficiently by runtime "vertical scaling" (i.e., adding or removing resources) of individual virtual machines (VMs) running the application. A layered performance model describing the relationship between the resource allocations and the observed application performance is automatically extracted and updated online using resource demand estimation techniques. Such a model is then used in a feedback controller to dynamically adapt the number of virtual CPUs of individual VMs. We have implemented the controller on top of the VMware vSphere platform and evaluated it in a case study using a real-world email and groupware server. The experimental results show that our approach allows the managed application to achieve SLO satisfaction in spite of workload demand variation while avoiding oscillations commonly observed with state-of-the-art threshold-based controllers.
• 
  {Resource Usage Control In Multi-Tenant Applications}. Krebs, Rouven; Spinner, Simon; Ahmed, Nadia; Kounev, Samuel; in Proceedings of the 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid 2014) 122–131. IEEE/ACM.

  Acceptance Rate: 19\% (54/283)
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  {Multi-tenancy is an approach to share one application instance among multiple customers by providing each of them a dedicated view. This approach is commonly used by SaaS providers to reduce the costs for service provisioning. Tenants also expect to be isolated in terms of the performance they observe and the providers inability to offer performance guarantees is a major obstacle for potential cloud customers. To guarantee an isolated performance it is essential to control the resources used by a tenant. This is a challenge, because the layers of the execution environment, responsible for controlling resource usage (e.g., operating system), normally do not have knowledge about entities defined at the application level and thus they cannot distinguish between different tenants. Furthermore, it is hard to predict how tenant requests propagate through the multiple layers of the execution environment down to the physical resource layer. The intended abstraction of the application from the resource controlling layers does not allow to solely solving this problem in the application. In this paper, we propose an approach which applies resource demand estimation techniques in combination with a request based admission control. The resource demand estimation is used to determine resource consumption information for individual requests. The admission control mechanism uses this knowledge to delay requests originating from tenants that exceed their allocated resource share. The proposed method is validated by a widely accepted benchmark showing its applicability in a setup motivated by today's platform environments.}

  @inproceedings{KrSpAhKo2014_CCGrid_ResourceIsolation, abstract = {{Multi-tenancy is an approach to share one application instance among multiple customers by providing each of them a dedicated view. This approach is commonly used by SaaS providers to reduce the costs for service provisioning. Tenants also expect to be isolated in terms of the performance they observe and the providers inability to offer performance guarantees is a major obstacle for potential cloud customers. To guarantee an isolated performance it is essential to control the resources used by a tenant. This is a challenge, because the layers of the execution environment, responsible for controlling resource usage (e.g., operating system), normally do not have knowledge about entities defined at the application level and thus they cannot distinguish between different tenants. Furthermore, it is hard to predict how tenant requests propagate through the multiple layers of the execution environment down to the physical resource layer. The intended abstraction of the application from the resource controlling layers does not allow to solely solving this problem in the application. In this paper, we propose an approach which applies resource demand estimation techniques in combination with a request based admission control. The resource demand estimation is used to determine resource consumption information for individual requests. The admission control mechanism uses this knowledge to delay requests originating from tenants that exceed their allocated resource share. The proposed method is validated by a widely accepted benchmark showing its applicability in a setup motivated by today's platform environments.}}, author = {Krebs, Rouven and Spinner, Simon and Ahmed, Nadia and Kounev, Samuel}, booktitle = {Proceedings of the 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid 2014)}, month = {may}, note = {Acceptance Rate: 19\% (54/283)}, pages = {122-131}, publisher = {IEEE/ACM}, title = {{Resource Usage Control In Multi-Tenant Applications}}, year = 2014 }

  %0 Conference Paper %1 KrSpAhKo2014_CCGrid_ResourceIsolation %A Krebs, Rouven %A Spinner, Simon %A Ahmed, Nadia %A Kounev, Samuel %B Proceedings of the 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid 2014) %D 2014 %I IEEE/ACM %P 122-131 %T {Resource Usage Control In Multi-Tenant Applications} %X {Multi-tenancy is an approach to share one application instance among multiple customers by providing each of them a dedicated view. This approach is commonly used by SaaS providers to reduce the costs for service provisioning. Tenants also expect to be isolated in terms of the performance they observe and the providers inability to offer performance guarantees is a major obstacle for potential cloud customers. To guarantee an isolated performance it is essential to control the resources used by a tenant. This is a challenge, because the layers of the execution environment, responsible for controlling resource usage (e.g., operating system), normally do not have knowledge about entities defined at the application level and thus they cannot distinguish between different tenants. Furthermore, it is hard to predict how tenant requests propagate through the multiple layers of the execution environment down to the physical resource layer. The intended abstraction of the application from the resource controlling layers does not allow to solely solving this problem in the application. In this paper, we propose an approach which applies resource demand estimation techniques in combination with a request based admission control. The resource demand estimation is used to determine resource consumption information for individual requests. The admission control mechanism uses this knowledge to delay requests originating from tenants that exceed their allocated resource share. The proposed method is validated by a widely accepted benchmark showing its applicability in a setup motivated by today's platform environments.}
• 
  {Performance Queries for Architecture-Level Performance Models}. Gorsler, Fabian; Brosig, Fabian; Kounev, Samuel; in Proceedings of the 5th ACM/SPEC International Conference on Performance Engineering (ICPE 2014)ACM, New York, NY, USA.

  Acceptance Rate (Full Paper): 14/56 = 25\%.
  • [ BibTeX ]
  • [ EndNote ]
  • [ Download ]
  @inproceedings{GoBrKo2014-PerformanceQueries, address = {New York, NY, USA}, author = {Gorsler, Fabian and Brosig, Fabian and Kounev, Samuel}, booktitle = {Proceedings of the 5th ACM/SPEC International Conference on Performance Engineering (ICPE 2014)}, month = {{March}}, note = {Acceptance Rate (Full Paper): 14/56 = 25\%.}, publisher = {ACM}, title = {{Performance Queries for Architecture-Level Performance Models}}, year = 2014 }

  %0 Conference Paper %1 GoBrKo2014-PerformanceQueries %A Gorsler, Fabian %A Brosig, Fabian %A Kounev, Samuel %B Proceedings of the 5th ACM/SPEC International Conference on Performance Engineering (ICPE 2014) %C New York, NY, USA %D 2014 %I ACM %T {Performance Queries for Architecture-Level Performance Models}
• 
  Enriching Software Architecture Models with Statistical Models for Performance Prediction in Modern Storage Environments. Noorshams, Qais; Reeb, Roland; Rentschler, Andreas; Kounev, Samuel; Reussner, Ralf; in Proceedings of the 17th International ACM Sigsoft Symposium on Component-based Software Engineering 45–54. ACM, New York, NY, USA.

  Acceptance Rate (Full Paper): 14/62 = 23\%.
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  @inproceedings{NoReReKoRe2014-CBSE-ArchitectureModelsWithStatisticalModels, address = {New York, NY, USA}, author = {Noorshams, Qais and Reeb, Roland and Rentschler, Andreas and Kounev, Samuel and Reussner, Ralf}, booktitle = {Proceedings of the 17th International ACM Sigsoft Symposium on Component-based Software Engineering}, month = {June}, note = {Acceptance Rate (Full Paper): 14/62 = 23\%.}, pages = {45--54}, publisher = {ACM}, series = {CBSE '14}, title = {Enriching Software Architecture Models with Statistical Models for Performance Prediction in Modern Storage Environments}, year = 2014 }

  %0 Conference Paper %1 NoReReKoRe2014-CBSE-ArchitectureModelsWithStatisticalModels %A Noorshams, Qais %A Reeb, Roland %A Rentschler, Andreas %A Kounev, Samuel %A Reussner, Ralf %B Proceedings of the 17th International ACM Sigsoft Symposium on Component-based Software Engineering %C New York, NY, USA %D 2014 %I ACM %P 45--54 %T Enriching Software Architecture Models with Statistical Models for Performance Prediction in Modern Storage Environments %U http://doi.acm.org/10.1145/2602458.2602475

• 
  {Predictive Performance Modeling of Virtualized Storage Systems using Optimized Statistical Regression Techniques}. Noorshams, Qais; Bruhn, Dominik; Kounev, Samuel; Reussner, Ralf; in Proceedings of the ACM/SPEC International Conference on Performance Engineering (ICPE 2013) 283–294. ACM, New York, NY, USA.
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  @inproceedings{NoBrKoRe2013-ICPE-IORegressionModels, address = {New York, NY, USA}, author = {Noorshams, Qais and Bruhn, Dominik and Kounev, Samuel and Reussner, Ralf}, booktitle = {Proceedings of the ACM/SPEC International Conference on Performance Engineering (ICPE 2013)}, pages = {283--294}, publisher = {ACM}, series = {ICPE'13}, title = {{Predictive Performance Modeling of Virtualized Storage Systems using Optimized Statistical Regression Techniques}}, year = 2013 }

  %0 Conference Paper %1 NoBrKoRe2013-ICPE-IORegressionModels %A Noorshams, Qais %A Bruhn, Dominik %A Kounev, Samuel %A Reussner, Ralf %B Proceedings of the ACM/SPEC International Conference on Performance Engineering (ICPE 2013) %C New York, NY, USA %D 2013 %I ACM %P 283--294 %T {Predictive Performance Modeling of Virtualized Storage Systems using Optimized Statistical Regression Techniques} %U http://doi.acm.org/10.1145/2479871.2479910
• 
  {I/O Performance Modeling of Virtualized Storage Systems}. Noorshams, Qais; Rostami, Kiana; Kounev, Samuel; T{u}ma, Petr; Reussner, Ralf; in Proceedings of the 21st IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS 2013) 121–130.

  Acceptance Rate (Full Paper): 44/163 = 27\%
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  @inproceedings{NoRoKoTuRe2013-MASCOTS-IOQNs, author = {Noorshams, Qais and Rostami, Kiana and Kounev, Samuel and T{u}ma, Petr and Reussner, Ralf}, booktitle = {Proceedings of the 21st IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS 2013)}, note = {Acceptance Rate (Full Paper): 44/163 = 27\%}, pages = {121-130}, series = {MASCOTS'13}, title = {{I/O Performance Modeling of Virtualized Storage Systems}}, year = 2013 }

  %0 Conference Paper %1 NoRoKoTuRe2013-MASCOTS-IOQNs %A Noorshams, Qais %A Rostami, Kiana %A Kounev, Samuel %A T{u}ma, Petr %A Reussner, Ralf %B Proceedings of the 21st IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS 2013) %D 2013 %P 121-130 %T {I/O Performance Modeling of Virtualized Storage Systems} %U http://dx.doi.org/10.1109/MASCOTS.2013.20
• 
  {Self-Adaptive Workload Classification and Forecasting for Proactive Resource Provisioning}. Herbst, Nikolas Roman; Huber, Nikolaus; Kounev, Samuel; Amrehn, Erich; in Proceedings of the 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013) 187–198. ACM, New York, NY, USA.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  {As modern enterprise software systems become increasingly dynamic, workload forecasting techniques are gaining in importance as a foundation for online capacity planning and resource management. Time series analysis covers a broad spectrum of methods to calculate workload forecasts based on history monitoring data. Related work in the field of workload forecasting mostly concentrates on evaluating specific methods and their individual optimisation potential or on predicting Quality-of-Service (QoS) metrics directly. As a basis, we present a survey on established forecasting methods of the time series analysis concerning their benefits and drawbacks and group them according to their computational overheads. In this paper, we propose a novel self-adaptive approach that selects suitable forecasting methods for a given context based on a decision tree and direct feedback cycles together with a corresponding implementation. The user needs to provide only his general forecasting objectives. In several experiments and case studies based on real world workload traces, we show that our implementation of the approach provides continuous and reliable forecast results at run-time. The results of this extensive evaluation show that the relative error of the individual forecast points is significantly reduced compared to statically applied forecasting methods, e.g. in an exemplary scenario on average by 37%. In a case study, between 55% and 75% of the violations of a given service level agreement can be prevented by applying proactive resource provisioning based on the forecast results of our implementation.}

  @inproceedings{HeHuKoAm2013-ICPE-WorkloadClassificationAndForecasting, abstract = {{As modern enterprise software systems become increasingly dynamic, workload forecasting techniques are gaining in importance as a foundation for online capacity planning and resource management. Time series analysis covers a broad spectrum of methods to calculate workload forecasts based on history monitoring data. Related work in the field of workload forecasting mostly concentrates on evaluating specific methods and their individual optimisation potential or on predicting Quality-of-Service (QoS) metrics directly. As a basis, we present a survey on established forecasting methods of the time series analysis concerning their benefits and drawbacks and group them according to their computational overheads. In this paper, we propose a novel self-adaptive approach that selects suitable forecasting methods for a given context based on a decision tree and direct feedback cycles together with a corresponding implementation. The user needs to provide only his general forecasting objectives. In several experiments and case studies based on real world workload traces, we show that our implementation of the approach provides continuous and reliable forecast results at run-time. The results of this extensive evaluation show that the relative error of the individual forecast points is significantly reduced compared to statically applied forecasting methods, e.g. in an exemplary scenario on average by 37%. In a case study, between 55% and 75% of the violations of a given service level agreement can be prevented by applying proactive resource provisioning based on the forecast results of our implementation.}}, address = {New York, NY, USA}, author = {Herbst, Nikolas Roman and Huber, Nikolaus and Kounev, Samuel and Amrehn, Erich}, booktitle = {Proceedings of the 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013)}, month = {April}, pages = {187--198}, publisher = {ACM}, title = {{Self-Adaptive Workload Classification and Forecasting for Proactive Resource Provisioning}}, year = 2013 }

  %0 Conference Paper %1 HeHuKoAm2013-ICPE-WorkloadClassificationAndForecasting %A Herbst, Nikolas Roman %A Huber, Nikolaus %A Kounev, Samuel %A Amrehn, Erich %B Proceedings of the 4th ACM/SPEC International Conference on Performance Engineering (ICPE 2013) %C New York, NY, USA %D 2013 %I ACM %P 187--198 %T {Self-Adaptive Workload Classification and Forecasting for Proactive Resource Provisioning} %U http://doi.acm.org/10.1145/2479871.2479899 %X {As modern enterprise software systems become increasingly dynamic, workload forecasting techniques are gaining in importance as a foundation for online capacity planning and resource management. Time series analysis covers a broad spectrum of methods to calculate workload forecasts based on history monitoring data. Related work in the field of workload forecasting mostly concentrates on evaluating specific methods and their individual optimisation potential or on predicting Quality-of-Service (QoS) metrics directly. As a basis, we present a survey on established forecasting methods of the time series analysis concerning their benefits and drawbacks and group them according to their computational overheads. In this paper, we propose a novel self-adaptive approach that selects suitable forecasting methods for a given context based on a decision tree and direct feedback cycles together with a corresponding implementation. The user needs to provide only his general forecasting objectives. In several experiments and case studies based on real world workload traces, we show that our implementation of the approach provides continuous and reliable forecast results at run-time. The results of this extensive evaluation show that the relative error of the individual forecast points is significantly reduced compared to statically applied forecasting methods, e.g. in an exemplary scenario on average by 37%. In a case study, between 55% and 75% of the violations of a given service level agreement can be prevented by applying proactive resource provisioning based on the forecast results of our implementation.}

• 
  {Modeling Parameter and Context Dependencies in Online Architecture-Level Performance Models}. Brosig, Fabian; Huber, Nikolaus; Kounev, Samuel; in Proceedings of the 15th ACM SIGSOFT International Symposium on Component Based Software Engineering (CBSE 2012), June 26--28, 2012, Bertinoro, Italy

  Acceptance Rate (Full Paper): 28.5\%.
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Modern enterprise applications have to satisfy increasingly stringent Quality-of-Service requirements. To ensure that a system meets its performance requirements, the ability to predict its performance under different configurations and workloads is essential. Architecture-level performance models describe performance-relevant aspects of software architectures and execution environments allowing to evaluate different usage profiles as well as system deployment and configuration options. However, building performance models manually requires a lot of time and effort. In this paper, we present a novel automated method for the extraction of architecture-level performance models of distributed component-based systems, based on monitoring data collected at run-time. The method is validated in a case study with the industry-standard SPECjEnterprise2010 Enterprise Java benchmark, a representative software system executed in a realistic environment. The obtained performance predictions match the measurements on the real system within an error margin of mostly 10-20 percent.

  @inproceedings{BrHuKo2012-CBSE-ParamAndContextDep, abstract = {Modern enterprise applications have to satisfy increasingly stringent Quality-of-Service requirements. To ensure that a system meets its performance requirements, the ability to predict its performance under different configurations and workloads is essential. Architecture-level performance models describe performance-relevant aspects of software architectures and execution environments allowing to evaluate different usage profiles as well as system deployment and configuration options. However, building performance models manually requires a lot of time and effort. In this paper, we present a novel automated method for the extraction of architecture-level performance models of distributed component-based systems, based on monitoring data collected at run-time. The method is validated in a case study with the industry-standard SPECjEnterprise2010 Enterprise Java benchmark, a representative software system executed in a realistic environment. The obtained performance predictions match the measurements on the real system within an error margin of mostly 10-20 percent.}, author = {Brosig, Fabian and Huber, Nikolaus and Kounev, Samuel}, booktitle = {Proceedings of the 15th ACM SIGSOFT International Symposium on Component Based Software Engineering (CBSE 2012), June 26--28, 2012, Bertinoro, Italy}, month = {June}, note = {Acceptance Rate (Full Paper): 28.5\%.}, title = {{Modeling Parameter and Context Dependencies in Online Architecture-Level Performance Models}}, year = 2012 }

  %0 Conference Paper %1 BrHuKo2012-CBSE-ParamAndContextDep %A Brosig, Fabian %A Huber, Nikolaus %A Kounev, Samuel %B Proceedings of the 15th ACM SIGSOFT International Symposium on Component Based Software Engineering (CBSE 2012), June 26--28, 2012, Bertinoro, Italy %D 2012 %T {Modeling Parameter and Context Dependencies in Online Architecture-Level Performance Models} %U http://cbse-conferences.org/2012/ %X Modern enterprise applications have to satisfy increasingly stringent Quality-of-Service requirements. To ensure that a system meets its performance requirements, the ability to predict its performance under different configurations and workloads is essential. Architecture-level performance models describe performance-relevant aspects of software architectures and execution environments allowing to evaluate different usage profiles as well as system deployment and configuration options. However, building performance models manually requires a lot of time and effort. In this paper, we present a novel automated method for the extraction of architecture-level performance models of distributed component-based systems, based on monitoring data collected at run-time. The method is validated in a case study with the industry-standard SPECjEnterprise2010 Enterprise Java benchmark, a representative software system executed in a realistic environment. The obtained performance predictions match the measurements on the real system within an error margin of mostly 10-20 percent.
• 
  {Modeling Dynamic Virtualized Resource Landscapes}. Huber, Nikolaus; Brosig, Fabian; Kounev, Samuel; in Proceedings of the 8th ACM SIGSOFT International Conference on the Quality of Software Architectures (QoSA 2012) 81–90. ACM, New York, NY, USA.

  Acceptance Rate (Full Paper): 25.6\%
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Modern data centers are subject to an increasing demand for flexibility. Increased flexibility and dynamics, however, also result in a higher system complexity. This complexity carries on to run-time resource management for Quality-of-Service (QoS) enforcement, rendering design-time approaches for QoS assurance inadequate. In this paper, we present a set of novel meta-models that can be used to describe the resource landscape, the architecture and resource layers of dynamic virtualized data center infrastructures, as well as their run-time adaptation and resource management aspects. With these meta-models we introduce new modeling concepts to improve model-based run-time QoS assurance. We evaluate our meta-models by modeling a representative virtualized service infrastructure and using these model instances for run-time resource allocation. The results demonstrate the benefits of the new meta-models and show how they can be used to improve model-based system adaptation and run-time resource management in dynamic virtualized data centers.

  @inproceedings{HuBrKo2012-QoSA-ModelingVirtResLandscapes, abstract = {Modern data centers are subject to an increasing demand for flexibility. Increased flexibility and dynamics, however, also result in a higher system complexity. This complexity carries on to run-time resource management for Quality-of-Service (QoS) enforcement, rendering design-time approaches for QoS assurance inadequate. In this paper, we present a set of novel meta-models that can be used to describe the resource landscape, the architecture and resource layers of dynamic virtualized data center infrastructures, as well as their run-time adaptation and resource management aspects. With these meta-models we introduce new modeling concepts to improve model-based run-time QoS assurance. We evaluate our meta-models by modeling a representative virtualized service infrastructure and using these model instances for run-time resource allocation. The results demonstrate the benefits of the new meta-models and show how they can be used to improve model-based system adaptation and run-time resource management in dynamic virtualized data centers.}, address = {New York, NY, USA}, author = {Huber, Nikolaus and Brosig, Fabian and Kounev, Samuel}, booktitle = {Proceedings of the 8th ACM SIGSOFT International Conference on the Quality of Software Architectures (QoSA 2012)}, month = {June}, note = {Acceptance Rate (Full Paper): 25.6\%}, pages = {81--90}, publisher = {ACM}, title = {{Modeling Dynamic Virtualized Resource Landscapes}}, year = 2012 }

  %0 Conference Paper %1 HuBrKo2012-QoSA-ModelingVirtResLandscapes %A Huber, Nikolaus %A Brosig, Fabian %A Kounev, Samuel %B Proceedings of the 8th ACM SIGSOFT International Conference on the Quality of Software Architectures (QoSA 2012) %C New York, NY, USA %D 2012 %I ACM %P 81--90 %T {Modeling Dynamic Virtualized Resource Landscapes} %U http://qosa.ipd.kit.edu/qosa_2012/ %X Modern data centers are subject to an increasing demand for flexibility. Increased flexibility and dynamics, however, also result in a higher system complexity. This complexity carries on to run-time resource management for Quality-of-Service (QoS) enforcement, rendering design-time approaches for QoS assurance inadequate. In this paper, we present a set of novel meta-models that can be used to describe the resource landscape, the architecture and resource layers of dynamic virtualized data center infrastructures, as well as their run-time adaptation and resource management aspects. With these meta-models we introduce new modeling concepts to improve model-based run-time QoS assurance. We evaluate our meta-models by modeling a representative virtualized service infrastructure and using these model instances for run-time resource allocation. The results demonstrate the benefits of the new meta-models and show how they can be used to improve model-based system adaptation and run-time resource management in dynamic virtualized data centers.
• 
  {M}etrics and {T}echniques for {Q}uantifying {P}erformance {I}solation in {C}loud {E}nvironments. Krebs, Rouven; Momm, Christof; Kounev, Samuel; in Proceedings of the 8th ACM SIGSOFT International Conference on the Quality of Software Architectures (QoSA 2012), B. Buhnova, A. Vallecillo (eds.) 91–100. ACM Press, New York, USA.

  Acceptance Rate (Full Paper): 25.6\%
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  @inproceedings{KrMoKo2012-QoSA-QuantifyingPerfIsoMetrics, address = {New York, USA}, author = {Krebs, Rouven and Momm, Christof and Kounev, Samuel}, booktitle = {Proceedings of the 8th ACM SIGSOFT International Conference on the Quality of Software Architectures (QoSA 2012)}, editor = {Buhnova, Barbora and Vallecillo, Antonio}, month = {June}, note = {Acceptance Rate (Full Paper): 25.6\%}, pages = {91--100}, publisher = {ACM Press}, title = {{M}etrics and {T}echniques for {Q}uantifying {P}erformance {I}solation in {C}loud {E}nvironments}, year = 2012 }

  %0 Conference Paper %1 KrMoKo2012-QoSA-QuantifyingPerfIsoMetrics %A Krebs, Rouven %A Momm, Christof %A Kounev, Samuel %B Proceedings of the 8th ACM SIGSOFT International Conference on the Quality of Software Architectures (QoSA 2012) %C New York, USA %D 2012 %E Buhnova, Barbora %E Vallecillo, Antonio %I ACM Press %P 91--100 %T {M}etrics and {T}echniques for {Q}uantifying {P}erformance {I}solation in {C}loud {E}nvironments %U http://qosa.ipd.kit.edu/qosa_2012/
• 
  {S/T/A: Meta-Modeling Run-Time Adaptation in Component-Based System Architectures}. Huber, Nikolaus; van Hoorn, Andr{\’e}; Koziolek, Anne; Brosig, Fabian; Kounev, Samuel; in Proceedings of the 9th IEEE International Conference on e-Business Engineering (ICEBE 2012) 70–77. IEEE Computer Society, Los Alamitos, CA, USA.

  Acceptance Rate (Full Paper): 19.7\% (26/132)
  • [ Abstract ]
  • [ BibTeX ]
  • [ EndNote ]
  • [ URL ]
  • [ Download ]
  Modern virtualized system environments usually host diverse applications of different parties and aim at utilizing resources efficiently while ensuring that quality-of-service requirements are continuously satisfied. In such scenarios, complex adaptations to changes in the system environment are still largely performed manually by humans. Over the past decade, autonomic self-adaptation techniques aiming to minimize human intervention have become increasingly popular. However, given that adaptation processes are usually highly system specific, it is a challenge to abstract from system details enabling the reuse of adaptation strategies. In this paper, we propose a novel modeling language (meta-model) providing means to describe system adaptation processes at the system architecture level in a generic, human-understandable and reusable way. We apply our approach to three different realistic contexts (dynamic resource allocation, software architecture optimization, and run-time adaptation planning) showing how the gap between complex manual adaptations and their autonomous execution can be closed by using a holistic model-based approach.

  @inproceedings{HuHoKoBrKo2012-ICEBE-STA, abstract = {Modern virtualized system environments usually host diverse applications of different parties and aim at utilizing resources efficiently while ensuring that quality-of-service requirements are continuously satisfied. In such scenarios, complex adaptations to changes in the system environment are still largely performed manually by humans. Over the past decade, autonomic self-adaptation techniques aiming to minimize human intervention have become increasingly popular. However, given that adaptation processes are usually highly system specific, it is a challenge to abstract from system details enabling the reuse of adaptation strategies. In this paper, we propose a novel modeling language (meta-model) providing means to describe system adaptation processes at the system architecture level in a generic, human-understandable and reusable way. We apply our approach to three different realistic contexts (dynamic resource allocation, software architecture optimization, and run-time adaptation planning) showing how the gap between complex manual adaptations and their autonomous execution can be closed by using a holistic model-based approach.}, address = {Los Alamitos, CA, USA}, author = {Huber, Nikolaus and van Hoorn, Andr{\'e} and Koziolek, Anne and Brosig, Fabian and Kounev, Samuel}, booktitle = {Proceedings of the 9th IEEE International Conference on e-Business Engineering (ICEBE 2012)}, month = {September}, note = {Acceptance Rate (Full Paper): 19.7\% (26/132)}, pages = {70--77}, publisher = {IEEE Computer Society}, title = {{S/T/A: Meta-Modeling Run-Time Adaptation in Component-Based System Architectures}}, year = 2012 }

  %0 Conference Paper %1 HuHoKoBrKo2012-ICEBE-STA %A Huber, Nikolaus %A van Hoorn, Andr{\'e} %A Koziolek, Anne %A Brosig, Fabian %A Kounev, Samuel %B Proceedings of the 9th IEEE International Conference on e-Business Engineering (ICEBE 2012) %C Los Alamitos, CA, USA %D 2012 %I IEEE Computer Society %P 70--77 %T {S/T/A: Meta-Modeling Run-Time Adaptation in Component-Based System Architectures} %U http://conferences.computer.org/icebe/2012/index.htm %X Modern virtualized system environments usually host diverse applications of different parties and aim at utilizing resources efficiently while ensuring that quality-of-service requirements are continuously satisfied. In such scenarios, complex adaptations to changes in the system environment are still largely performed manually by humans. Over the past decade, autonomic self-adaptation techniques aiming to minimize human intervention have become increasingly popular. However, given that adaptation processes are usually highly system specific, it is a challenge to abstract from system details enabling the reuse of adaptation strategies. In this paper, we propose a novel modeling language (meta-model) providing means to describe system adaptation processes at the system architecture level in a generic, human-understandable and reusable way. We apply our approach to three different realistic contexts (dynamic resource allocation, software architecture optimization, and run-time adaptation planning) showing how the gap between complex manual adaptations and their autonomous execution can be closed by using a holistic model-based approach.

• 
  Automated Simulation-Based Capacity Planning for Enterprise Data Fabrics. Kounev, Samuel; Bender, Konstantin; Brosig, Fabian; Huber, Nikolaus; Okamoto, Russell; in 4th International ICST Conference on Simulation Tools and Techniques (SIMUtools 2011), March 21--25, 2011, Barcelona, Spain

  Acceptance Rate (Full Paper): 29.8\% (23/77), ICST Best Paper Award