Simon Eismann

Sizeless: Predicting the Optimal Size of Serverless Functions. Eismann, Simon; Bui, Long; Grohmann, Johannes; Abad, Cristina; Herbst, Nikolas; Kounev, Samuel; in Proceedings of the 22nd International MIDDLEWARE Conference (2021).
[ Abstract ] [ BibTeX ] [ Download ]
 
Serverless functions are an emerging cloud computing paradigm that is being rapidly adopted by both industry and academia. In this cloud computing model, the provider opaquely handles resource management tasks such as resource provisioning, deployment, and auto-scaling. The only resource management task that developers are still in charge of is selecting how much resources are allocated to each worker instance. However, selecting the optimal size of serverless functions is quite challenging, so developers often neglect it despite its significant cost and performance benefits. Existing approaches aiming to automate serverless functions resource sizing require dedicated performance tests, which are time-consuming to implement and maintain. In this paper, we introduce an approach to predict the optimal resource size of a serverless function using monitoring data from a single resource size. As our approach does not require dedicated performance tests, it enables cloud providers to implement resource sizing on a platform level and automate the last resource management task associated with serverless functions. We evaluate our approach on four different serverless applications on AWS, where it predicts the execution time of the other memory sizes based on monitoring data for a single memory size with an average prediction error of 15.3%. Based on these predictions, it selects the optimal memory size for 79.0% of the serverless functions and the secondbest memory size for 12.3% of the serverless functions, which results in an average speedup of 39.7% while also decreasing average costs by 2.6%.
@inproceedings{eismann2021sizeless, abstract = {Serverless functions are an emerging cloud computing paradigm that is being rapidly adopted by both industry and academia. In this cloud computing model, the provider opaquely handles resource management tasks such as resource provisioning, deployment, and auto-scaling. The only resource management task that developers are still in charge of is selecting how much resources are allocated to each worker instance. However, selecting the optimal size of serverless functions is quite challenging, so developers often neglect it despite its significant cost and performance benefits. Existing approaches aiming to automate serverless functions resource sizing require dedicated performance tests, which are time-consuming to implement and maintain. In this paper, we introduce an approach to predict the optimal resource size of a serverless function using monitoring data from a single resource size. As our approach does not require dedicated performance tests, it enables cloud providers to implement resource sizing on a platform level and automate the last resource management task associated with serverless functions. We evaluate our approach on four different serverless applications on AWS, where it predicts the execution time of the other memory sizes based on monitoring data for a single memory size with an average prediction error of 15.3%. Based on these predictions, it selects the optimal memory size for 79.0% of the serverless functions and the secondbest memory size for 12.3% of the serverless functions, which results in an average speedup of 39.7% while also decreasing average costs by 2.6%.}, author = {Eismann, Simon and Bui, Long and Grohmann, Johannes and Abad, Cristina and Herbst, Nikolas and Kounev, Samuel}, booktitle = {Proceedings of the 22nd International MIDDLEWARE Conference}, keywords = {t_full myown descartes select}, title = {Sizeless: Predicting the Optimal Size of Serverless Functions}, year = 2021 }
The State of Serverless Applications: Collection, Characterization, and Community Consensus. Eismann, Simon; Scheuner, Joel; van Eyk, Erwin; Schwinger, Maximilian; Grohmann, Johannes; Herbst, Nikolas; Abad, Cristina; Iosup, Alexandru; in Transactions on Software Engineering (2021).
[ Abstract ] [ BibTeX ] [ URL ] [ Download ]
 
Over the last five years, all major cloud platform providers have increased their serverless offerings. Many early adopters report significant benefits for serverless-based over traditional applications, and many companies are considering moving to serverless themselves. However, currently there exist only few, scattered, and sometimes even conflicting reports on when serverless applications are well suited and what the best practices for their implementation are. We address this problem in the present study about the state of serverless applications. We collect descriptions of 89 serverless applications from open-source projects, academic literature, industrial literature, and domain-specific feedback. We analyze 16 characteristics that describe why and when successful adopters are using serverless applications, and how they are building them. We further compare the results of our characterization study to 10 existing, mostly industrial, studies and datasets; this allows us to identify points of consensus across multiple studies, investigate points of disagreement, and overall confirm the validity of our results. The results of this study can help managers to decide if they should adopt serverless technology, engineers to learn about current practices of building serverless applications, and researchers and platform providers to better understand the current landscape of serverless applications.
@article{eismann2021state, abstract = {Over the last five years, all major cloud platform providers have increased their serverless offerings. Many early adopters report significant benefits for serverless-based over traditional applications, and many companies are considering moving to serverless themselves. However, currently there exist only few, scattered, and sometimes even conflicting reports on when serverless applications are well suited and what the best practices for their implementation are. We address this problem in the present study about the state of serverless applications. We collect descriptions of 89 serverless applications from open-source projects, academic literature, industrial literature, and domain-specific feedback. We analyze 16 characteristics that describe why and when successful adopters are using serverless applications, and how they are building them. We further compare the results of our characterization study to 10 existing, mostly industrial, studies and datasets; this allows us to identify points of consensus across multiple studies, investigate points of disagreement, and overall confirm the validity of our results. The results of this study can help managers to decide if they should adopt serverless technology, engineers to learn about current practices of building serverless applications, and researchers and platform providers to better understand the current landscape of serverless applications.}, author = {Eismann, Simon and Scheuner, Joel and van Eyk, Erwin and Schwinger, Maximilian and Grohmann, Johannes and Herbst, Nikolas and Abad, Cristina and Iosup, Alexandru}, journal = {Transactions on Software Engineering}, keywords = {t_journalmagazine myown select descartes}, title = {The State of Serverless Applications: Collection, Characterization, and Community Consensus}, year = 2021 }
Serverless Applications:Why, When, and How?. Eismann, Simon; Joel, Scheuner; van Eyk, Erwin; Schwinger, Maximilian; Grohmann, Johannes; Herbst, Nikolas; Abad, Cristina; Iosup, Alexandru; in IEEE Software (2021). 38(1) 32–39.
[ Abstract ] [ BibTeX ] [ URL ] [ Download ]
 
Serverless computing shows good promise for efficiency and ease-of-use. Yet, there are only a few, scattered and sometimes conflicting reports on questions such as Why do so many companies adopt serverless?, When are serverless applications well suited?, and How are serverless applications currently implemented? To address these questions, we analyze 89 serverless applications from open-source projects, industrial sources, academic literature, and scientific computing—the most extensive study to date.
@article{eismann2020serverless, abstract = {Serverless computing shows good promise for efficiency and ease-of-use. Yet, there are only a few, scattered and sometimes conflicting reports on questions such as Why do so many companies adopt serverless?, When are serverless applications well suited?, and How are serverless applications currently implemented? To address these questions, we analyze 89 serverless applications from open-source projects, industrial sources, academic literature, and scientific computing—the most extensive study to date.}, author = {Eismann, Simon and Joel, Scheuner and van Eyk, Erwin and Schwinger, Maximilian and Grohmann, Johannes and Herbst, Nikolas and Abad, Cristina and Iosup, Alexandru}, journal = {IEEE Software}, keywords = {myown t_journalmagazine descartes select}, number = 1, pages = {32–39}, title = {Serverless Applications:Why, When, and How?}, volume = 38, year = 2021 }
Microservices: A Performance Tester’s Dream or Nightmare?. Eismann, Simon; Bezemer, Cor-Paul; Shang, Weiyi; Okanovic, Dusan; van Hoorn, Andre; in Proceedings of the 2020 ACM/SPEC International Conference on Performance Engineering (ICPE) (2020).
Acceptance Rate: 23.4% (15/64)
[ Abstract ] [ BibTeX ] [ URL ] [ Download ]
 
In recent years, there has been a shift in software development towards microservice-based architectures, which consist of small services that focus on one particular functionality. Many companies are migrating their applications to such architectures to reap the benefits of microservices, such as increased flexibility, scalability and a smaller granularity of the offered functionality by a service. On the one hand, the benefits of microservices for functional testing are often praised, as the focus on one functionality and their smaller granularity allow for more targeted and more convenient testing. On the other hand, using microservices has their consequences (both positive and negative) on other types of testing, such as performance testing. Performance testing is traditionally done by establishing the baseline performance of a software version, which is then used to compare the performance testing results of later software versions. However, as we show in this paper, establishing such a baseline performance is challenging in microservice applications. In this paper, we discuss the benefits and challenges of microservices from a performance testers point of view. Through a series of experiments on the TeaStore application, we demonstrate how microservices affect the performance testing process, and we demonstrate that it is not straightforward to achieve reliable performance testing results for a microservice application.
@inproceedings{EiBeShOkHo2020-ICPE-ServerlessWorkflows, abstract = {In recent years, there has been a shift in software development towards microservice-based architectures, which consist of small services that focus on one particular functionality. Many companies are migrating their applications to such architectures to reap the benefits of microservices, such as increased flexibility, scalability and a smaller granularity of the offered functionality by a service. On the one hand, the benefits of microservices for functional testing are often praised, as the focus on one functionality and their smaller granularity allow for more targeted and more convenient testing. On the other hand, using microservices has their consequences (both positive and negative) on other types of testing, such as performance testing. Performance testing is traditionally done by establishing the baseline performance of a software version, which is then used to compare the performance testing results of later software versions. However, as we show in this paper, establishing such a baseline performance is challenging in microservice applications. In this paper, we discuss the benefits and challenges of microservices from a performance testers point of view. Through a series of experiments on the TeaStore application, we demonstrate how microservices affect the performance testing process, and we demonstrate that it is not straightforward to achieve reliable performance testing results for a microservice application.}, author = {Eismann, Simon and Bezemer, Cor-Paul and Shang, Weiyi and Okanovic, Dusan and van Hoorn, Andre}, booktitle = {Proceedings of the 2020 ACM/SPEC International Conference on Performance Engineering (ICPE)}, keywords = {se2 t_full select myown descartes}, month = {April}, note = {{Acceptance Rate: 23.4% (15/64)}}, series = {ICPE'20}, title = {{Microservices: A Performance Tester's Dream or Nightmare?}}, year = 2020 }
Predicting the Costs of Serverless Workflows. Eismann, Simon; Grohmann, Johannes; van Eyk, Erwin; Herbst, Nikolas; Kounev, Samuel; in Proceedings of the 2020 ACM/SPEC International Conference on Performance Engineering (ICPE) (2020). 265–276.
Acceptance Rate: 23.4% (15/64)
[ Abstract ] [ BibTeX ] [ URL ] [ Download ]
 
Function-as-a-Service (FaaS) platforms enable users to run arbitrary functions without being concerned about operational issues, while only paying for the consumed resources. Individual functions are often composed into workflows for complex tasks. However, the pay-per-use model and nontransparent reporting by cloud providers make it challenging to estimate the expected cost of a workflow, which prevents informed business decisions. Existing cost-estimation approaches assume a static response time for the serverless functions, without taking input parameters into account. In this paper, we propose a methodology for the cost prediction of serverless workflows consisting of input-parameter sensitive function models and a monte-carlo simulation of an abstract workflow model. Our approach enables workflow designers to predict, compare, and optimize the expected costs and performance of a planned workflow, which currently requires time-intensive experimentation. In our evaluation, we show that our approach can predict the response time and output parameters of a function based on its input parameters with an accuracy of 96.1%. In a case study with two audio-processing workflows, our approach predicts the costs of the two workflows with an accuracy of 96.2%.
@inproceedings{EiGrEyHeKo2020-ICPE-ServerlessWorkflows, abstract = {Function-as-a-Service (FaaS) platforms enable users to run arbitrary functions without being concerned about operational issues, while only paying for the consumed resources. Individual functions are often composed into workflows for complex tasks. However, the pay-per-use model and nontransparent reporting by cloud providers make it challenging to estimate the expected cost of a workflow, which prevents informed business decisions. Existing cost-estimation approaches assume a static response time for the serverless functions, without taking input parameters into account. In this paper, we propose a methodology for the cost prediction of serverless workflows consisting of input-parameter sensitive function models and a monte-carlo simulation of an abstract workflow model. Our approach enables workflow designers to predict, compare, and optimize the expected costs and performance of a planned workflow, which currently requires time-intensive experimentation. In our evaluation, we show that our approach can predict the response time and output parameters of a function based on its input parameters with an accuracy of 96.1%. In a case study with two audio-processing workflows, our approach predicts the costs of the two workflows with an accuracy of 96.2%.}, address = {New York, NY, USA}, author = {Eismann, Simon and Grohmann, Johannes and van Eyk, Erwin and Herbst, Nikolas and Kounev, Samuel}, booktitle = {Proceedings of the 2020 ACM/SPEC International Conference on Performance Engineering (ICPE)}, keywords = {t_full se2 myown descartes select}, month = {April}, note = {{Acceptance Rate: 23.4% (15/64)}}, pages = {265–276}, publisher = {Association for Computing Machinery (ACM)}, series = {ICPE '20}, title = {Predicting the Costs of Serverless Workflows}, year = 2020 }

2020

Predicting the Impact of Function Size in Serverless Computing. Bui, Long; Thesis; University of Würzburg; Am Hubland, Informatikgebäude, 97074 Würzburg, Germany. (2020).
[ BibTeX ] [ EndNote ]
 
@mastersthesis{bui2020predicting, address = {Am Hubland, Informatikgeb{ä}ude, 97074 W{ü}rzburg, Germany}, author = {Bui, Long}, keywords = {Thesis_supervised_by_SE_member Thesis_supervised_by_Simon_Eismann se2_mastersthesis t_studentthesis}, school = {University of W{ü}rzburg}, title = {Predicting the Impact of Function Size in Serverless Computing}, year = 2020 }
%0 Thesis %1 bui2020predicting %A Bui, Long %C Am Hubland, Informatikgebäude, 97074 Würzburg, Germany %D 2020 %T Predicting the Impact of Function Size in Serverless Computing
Generating Statistical Response Time Models from Architectural Performance Models. Schaffarczyk, Kai; Thesis; University of Würzburg; Am Hubland, Informatikgebäude, 97074 Würzburg, Germany. (2020).
[ BibTeX ] [ EndNote ]
 
@mastersthesis{schaffarczyk2020generating, address = {Am Hubland, Informatikgeb{ä}ude, 97074 W{ü}rzburg, Germany}, author = {Schaffarczyk, Kai}, keywords = {Thesis_supervised_by_SE_member Thesis_supervised_by_Simon_Eismann se2_mastersthesis}, school = {University of W{ü}rzburg}, title = {Generating Statistical Response Time Models from Architectural Performance Models}, year = 2020 }
%0 Thesis %1 schaffarczyk2020generating %A Schaffarczyk, Kai %C Am Hubland, Informatikgebäude, 97074 Würzburg, Germany %D 2020 %T Generating Statistical Response Time Models from Architectural Performance Models
Generating Performance-Aware Web Applications from BPMN Models. Hümmer, Simon; Thesis; University of Würzburg; Am Hubland, Informatikgebäude, 97074 Würzburg, Germany. (2020).
[ BibTeX ] [ EndNote ]
 
@mastersthesis{hummer2020generating, address = {Am Hubland, Informatikgeb{ä}ude, 97074 W{ü}rzburg, Germany}, author = {Hümmer, Simon}, keywords = {Thesis_supervised_by_SE_member Thesis_supervised_by_Simon_Eismann se2_mastersthesis t_studentthesis}, school = {University of W{ü}rzburg}, title = {Generating Performance-Aware Web Applications from BPMN Models}, year = 2020 }
%0 Thesis %1 hummer2020generating %A Hümmer, Simon %C Am Hubland, Informatikgebäude, 97074 Würzburg, Germany %D 2020 %T Generating Performance-Aware Web Applications from BPMN Models

2019

Exploring the Limitations of Statistical Response Time Models in Architectural Performance Models. Trapp, Simon; Thesis; University of Würzburg; Am Hubland, Informatikgebäude, 97074 Würzburg, Germany. (2019, Dezember).
[ BibTeX ] [ EndNote ]
 
@mastersthesis{Trapp2019, address = {Am Hubland, Informatikgeb{ä}ude, 97074 W{ü}rzburg, Germany}, author = {Trapp, Simon}, keywords = {Thesis_supervised_by_Simon_Eismann descartes se2_mastersthesis}, month = {December}, school = {University of W{ü}rzburg}, title = {Exploring the {L}imitations of {S}tatistical {R}esponse {T}ime {M}odels in {A}rchitectural {P}erformance {M}odels}, type = {{Bachelor Thesis}}, year = 2019 }
%0 Thesis %1 Trapp2019 %A Trapp, Simon %C Am Hubland, Informatikgebäude, 97074 Würzburg, Germany %D 2019 %T Exploring the Limitations of Statistical Response Time Models in Architectural Performance Models
Erweiterung von Komponenten-basierten Performance Modellen für Micro-service Architekturen. Hick, Lars; Thesis; University of Würzburg; Am Hubland, Informatikgebäude, 97074 Würzburg, Germany. (2019, Februar).
[ BibTeX ] [ EndNote ]
 
@mastersthesis{Hick2019, address = {Am Hubland, Informatikgeb{ä}ude, 97074 W{ü}rzburg, Germany}, author = {Hick, Lars}, keywords = {Thesis_supervised_by_SE_member Thesis_supervised_by_Simon_Eismann descartes se2_mastersthesis}, month = {February}, school = {University of W{ü}rzburg}, title = {{Erweiterung von Komponenten-basierten Performance Modellen f{ü}r Micro-service Architekturen}}, type = {{Bachelor Thesis}}, year = 2019 }
%0 Thesis %1 Hick2019 %A Hick, Lars %C Am Hubland, Informatikgebäude, 97074 Würzburg, Germany %D 2019 %T Erweiterung von Komponenten-basierten Performance Modellen für Micro-service Architekturen

2018

Blackbox Learning of Parametric Dependencies for Performance Models from Monitoring Data. Ackermann, Vanessa; Thesis; University of Würzburg; Am Hubland, Informatikgebäude, 97074 Würzburg, Germany. (2018, März).
[ BibTeX ] [ EndNote ]
 
@mastersthesis{Ackermann2018, address = {Am Hubland, Informatikgeb{ä}ude, 97074 W{ü}rzburg, Germany}, author = {Ackermann, Vanessa}, keywords = {Thesis_supervised_by_Johannes_Grohmann Thesis_supervised_by_SE_member Thesis_supervised_by_Simon_Eismann descartes se2_mastersthesis t_studentthesis}, month = {March}, school = {University of W{ü}rzburg}, title = {{Blackbox Learning of Parametric Dependencies for Performance Models from Monitoring Data}}, type = {{Bachelor Thesis}}, year = 2018 }
%0 Thesis %1 Ackermann2018 %A Ackermann, Vanessa %C Am Hubland, Informatikgebäude, 97074 Würzburg, Germany %D 2018 %T Blackbox Learning of Parametric Dependencies for Performance Models from Monitoring Data
Automatic Identification of Parametric Dependencies for Performance Models. Elflein, Sven; Thesis; University of Würzburg; Am Hubland, Informatikgebäude, 97074 Würzburg, Germany. (2018, September).
[ BibTeX ] [ EndNote ]
 
@mastersthesis{Elflein2018, address = {Am Hubland, Informatikgeb{ä}ude, 97074 W{ü}rzburg, Germany}, author = {Elflein, Sven}, keywords = {Thesis_supervised_by_Johannes_Grohmann Thesis_supervised_by_SE_member Thesis_supervised_by_Simon_Eismann descartes se2_mastersthesis t_studentthesis}, month = {September}, note = {<b>W{ü}rzburg Software Engineering Award sponsored by Bosch Rexroth</b>}, school = {University of W{ü}rzburg}, title = {{Automatic Identification of Parametric Dependencies for Performance Models}}, type = {{Bachelor Thesis}}, year = 2018 }
%0 Thesis %1 Elflein2018 %A Elflein, Sven %C Am Hubland, Informatikgebäude, 97074 Würzburg, Germany %D 2018 %T Automatic Identification of Parametric Dependencies for Performance Models