Simon Eismann

Chair of Software Engineering (Informatik II)
Department of Computer Science
University of Würzburg

Am Hubland, 97074 Würzburg

Informatikgebäude, 1.OG, Room A107/8

Phone: +49 (931) 31 89655

simon.eismann@uni-wuerzburg.de
Google Scholar Profile
Research Gate Profile
List of all publications

Research Interests

Performance Modeling and Prediction of Software Systems
Performance and Regression Testing of Cloud-native Applications
Design and Evaluation of Microservice Architectures
Analysis and Optimization of Serverless Functions
Empirical Analysis of Serverless Application Architectures

Functions and Memberships

Secretary of the SPEC Research Group "DevOps Performance"
Co-Organizer of the HotCloudPerf 2022
Artifact Evaluation Chair of the ICPE 2022
Artifact Evaluation Committee Member of the ICPE 2021
Artifact Evaluation Committee Member of the ICPE 2020
Artifact Evaluation Committee Member of the OSDI 2020
PC Member of the PESARO 2019
Local Organizer of the SSP 2019

Awards

SPECtacular Award for dedicated service in the SPEC research groups, April 2022
Best Student Paper Award at MIDDLEWARE2021, Québec City, December 2021
ASQF Förderpreis for an outstanding master thesis, Würzburg, July 2018

Selected Publications

The State of Serverless Applications: Collection, Characterization, and Community Consensus. Eismann, Simon; Scheuner, Joel; van Eyk, Erwin; Schwinger, Maximilian; Grohmann, Johannes; Herbst, Nikolas; Abad, Cristina; Iosup, Alexandru; in Transactions on Software Engineering (2022). 48(10) 4152–4166.
- [ Abstract ]
- [ BibTeX ]
- [ URL ]
- [ DOI ]
- [ Download ]
Over the last five years, all major cloud platform providers have increased their serverless offerings. Many early adopters report significant benefits for serverless-based over traditional applications, and many companies are considering moving to serverless themselves. However, currently there exist only few, scattered, and sometimes even conflicting reports on when serverless applications are well suited and what the best practices for their implementation are. We address this problem in the present study about the state of serverless applications. We collect descriptions of 89 serverless applications from open-source projects, academic literature, industrial literature, and domain-specific feedback. We analyze 16 characteristics that describe why and when successful adopters are using serverless applications, and how they are building them. We further compare the results of our characterization study to 10 existing, mostly industrial, studies and datasets; this allows us to identify points of consensus across multiple studies, investigate points of disagreement, and overall confirm the validity of our results. The results of this study can help managers to decide if they should adopt serverless technology, engineers to learn about current practices of building serverless applications, and researchers and platform providers to better understand the current landscape of serverless applications.

@article{eismann2021state, abstract = {Over the last five years, all major cloud platform providers have increased their serverless offerings. Many early adopters report significant benefits for serverless-based over traditional applications, and many companies are considering moving to serverless themselves. However, currently there exist only few, scattered, and sometimes even conflicting reports on when serverless applications are well suited and what the best practices for their implementation are. We address this problem in the present study about the state of serverless applications. We collect descriptions of 89 serverless applications from open-source projects, academic literature, industrial literature, and domain-specific feedback. We analyze 16 characteristics that describe why and when successful adopters are using serverless applications, and how they are building them. We further compare the results of our characterization study to 10 existing, mostly industrial, studies and datasets; this allows us to identify points of consensus across multiple studies, investigate points of disagreement, and overall confirm the validity of our results. The results of this study can help managers to decide if they should adopt serverless technology, engineers to learn about current practices of building serverless applications, and researchers and platform providers to better understand the current landscape of serverless applications.}, author = {Eismann, Simon and Scheuner, Joel and van Eyk, Erwin and Schwinger, Maximilian and Grohmann, Johannes and Herbst, Nikolas and Abad, Cristina and Iosup, Alexandru}, journal = {Transactions on Software Engineering}, keywords = {t_journalmagazine}, number = 10, pages = {4152- 4166}, title = {The State of Serverless Applications: Collection, Characterization, and Community Consensus}, volume = 48, year = 2022 }
Serverless Applications:Why, When, and How?. Eismann, Simon; Joel, Scheuner; van Eyk, Erwin; Schwinger, Maximilian; Grohmann, Johannes; Herbst, Nikolas; Abad, Cristina; Iosup, Alexandru; in IEEE Software (2021). 38(1) 32–39.
- [ Abstract ]
- [ BibTeX ]
- [ URL ]
- [ DOI ]
- [ Download ]
Serverless computing shows good promise for efficiency and ease-of-use. Yet, there are only a few, scattered and sometimes conflicting reports on questions such as Why do so many companies adopt serverless?, When are serverless applications well suited?, and How are serverless applications currently implemented? To address these questions, we analyze 89 serverless applications from open-source projects, industrial sources, academic literature, and scientific computing—the most extensive study to date.

@article{eismann2020serverless, abstract = {Serverless computing shows good promise for efficiency and ease-of-use. Yet, there are only a few, scattered and sometimes conflicting reports on questions such as Why do so many companies adopt serverless?, When are serverless applications well suited?, and How are serverless applications currently implemented? To address these questions, we analyze 89 serverless applications from open-source projects, industrial sources, academic literature, and scientific computing—the most extensive study to date.}, author = {Eismann, Simon and Joel, Scheuner and van Eyk, Erwin and Schwinger, Maximilian and Grohmann, Johannes and Herbst, Nikolas and Abad, Cristina and Iosup, Alexandru}, journal = {IEEE Software}, keywords = {t_journalmagazine}, number = 1, pages = {32–39}, title = {Serverless Applications:Why, When, and How?}, volume = 38, year = 2021 }
Sizeless: Predicting the Optimal Size of Serverless Functions. Eismann, Simon; Bui, Long; Grohmann, Johannes; Abad, Cristina; Herbst, Nikolas; Kounev, Samuel; in Proceedings of the 22nd International MIDDLEWARE Conference (2021). 248–259.

Best Student Paper Award, ACM Artifacts Evaluated — Functional
- [ Abstract ]
- [ BibTeX ]
- [ URL ]
- [ DOI ]
- [ Download ]
Serverless functions are an emerging cloud computing paradigm that is being rapidly adopted by both industry and academia. In this cloud computing model, the provider opaquely handles resource management tasks such as resource provisioning, deployment, and auto-scaling. The only resource management task that developers are still in charge of is selecting how much resources are allocated to each worker instance. However, selecting the optimal size of serverless functions is quite challenging, so developers often neglect it despite its significant cost and performance benefits. Existing approaches aiming to automate serverless functions resource sizing require dedicated performance tests, which are time-consuming to implement and maintain. In this paper, we introduce an approach to predict the optimal resource size of a serverless function using monitoring data from a single resource size. As our approach does not require dedicated performance tests, it enables cloud providers to implement resource sizing on a platform level and automate the last resource management task associated with serverless functions. We evaluate our approach on four different serverless applications on AWS, where it predicts the execution time of the other memory sizes based on monitoring data for a single memory size with an average prediction error of 15.3%. Based on these predictions, it selects the optimal memory size for 79.0% of the serverless functions and the secondbest memory size for 12.3% of the serverless functions, which results in an average speedup of 39.7% while also decreasing average costs by 2.6%.

@inproceedings{eismann2021sizeless, abstract = {Serverless functions are an emerging cloud computing paradigm that is being rapidly adopted by both industry and academia. In this cloud computing model, the provider opaquely handles resource management tasks such as resource provisioning, deployment, and auto-scaling. The only resource management task that developers are still in charge of is selecting how much resources are allocated to each worker instance. However, selecting the optimal size of serverless functions is quite challenging, so developers often neglect it despite its significant cost and performance benefits. Existing approaches aiming to automate serverless functions resource sizing require dedicated performance tests, which are time-consuming to implement and maintain. In this paper, we introduce an approach to predict the optimal resource size of a serverless function using monitoring data from a single resource size. As our approach does not require dedicated performance tests, it enables cloud providers to implement resource sizing on a platform level and automate the last resource management task associated with serverless functions. We evaluate our approach on four different serverless applications on AWS, where it predicts the execution time of the other memory sizes based on monitoring data for a single memory size with an average prediction error of 15.3%. Based on these predictions, it selects the optimal memory size for 79.0% of the serverless functions and the secondbest memory size for 12.3% of the serverless functions, which results in an average speedup of 39.7% while also decreasing average costs by 2.6%.}, author = {Eismann, Simon and Bui, Long and Grohmann, Johannes and Abad, Cristina and Herbst, Nikolas and Kounev, Samuel}, booktitle = {Proceedings of the 22nd International MIDDLEWARE Conference}, keywords = {descartes}, note = {Best Student Paper Award, ACM Artifacts Evaluated — Functional}, pages = {248–259}, title = {Sizeless: Predicting the Optimal Size of Serverless Functions}, year = 2021 }
Predicting the Costs of Serverless Workflows. Eismann, Simon; Grohmann, Johannes; van Eyk, Erwin; Herbst, Nikolas; Kounev, Samuel; in Proceedings of the 2020 ACM/SPEC International Conference on Performance Engineering (ICPE) (2020). 265–276.

{Acceptance Rate: 23.4% (15/64)}
- [ Abstract ]
- [ BibTeX ]
- [ URL ]
- [ DOI ]
- [ Download ]
Function-as-a-Service (FaaS) platforms enable users to run arbitrary functions without being concerned about operational issues, while only paying for the consumed resources. Individual functions are often composed into workflows for complex tasks. However, the pay-per-use model and nontransparent reporting by cloud providers make it challenging to estimate the expected cost of a workflow, which prevents informed business decisions. Existing cost-estimation approaches assume a static response time for the serverless functions, without taking input parameters into account. In this paper, we propose a methodology for the cost prediction of serverless workflows consisting of input-parameter sensitive function models and a monte-carlo simulation of an abstract workflow model. Our approach enables workflow designers to predict, compare, and optimize the expected costs and performance of a planned workflow, which currently requires time-intensive experimentation. In our evaluation, we show that our approach can predict the response time and output parameters of a function based on its input parameters with an accuracy of 96.1%. In a case study with two audio-processing workflows, our approach predicts the costs of the two workflows with an accuracy of 96.2%.

@inproceedings{EiGrEyHeKo2020-ICPE-ServerlessWorkflows, abstract = {Function-as-a-Service (FaaS) platforms enable users to run arbitrary functions without being concerned about operational issues, while only paying for the consumed resources. Individual functions are often composed into workflows for complex tasks. However, the pay-per-use model and nontransparent reporting by cloud providers make it challenging to estimate the expected cost of a workflow, which prevents informed business decisions. Existing cost-estimation approaches assume a static response time for the serverless functions, without taking input parameters into account. In this paper, we propose a methodology for the cost prediction of serverless workflows consisting of input-parameter sensitive function models and a monte-carlo simulation of an abstract workflow model. Our approach enables workflow designers to predict, compare, and optimize the expected costs and performance of a planned workflow, which currently requires time-intensive experimentation. In our evaluation, we show that our approach can predict the response time and output parameters of a function based on its input parameters with an accuracy of 96.1%. In a case study with two audio-processing workflows, our approach predicts the costs of the two workflows with an accuracy of 96.2%.}, address = {New York, NY, USA}, author = {Eismann, Simon and Grohmann, Johannes and van Eyk, Erwin and Herbst, Nikolas and Kounev, Samuel}, booktitle = {Proceedings of the 2020 ACM/SPEC International Conference on Performance Engineering (ICPE)}, keywords = {se2}, month = {04}, note = {{Acceptance Rate: 23.4% (15/64)}}, pages = {265–276}, publisher = {Association for Computing Machinery (ACM)}, series = {ICPE '20}, title = {Predicting the Costs of Serverless Workflows}, year = 2020 }
{Microservices: A Performance Tester’s Dream or Nightmare?}. Eismann, Simon; Bezemer, Cor-Paul; Shang, Weiyi; Okanovic, Dusan; van Hoorn, Andre; in Proceedings of the 2020 ACM/SPEC International Conference on Performance Engineering (ICPE) (2020).

{Acceptance Rate: 23.4% (15/64), ACM Artifacts Evaluated — Functional}
- [ Abstract ]
- [ BibTeX ]
- [ URL ]
- [ Download ]
In recent years, there has been a shift in software development towards microservice-based architectures, which consist of small services that focus on one particular functionality. Many companies are migrating their applications to such architectures to reap the benefits of microservices, such as increased flexibility, scalability and a smaller granularity of the offered functionality by a service. On the one hand, the benefits of microservices for functional testing are often praised, as the focus on one functionality and their smaller granularity allow for more targeted and more convenient testing. On the other hand, using microservices has their consequences (both positive and negative) on other types of testing, such as performance testing. Performance testing is traditionally done by establishing the baseline performance of a software version, which is then used to compare the performance testing results of later software versions. However, as we show in this paper, establishing such a baseline performance is challenging in microservice applications. In this paper, we discuss the benefits and challenges of microservices from a performance testers point of view. Through a series of experiments on the TeaStore application, we demonstrate how microservices affect the performance testing process, and we demonstrate that it is not straightforward to achieve reliable performance testing results for a microservice application.

@inproceedings{EiBeShOkHo2020-ICPE-ServerlessWorkflows, abstract = {In recent years, there has been a shift in software development towards microservice-based architectures, which consist of small services that focus on one particular functionality. Many companies are migrating their applications to such architectures to reap the benefits of microservices, such as increased flexibility, scalability and a smaller granularity of the offered functionality by a service. On the one hand, the benefits of microservices for functional testing are often praised, as the focus on one functionality and their smaller granularity allow for more targeted and more convenient testing. On the other hand, using microservices has their consequences (both positive and negative) on other types of testing, such as performance testing. Performance testing is traditionally done by establishing the baseline performance of a software version, which is then used to compare the performance testing results of later software versions. However, as we show in this paper, establishing such a baseline performance is challenging in microservice applications. In this paper, we discuss the benefits and challenges of microservices from a performance testers point of view. Through a series of experiments on the TeaStore application, we demonstrate how microservices affect the performance testing process, and we demonstrate that it is not straightforward to achieve reliable performance testing results for a microservice application.}, author = {Eismann, Simon and Bezemer, Cor-Paul and Shang, Weiyi and Okanovic, Dusan and van Hoorn, Andre}, booktitle = {Proceedings of the 2020 ACM/SPEC International Conference on Performance Engineering (ICPE)}, keywords = {se2}, month = {04}, note = {{Acceptance Rate: 23.4% (15/64), ACM Artifacts Evaluated — Functional}}, series = {ICPE'20}, title = {{Microservices: A Performance Tester's Dream or Nightmare?}}, year = 2020 }

All Publications Link

Teaching

Exercise coordination for the master lecture "Software Architektur"
Supervision of students during the "Seminar Software Engineering"
Supervision of students during the "Software Praktikum"

Supervised Theses

2020[ to top ]

Predicting the Impact of Function Size in Serverless Computing. Bui, Long; Thesis; University of W{ü}rzburg; Am Hubland, Informatikgeb{ä}ude, 97074 W{ü}rzburg, Germany. (2020).
- [ BibTeX ]
- [ EndNote ]
@mastersthesis{bui2020predicting, address = {Am Hubland, Informatikgeb{\"a}ude, 97074 W{\"u}rzburg, Germany}, author = {Bui, Long}, keywords = {Thesis_supervised_by_Simon_Eismann}, school = {University of W{\"u}rzburg}, title = {Predicting the Impact of Function Size in Serverless Computing}, year = 2020 }

%0 Thesis %1 bui2020predicting %A Bui, Long %C Am Hubland, Informatikgeb{ä}ude, 97074 W{ü}rzburg, Germany %D 2020 %T Predicting the Impact of Function Size in Serverless Computing
Generating Statistical Response Time Models from Architectural Performance Models. Schaffarczyk, Kai; Thesis; University of W{ü}rzburg; Am Hubland, Informatikgeb{ä}ude, 97074 W{ü}rzburg, Germany. (2020).
- [ BibTeX ]
- [ EndNote ]
@mastersthesis{schaffarczyk2020generating, address = {Am Hubland, Informatikgeb{\"a}ude, 97074 W{\"u}rzburg, Germany}, author = {Schaffarczyk, Kai}, keywords = {Thesis_supervised_by_Simon_Eismann}, school = {University of W{\"u}rzburg}, title = {Generating Statistical Response Time Models from Architectural Performance Models}, year = 2020 }

%0 Thesis %1 schaffarczyk2020generating %A Schaffarczyk, Kai %C Am Hubland, Informatikgeb{ä}ude, 97074 W{ü}rzburg, Germany %D 2020 %T Generating Statistical Response Time Models from Architectural Performance Models
Generating Performance-Aware Web Applications from BPMN Models. Hümmer, Simon; Thesis; University of W{ü}rzburg; Am Hubland, Informatikgeb{ä}ude, 97074 W{ü}rzburg, Germany. (2020).
- [ BibTeX ]
- [ EndNote ]
@mastersthesis{hummer2020generating, address = {Am Hubland, Informatikgeb{\"a}ude, 97074 W{\"u}rzburg, Germany}, author = {Hümmer, Simon}, keywords = {Thesis_supervised_by_Simon_Eismann}, school = {University of W{\"u}rzburg}, title = {Generating Performance-Aware Web Applications from BPMN Models}, year = 2020 }

%0 Thesis %1 hummer2020generating %A Hümmer, Simon %C Am Hubland, Informatikgeb{ä}ude, 97074 W{ü}rzburg, Germany %D 2020 %T Generating Performance-Aware Web Applications from BPMN Models

2019[ to top ]

Exploring the {L}imitations of {S}tatistical {R}esponse {T}ime {M}odels in {A}rchitectural {P}erformance {M}odels. Trapp, Simon; Thesis; University of W{ü}rzburg; Am Hubland, Informatikgeb{ä}ude, 97074 W{ü}rzburg, Germany. (2019, Dezember).
- [ BibTeX ]
- [ EndNote ]
@mastersthesis{Trapp2019, address = {Am Hubland, Informatikgeb{\"a}ude, 97074 W{\"u}rzburg, Germany}, author = {Trapp, Simon}, keywords = {descartes}, month = 12, school = {University of W{\"u}rzburg}, title = {Exploring the {L}imitations of {S}tatistical {R}esponse {T}ime {M}odels in {A}rchitectural {P}erformance {M}odels}, type = {{Bachelor Thesis}}, year = 2019 }

%0 Thesis %1 Trapp2019 %A Trapp, Simon %C Am Hubland, Informatikgeb{ä}ude, 97074 W{ü}rzburg, Germany %D 2019 %T Exploring the {L}imitations of {S}tatistical {R}esponse {T}ime {M}odels in {A}rchitectural {P}erformance {M}odels
{Erweiterung von Komponenten-basierten Performance Modellen f{ü}r Micro-service Architekturen}. Hick, Lars; Thesis; University of W{ü}rzburg; Am Hubland, Informatikgeb{ä}ude, 97074 W{ü}rzburg, Germany. (2019, Februar).
- [ BibTeX ]
- [ EndNote ]
@mastersthesis{Hick2019, address = {Am Hubland, Informatikgeb{\"a}ude, 97074 W{\"u}rzburg, Germany}, author = {Hick, Lars}, keywords = {descartes}, month = {02}, school = {University of W{\"u}rzburg}, title = {{Erweiterung von Komponenten-basierten Performance Modellen f{\"u}r Micro-service Architekturen}}, type = {{Bachelor Thesis}}, year = 2019 }

%0 Thesis %1 Hick2019 %A Hick, Lars %C Am Hubland, Informatikgeb{ä}ude, 97074 W{ü}rzburg, Germany %D 2019 %T {Erweiterung von Komponenten-basierten Performance Modellen f{ü}r Micro-service Architekturen}

2018[ to top ]

{Blackbox Learning of Parametric Dependencies for Performance Models from Monitoring Data}. Ackermann, Vanessa; Thesis; University of W{ü}rzburg; Am Hubland, Informatikgeb{ä}ude, 97074 W{ü}rzburg, Germany. (2018, März).
- [ BibTeX ]
- [ EndNote ]
@mastersthesis{Ackermann2018, address = {Am Hubland, Informatikgeb{\"a}ude, 97074 W{\"u}rzburg, Germany}, author = {Ackermann, Vanessa}, keywords = {descartes}, month = {03}, school = {University of W{\"u}rzburg}, title = {{Blackbox Learning of Parametric Dependencies for Performance Models from Monitoring Data}}, type = {{Bachelor Thesis}}, year = 2018 }

%0 Thesis %1 Ackermann2018 %A Ackermann, Vanessa %C Am Hubland, Informatikgeb{ä}ude, 97074 W{ü}rzburg, Germany %D 2018 %T {Blackbox Learning of Parametric Dependencies for Performance Models from Monitoring Data}
{Automatic Identification of Parametric Dependencies for Performance Models}. Elflein, Sven; Thesis; University of W{ü}rzburg; Am Hubland, Informatikgeb{ä}ude, 97074 W{ü}rzburg, Germany. (2018, September).
- [ BibTeX ]
- [ EndNote ]
@mastersthesis{Elflein2018, address = {Am Hubland, Informatikgeb{\"a}ude, 97074 W{\"u}rzburg, Germany}, author = {Elflein, Sven}, keywords = {descartes}, month = {09}, note = {<b>W{\"u}rzburg Software Engineering Award sponsored by Bosch Rexroth</b>}, school = {University of W{\"u}rzburg}, title = {{Automatic Identification of Parametric Dependencies for Performance Models}}, type = {{Bachelor Thesis}}, year = 2018 }

%0 Thesis %1 Elflein2018 %A Elflein, Sven %C Am Hubland, Informatikgeb{ä}ude, 97074 W{ü}rzburg, Germany %D 2018 %T {Automatic Identification of Parametric Dependencies for Performance Models}

Curriculum Vitae / Resume

since August 2017	Doctoral reseacher at the Chair of Software-Engineering (Informatik II)
Aug 2015 - Aug 2017	Master of Computer Science at the University of Würzburg, Germany
Oct 2012- Aug 2015	Bachelor of Computer Science at the University of Würzburg, Germany