Publications

Journal and Magazine Articles

Full Papers

The Impact of Memory Configuration on Server Efficiency. Meissner, Maximilian; Pham, Khang; Cragin, Aaron; Lange, Klaus-Dieter; Kounev, Samuel; in Proceedings of the 17th ACM/SPEC International Conference on Performance Engineering (2026). 63–73. Association for Computing Machinery, Italy.
- [ Abstract ]
- [ BibTeX ]
- [ URL ]
- [ DOI ]
The SPEC SERT suite is the industry-standard benchmark for evaluating server energy efficiency and is widely adopted in government regulations and certification programs. Current certification rules require every CPU memory channel to be populated with at least one Dual In-line Memory Module (DIMM). In real-world deployments, however, servers are sometimes configured with fewer DIMMs, leaving some channels unpopulated. This discrepancy introduces a significant gap between certified efficiency scores and the actual efficiency of deployed systems. To address this issue, there are ongoing discussions about certifying servers with partially populated memory channels. However, the impact of memory configuration on SPEC SERT results has not been systematically studied. In this paper, we present a comprehensive analysis of how different memory configurations affect performance, power consumption, and energy efficiency on two state-of-the-art server systems using the SPEC SERT 2 suite. We vary both the number of populated channels and the type of DIMMs. Our experimental results show that server efficiency scores can be up to 3.4 times lower with only one channel populated and still up to 1.3 times lower with half the channels, compared to fully populated configurations. Detailed analysis of individual SPEC SERT 2 worklets reveals that some CPU worklets are highly sensitive to memory bandwidth, and that the impact of memory configuration is dependent on both server architecture and workload intensity. These findings underscore the need to reconsider certification criteria and highlight the importance of memory configuration for accurate energy efficiency assessment.

@inproceedings{10.1145/3777884.3796988, abstract = {The SPEC SERT suite is the industry-standard benchmark for evaluating server energy efficiency and is widely adopted in government regulations and certification programs. Current certification rules require every CPU memory channel to be populated with at least one Dual In-line Memory Module (DIMM). In real-world deployments, however, servers are sometimes configured with fewer DIMMs, leaving some channels unpopulated. This discrepancy introduces a significant gap between certified efficiency scores and the actual efficiency of deployed systems. To address this issue, there are ongoing discussions about certifying servers with partially populated memory channels. However, the impact of memory configuration on SPEC SERT results has not been systematically studied. In this paper, we present a comprehensive analysis of how different memory configurations affect performance, power consumption, and energy efficiency on two state-of-the-art server systems using the SPEC SERT 2 suite. We vary both the number of populated channels and the type of DIMMs. Our experimental results show that server efficiency scores can be up to 3.4 times lower with only one channel populated and still up to 1.3 times lower with half the channels, compared to fully populated configurations. Detailed analysis of individual SPEC SERT 2 worklets reveals that some CPU worklets are highly sensitive to memory bandwidth, and that the impact of memory configuration is dependent on both server architecture and workload intensity. These findings underscore the need to reconsider certification criteria and highlight the importance of memory configuration for accurate energy efficiency assessment.}, address = {New York, NY, USA}, author = {Meissner, Maximilian and Pham, Khang and Cragin, Aaron and Lange, Klaus-Dieter and Kounev, Samuel}, booktitle = {Proceedings of the 17th ACM/SPEC International Conference on Performance Engineering}, keywords = {descartes}, pages = {63–73}, publisher = {Association for Computing Machinery}, series = {ICPE '26}, title = {The Impact of Memory Configuration on Server Efficiency}, year = 2026 }

Short Papers

Automated Triage of Performance Change Points Using Time Series Analysis and Machine Learning. Bauer, André; Straesser, Martin; Beierlieb, Lukas; Meissner, Maximilian; Kounev, Samuel; in Proceedings of the 2022 ACM/SPEC on International Conference on Performance Engineering (2022). ACM.
- [ BibTeX ]
- [ Download ]
@conference{bauer2022automated, author = {Bauer, André and Straesser, Martin and Beierlieb, Lukas and Meissner, Maximilian and Kounev, Samuel}, booktitle = {Proceedings of the 2022 ACM/SPEC on International Conference on Performance Engineering}, keywords = {machine_learning}, month = {04}, publisher = {ACM}, title = {Automated Triage of Performance Change Points Using Time Series Analysis and Machine Learning}, year = 2022 }

Workshop Papers

Experience and Guidelines for Sorting Algorithm Choices and Their Energy Efficiency. Meissner, Maximilian; Kamthania, Supriya; Rawtani, Nishant; Bucek, James; Lange, Klaus-Dieter; Kounev, Samuel; in Companion of the 2022 ACM/SPEC International Conference on Performance Engineering (2022). ACM, New York, NY, USA.
- [ BibTeX ]
- [ Download ]
@inproceedings{meissner2022experience, address = {New York, NY, USA}, author = {Meissner, Maximilian and Kamthania, Supriya and Rawtani, Nishant and Bucek, James and Lange, Klaus-Dieter and Kounev, Samuel}, booktitle = {Companion of the 2022 ACM/SPEC International Conference on Performance Engineering}, keywords = {t_workshop}, month = {04}, publisher = {ACM}, title = {Experience and Guidelines for Sorting Algorithm Choices and Their Energy Efficiency}, year = 2022 }

Book Chapters

Server Energy Efficiency Benchmarks. von Kistowski, J’oakim; Meißner, Maximilian; Lange, Klaus-Dieter; Arnold, Jeremy A. In Systems Benchmarking: For Scientists and Engineers, pp. 247–265. Springer Nature Switzerland, Cham, 2025.
- [ Abstract ]
- [ BibTeX ]
- [ DOI ]
This chapter starts by introducing the fundamentals of power and efficiency measurements and then describes a power and performance benchmark methodology developed by the SPECpower committee for commodity servers. It is designed to characterize and rate the energy efficiency of a system under test for multiple load levels, showcasing load level differences in system behavior regarding energy efficiency. The methodology was first implemented in the SPECpower_ssj 2008 benchmark and later extended with more workloads, metrics, and other application areas for the SPEC Server Efficiency Rating Tool (SERT) suite. The SERT suite was developed to fill the need for a rating tool that can be utilized by government agencies in their regulatory programs, for example, the U.S. Environmental Protection Agency (EPA) for the use in their ENERGY STAR program for servers.

@incollection{sbBookSecondEd, abstract = {This chapter starts by introducing the fundamentals of power and efficiency measurements and then describes a power and performance benchmark methodology developed by the SPECpower committee for commodity servers. It is designed to characterize and rate the energy efficiency of a system under test for multiple load levels, showcasing load level differences in system behavior regarding energy efficiency. The methodology was first implemented in the SPECpower{_}ssj 2008 benchmark and later extended with more workloads, metrics, and other application areas for the SPEC Server Efficiency Rating Tool (SERT) suite. The SERT suite was developed to fill the need for a rating tool that can be utilized by government agencies in their regulatory programs, for example, the U.S. Environmental Protection Agency (EPA) for the use in their ENERGY STAR program for servers.}, address = {Cham}, author = {von Kistowski, J{'o}akim and Mei{\ss}ner, Maximilian and Lange, Klaus-Dieter and Arnold, Jeremy A.}, booktitle = {Systems Benchmarking: For Scientists and Engineers}, keywords = {t_bookchapter}, pages = {247–265}, publisher = {Springer Nature Switzerland}, title = {Server Energy Efficiency Benchmarks}, year = 2025 }

Vision and Position Papers

Challenges and Future Directions in Efficiency Benchmarking (Vision Paper). Meissner, Maximilian; Lange, Klaus-Dieter; Sharma, Sanjay; Arnold, Jeremy; Cragin, Aaron; Galizia, Patrick; Petrich, Mike; Zhang, Bin; Kounev, Samuel; in Companion of the 2023 ACM/SPEC International Conference on Performance Engineering (2023). 51–55. Association for Computing Machinery, New York, NY, USA.
- [ Abstract ]
- [ BibTeX ]
- [ URL ]
SPEC benchmarks are crucial contributors behind the improvement of server efficiency since 2007, given their role in making the power consumption and efficiency of servers transparent for government regulators, customers, and the manufacturers themselves.As the IT landscape experiences radical transformations, efficiency benchmarks need to be updated accordingly to generate results relevant to government regulators, manufacturers, and customers. In this paper, we outline current challenges efficiency benchmark developers are tackling and highlight recent technological developments the next generation of efficiency benchmarks should take into account.

@inproceedings{10.1145/3578245.3585034, abstract = {SPEC benchmarks are crucial contributors behind the improvement of server efficiency since 2007, given their role in making the power consumption and efficiency of servers transparent for government regulators, customers, and the manufacturers themselves.As the IT landscape experiences radical transformations, efficiency benchmarks need to be updated accordingly to generate results relevant to government regulators, manufacturers, and customers. In this paper, we outline current challenges efficiency benchmark developers are tackling and highlight recent technological developments the next generation of efficiency benchmarks should take into account.}, address = {New York, NY, USA}, author = {Meissner, Maximilian and Lange, Klaus-Dieter and Sharma, Sanjay and Arnold, Jeremy and Cragin, Aaron and Galizia, Patrick and Petrich, Mike and Zhang, Bin and Kounev, Samuel}, booktitle = {Companion of the 2023 ACM/SPEC International Conference on Performance Engineering}, keywords = {descartes}, pages = {51–55}, publisher = {Association for Computing Machinery}, series = {ICPE '23 Companion}, title = {Challenges and Future Directions in Efficiency Benchmarking (Vision Paper)}, year = 2023 }

Poster Papers

Towards Performance-Oriented Planning and Development of IoT Applications and Networks. Prantl, Thomas; Horn, Lukas; Engel, Simon; Meissner, Maximilian; Bonilla, Rafael; Kounev, Samuel; in Companion of the 17th ACM/SPEC International Conference on Performance Engineering (2026). 92–93. Association for Computing Machinery, Italy.
- [ Abstract ]
- [ BibTeX ]
- [ URL ]
- [ DOI ]
The planning of IoT applications requires consideration of the performance, energy consumption, and security of resource-constrained devices. In particular, n-to-n communication patterns and security mechanisms significantly affect the runtime, energy consumption, and reliability of IoT systems. In this article, we summarize our previous work on the performance and security analysis of secure group communication and attribute-based encryption methods, as well as on the evaluation of publish/subscribe brokers. Based on measurements on real-world IoT hardware, we outline an approach for modeling and combining individual IoT operations to estimate time and energy consumption in advance. The goal is to support developers in performance-oriented planning and development of energy-efficient, secure IoT applications.

@inproceedings{10.1145/3777911.3799728, abstract = {The planning of IoT applications requires consideration of the performance, energy consumption, and security of resource-constrained devices. In particular, n-to-n communication patterns and security mechanisms significantly affect the runtime, energy consumption, and reliability of IoT systems. In this article, we summarize our previous work on the performance and security analysis of secure group communication and attribute-based encryption methods, as well as on the evaluation of publish/subscribe brokers. Based on measurements on real-world IoT hardware, we outline an approach for modeling and combining individual IoT operations to estimate time and energy consumption in advance. The goal is to support developers in performance-oriented planning and development of energy-efficient, secure IoT applications.}, address = {New York, NY, USA}, author = {Prantl, Thomas and Horn, Lukas and Engel, Simon and Meissner, Maximilian and Bonilla, Rafael and Kounev, Samuel}, booktitle = {Companion of the 17th ACM/SPEC International Conference on Performance Engineering}, keywords = {myown}, pages = {92–93}, publisher = {Association for Computing Machinery}, series = {ICPE Companion '26}, title = {Towards Performance-Oriented Planning and Development of IoT Applications and Networks}, year = 2026 }
SPEC Efficiency Benchmarks - Achievements and Prospects. Meissner, Maximilian; Lange, Klaus-Dieter; Cragin, Aaron; Sharma, Sanjay; Galizia, Patrick; Kounev, Samuel; in Companion of the 17th ACM/SPEC International Conference on Performance Engineering (2026). 96–97. Association for Computing Machinery, Italy.

Best Poster Award
- [ Abstract ]
- [ BibTeX ]
- [ URL ]
- [ DOI ]
For almost two decades, the Standard Performance Evaluation Corporation (SPEC) has developed widely used tools to reliably evaluate system efficiency: the SPEC PTDaemonR interface, the SPECpower_ssjR 2008 benchmark, and the SPEC SERTR> suite. To ensure relevant, high?quality, and reproducible measurements, and to adapt continuously to evolving industry and regulatory requirements, these tools are under continuous development. We present an overview of their evolution and describe current progress towards the next major versions of the benchmarks.

@inproceedings{10.1145/3777911.3799730, abstract = {For almost two decades, the Standard Performance Evaluation Corporation (SPEC) has developed widely used tools to reliably evaluate system efficiency: the SPEC PTDaemonR interface, the SPECpower_ssjR 2008 benchmark, and the SPEC SERTR> suite. To ensure relevant, high?quality, and reproducible measurements, and to adapt continuously to evolving industry and regulatory requirements, these tools are under continuous development. We present an overview of their evolution and describe current progress towards the next major versions of the benchmarks.}, address = {New York, NY, USA}, author = {Meissner, Maximilian and Lange, Klaus-Dieter and Cragin, Aaron and Sharma, Sanjay and Galizia, Patrick and Kounev, Samuel}, booktitle = {Companion of the 17th ACM/SPEC International Conference on Performance Engineering}, keywords = {myown}, note = {Best Poster Award}, pages = {96–97}, publisher = {Association for Computing Machinery}, series = {ICPE Companion '26}, title = {SPEC Efficiency Benchmarks - Achievements and Prospects}, year = 2026 }

Tutorial Papers

Energy Efficiency Benchmarking of GPU Servers - Why It’s Challenging, What Matters, and How to Do It Right. Meissner, Maximilian; Lange, Klaus-Dieter; Arnold, Jeremy; Sharma, Sanjay; Cragin, Aaron; Reiner, David; Kounev, Samuel; in Companion of the 17th ACM/SPEC International Conference on Performance Engineering (2026). 110–113. Association for Computing Machinery, Italy.
- [ Abstract ]
- [ BibTeX ]
- [ URL ]
- [ DOI ]
The rapid adoption of GPU-accelerated workloads for AI and high-performance computing has driven a sharp rise in data center energy consumption, making reliable energy efficiency measurement critical for research, system design, and operational decisions. Unlike conventional performance benchmarking, energy efficiency benchmarking introduces new sources of variability (e.g., power-domain selection, sampling strategy, thermal and power-management policies, and workload non-determinism) that must be explicitly controlled and documented. Small changes in drivers, firmware, or tuning can produce large, misleading differences in reported efficiency, particularly on modern GPU platforms where hardware/software co-optimization and vendor-specific features are pervasive. In this paper, we summarize best practices and lessons learned from implementing GPU workloads for the latest SPEC efficiency benchmarks, providing guidance for reliable, reproducible energy efficiency experiments on GPU servers. Our contributions aim to standardize methodology, reduce ambiguity in comparisons, and enable trusted energy efficiency benchmarking of GPU-accelerated systems.

@inproceedings{10.1145/3777911.3799724, abstract = {The rapid adoption of GPU-accelerated workloads for AI and high-performance computing has driven a sharp rise in data center energy consumption, making reliable energy efficiency measurement critical for research, system design, and operational decisions. Unlike conventional performance benchmarking, energy efficiency benchmarking introduces new sources of variability (e.g., power-domain selection, sampling strategy, thermal and power-management policies, and workload non-determinism) that must be explicitly controlled and documented. Small changes in drivers, firmware, or tuning can produce large, misleading differences in reported efficiency, particularly on modern GPU platforms where hardware/software co-optimization and vendor-specific features are pervasive. In this paper, we summarize best practices and lessons learned from implementing GPU workloads for the latest SPEC efficiency benchmarks, providing guidance for reliable, reproducible energy efficiency experiments on GPU servers. Our contributions aim to standardize methodology, reduce ambiguity in comparisons, and enable trusted energy efficiency benchmarking of GPU-accelerated systems.}, address = {New York, NY, USA}, author = {Meissner, Maximilian and Lange, Klaus-Dieter and Arnold, Jeremy and Sharma, Sanjay and Cragin, Aaron and Reiner, David and Kounev, Samuel}, booktitle = {Companion of the 17th ACM/SPEC International Conference on Performance Engineering}, keywords = {t_tutorial}, pages = {110–113}, publisher = {Association for Computing Machinery}, series = {ICPE Companion '26}, title = {Energy Efficiency Benchmarking of GPU Servers - Why It's Challenging, What Matters, and How to Do It Right}, year = 2026 }
Next Generation Energy Efficiency Benchmarking: Reliable and Reproducible Efficiency Measurements in a Diverse IT-Landscape. Meissner, Maximilian; Lange, Klaus-Dieter; Cragin, Aaron; H J, Pushpa; Proulx, Robert; Kounev, Samuel; in Companion of the 16th ACM/SPEC International Conference on Performance Engineering (2025). 60–64. Association for Computing Machinery, Toronto ON, Canada.
- [ Abstract ]
- [ BibTeX ]
- [ URL ]
- [ DOI ]
Energy efficiency has become a critical quality criterion of IT-systems. While hardware designers have long worked on improving efficiency, researchers in other fields, such as software design and Artificial Intelligence, are increasingly focusing on this issue as well. Such combined efforts are essential to mitigate the significant energy demands of current and future hardware and software. In comparison to performance benchmarking, energy efficiency benchmarking involves a variety of additional factors that must be carefully considered and controlled. Without proper setup and oversight, measurements can easily lead to misleading, inconclusive, or unreliable results. Because reproducibility of experiment results, or lack thereof, is a critical concern in the scientific community, knowing the intricacies of power and efficiency measurements is a crucial pre-requisite for carrying out meaningful experiments. This paper offers insight into aspects important for conducting energy efficiency experiments based on best practices. It explains the effects of subtle differences in experiment setup and discusses current trends and challenges in efficiency benchmarking.

@inproceedings{10.1145/3680256.3721255, abstract = {Energy efficiency has become a critical quality criterion of IT-systems. While hardware designers have long worked on improving efficiency, researchers in other fields, such as software design and Artificial Intelligence, are increasingly focusing on this issue as well. Such combined efforts are essential to mitigate the significant energy demands of current and future hardware and software. In comparison to performance benchmarking, energy efficiency benchmarking involves a variety of additional factors that must be carefully considered and controlled. Without proper setup and oversight, measurements can easily lead to misleading, inconclusive, or unreliable results. Because reproducibility of experiment results, or lack thereof, is a critical concern in the scientific community, knowing the intricacies of power and efficiency measurements is a crucial pre-requisite for carrying out meaningful experiments. This paper offers insight into aspects important for conducting energy efficiency experiments based on best practices. It explains the effects of subtle differences in experiment setup and discusses current trends and challenges in efficiency benchmarking.}, address = {New York, NY, USA}, author = {Meissner, Maximilian and Lange, Klaus-Dieter and Cragin, Aaron and H J, Pushpa and Proulx, Robert and Kounev, Samuel}, booktitle = {Companion of the 16th ACM/SPEC International Conference on Performance Engineering}, keywords = {t_tutorial}, pages = {60–64}, publisher = {Association for Computing Machinery}, series = {ICPE '25}, title = {Next Generation Energy Efficiency Benchmarking: Reliable and Reproducible Efficiency Measurements in a Diverse IT-Landscape}, year = 2025 }
SPEC Efficiency Benchmark Development: How to Contribute to the Future of Energy Conservation. Meissner, Maximilian; Lange, Klaus-Dieter; Arnold, Jeremy; Sharma, Sanjay; Tipley, Roger; Rawtani, Nishant; Reiner, David; Petrich, Mike; Cragin, Aaron; in Companion of the 2022 ACM/SPEC International Conference on Performance Engineering (2022). ACM, New York, NY, USA.
- [ Abstract ]
- [ BibTeX ]
- [ Download ]
A driving force behind the improvement of server efficiency in recent years is the use of SPEC benchmarks. They are used in mandatory government regulations, the ISO/IEC 21836:2020 standard, and product marketing, giving server manufacturers and buyers significant incentive to improve energy efficiency. To produce relevant results, benchmarks need to take into account future trends in hardware and software development, such as the introduction of new accelerators and workloads. To keep pace with the development of the fast moving IT landscape, SPEC plans to introduce a workload bounty program to encourage researchers to develop novel workloads. Submitted workloads will be considered for inclusion in future SPEC Efficiency benchmarks and rewarded. In this paper, we outline the process of energy-efficiency benchmark development. SPEC ensures the development of high-quality benchmarks for government regulations through its extensive experience and collaboration with stakeholders from industry, academia, and governments. One of the tools that emerged from this process is the Chauffeur Worklet Development Kit (WDK), which can be used by researchers to develop next-generation workloads to enhance the real-world relevance of future SPEC benchmarks, a critical element for the benchmarks to contribute to future energy conservation.

@inproceedings{meissner2022efficiency, abstract = {A driving force behind the improvement of server efficiency in recent years is the use of SPEC benchmarks. They are used in mandatory government regulations, the ISO/IEC 21836:2020 standard, and product marketing, giving server manufacturers and buyers significant incentive to improve energy efficiency. To produce relevant results, benchmarks need to take into account future trends in hardware and software development, such as the introduction of new accelerators and workloads. To keep pace with the development of the fast moving IT landscape, SPEC plans to introduce a workload bounty program to encourage researchers to develop novel workloads. Submitted workloads will be considered for inclusion in future SPEC Efficiency benchmarks and rewarded. In this paper, we outline the process of energy-efficiency benchmark development. SPEC ensures the development of high-quality benchmarks for government regulations through its extensive experience and collaboration with stakeholders from industry, academia, and governments. One of the tools that emerged from this process is the Chauffeur Worklet Development Kit (WDK), which can be used by researchers to develop next-generation workloads to enhance the real-world relevance of future SPEC benchmarks, a critical element for the benchmarks to contribute to future energy conservation.}, address = {New York, NY, USA}, author = {Meissner, Maximilian and Lange, Klaus-Dieter and Arnold, Jeremy and Sharma, Sanjay and Tipley, Roger and Rawtani, Nishant and Reiner, David and Petrich, Mike and Cragin, Aaron}, booktitle = {Companion of the 2022 ACM/SPEC International Conference on Performance Engineering}, keywords = {spec}, month = {04}, publisher = {ACM}, title = {SPEC Efficiency Benchmark Development: How to Contribute to the Future of Energy Conservation}, year = 2022 }

Journal and Magazine Articles

Full Papers

Short Papers

Workshop Papers

Book Chapters

Vision and Position Papers

Poster Papers

Tutorial Papers

Talks

Techreports

Bildnachweise