Benchmark Generation for Timing Analysis

Wägemann, Peter; Distler, Tobias; Eichler, Christian; Schröder-Preikschat, Wolfgang

doi:10.1109/rtas.2017.6

Cited by 11 publications

(7 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The GenE benchmark [26,28] is one possible solution for the problem of conducting comprehensive evaluations and validations of worst-case tools. GenE generates benchmarks in a way that all program facts are known.…”

Section: Gene: Benchmark Generator For Wcet Toolsmentioning

confidence: 99%

“…From the perspective of assessing an analyzer's performance for determining loop bounds, this benchmark with zero resilience is unsuited since the problem of loop bounds is already straightforward to solve for optimizing compilers. The main observations [26] when applying Metrics WCA to GenE's benchmarks are (1) a high resilience against compiler optimizations and (2) comparable complexities with respect to a benchmark suite for WCET analysis [8]. These experiments used the standard configuration of GenE's path budget.…”

Section: Metrics Wca : Validation Of Gene's Benchmarksmentioning

confidence: 99%

“…2. The benchmark generator GenE allows for the first time comprehensive evaluations and validations of static worst-case tools based on automatically generated benchmarks, whose actual worst case is known [26,28,31] (see Sect. 3).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Static Worst-Case Analyses and Their Validation Techniques for Safety-Critical Systems

Wägemann

2022

Ernst Denert Award for Software Engineering 2020

View full text Add to dashboard Cite

The reliable operation of systems with both timing and energy requirements is a fundamental challenge in the area of safety-critical embedded systems. In order to provide guarantees for the execution of tasks within given resource budgets, these systems demand bounds of the worst-case execution time (WCET) and the worst-case energy consumption (WCEC). While static WCET analysis techniques are well established in the software development process of real-time systems nowadays, these program analysis techniques are not directly applicable to the fundamentally different behavior of energy consumption and the determination of the WCEC. Besides the missing approaches for WCEC bounds, the domain of worst-case analyses generally faces the problem that the accuracy and validity of reported analysis bounds are unknown: Since the actual worst-case resource consumption of existing benchmark programs cannot be automatically determined, a comprehensive validation of these program analysis tools is not possible.This summary of my dissertation addresses these problems by first describing a novel program analysis approach for WCEC bounds, which accounts for temporarily power-consuming devices, scheduling with fixed real-time priorities, synchronous task activations, and asynchronous interrupt service routines. Regarding the fundamental problem of validating worst-case tools, this dissertation presents a technique for automatically generating benchmark programs. The generator combines program patterns so that the worst-case resource consumption is available along with the generated benchmark. Knowledge about the actual worst-case resource demand then serves as the baseline for evaluating and validating program analysis tools. The fact the benchmark generator helped to reveal previously undiscovered software bugs in a widespread WCET tool for safety-critical systems underlines the relevance of such a structured testing technique.

show abstract

Section: Gene: Benchmark Generator For Wcet Toolsmentioning

confidence: 99%

Section: Metrics Wca : Validation Of Gene's Benchmarksmentioning

confidence: 99%

See 1 more Smart Citation

Static Worst-Case Analyses and Their Validation Techniques for Safety-Critical Systems

Wägemann

2022

Ernst Denert Award for Software Engineering 2020

View full text Add to dashboard Cite

show abstract

“…One line of work follows a generation-from-scratch strategy that generates a system given some input parameters. Wägemann et al [42] propose an iterated process of selecting and inserting programming patterns from a given library into an emerging program such that it ensures a designated program input to lead to the worst-case execution time, providing the ground-truth for the final benchmark.…”

Section: Background and Related Workmentioning

confidence: 99%

A Generator Framework For Evolving Variant-Rich Software

Derks¹,

Strüber²,

Berger³

2021

Preprint

View full text Add to dashboard Cite

Evolving software is challenging, even more when it exists in many different variants. Such software evolves not only in time, but also in space-another dimension of complexity. While evolution in space is supported by a variety of product-line and variability management tools, many of which originating from research, their level of evaluation varies significantly, which threatens their relevance for practitioners and future research. Many tools have only been evaluated on ad hoc datasets, minimal examples or available preprocessor-based product lines, missing the early clone & own phases and the re-engineering into configurable platforms-large parts of the actual evolution lifecycle of variant-rich systems. Our long-term goal is to provide benchmarks to increase the maturity of evaluating such tools. However, providing manually curated benchmarks that cover the whole evolution lifecycle and that are detailed enough to serve as ground truths, is challenging.We present the framework vpbench to generates source-code histories of variant-rich systems. Vpbench comprises several modular generators relying on evolution operators that systematically and automatically evolve real codebases and document the evolution in detail. We provide simple and more advanced generators-e.g., relying on code transplantation techniques to obtain whole features from external, real-world projects. We define requirements and demonstrate how vpbench addresses them for the generated version histories, focusing on support for evolution in time and space, the generation of detailed meta-data about the evolution, also considering compileability and extensibility. CCS CONCEPTS• Software and its engineering → Software product lines.

show abstract

“…Wägemann et al proposed GenE [8], a tool to generate benchmarks for timing analysis. It combines code patterns from real-time applications that are both representative for real-time applications and sufficiently challenging to WCET analysis tools.…”

Section: Related Workmentioning

confidence: 99%

Task-set generator for schedulability analysis using the TACLebench benchmark suite

et al. 2018

View full text Add to dashboard Cite

Currently, real-time embedded systems evolve towards complex systems using new state of the art technologies such as multi-core processors and virtualization techniques. Both technologies require new real-time scheduling algorithms. For uniprocessor scheduling, utilization-based evaluation methodologies are well-established. For multi-core systems and virtualization, evaluating and comparing scheduling techniques using the tasks' parameters is more realistic. Evaluating such scheduling techniques requires relevant and standardised task sets. Scheduling algorithms can be evaluated at three levels: 1) using a mathematical model of the algorithm, 2) simulating the algorithm and 3) implementing the algorithm on the target platform. Generating task sets is straightforward in the case of the first two levels; only the parameters of the tasks are required. Evaluating and comparing scheduling algorithms on the target platform itself, however, requires executable tasks matching the predefined standardised task sets. Generating those executable tasks is not standardized yet. Therefore, we developed a task-set generator that generates reproducible, standardised task sets that are suitable at all levels. Besides generating the tasks' parameters, it includes a method that generates executables by combining publicly available benchmarks with known execution times. This paper presents and evaluates this task-set generator.

show abstract

Benchmark Generation for Timing Analysis

Cited by 11 publications

References 18 publications

Static Worst-Case Analyses and Their Validation Techniques for Safety-Critical Systems

Static Worst-Case Analyses and Their Validation Techniques for Safety-Critical Systems

A Generator Framework For Evolving Variant-Rich Software

Task-set generator for schedulability analysis using the TACLebench benchmark suite

Contact Info

Product

Resources

About