Benchmarking the Dependability of Windows and Linux Using PostMark&amp;#8482; Workloads

Kanoun, Karama; Crouzet, Yves; Kalakech, Ali; Rugina, Ana-Elena; Rumeau, Pierre

doi:10.1109/issre.2005.13

Cited by 29 publications

(15 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Doing this in a systematic and repeatable way is essential when different evaluators need to make a fair comparison of their results, so the methodology relies on a mathematical formalism. Second, defining our methodology in such a way it may satisfy the conflicting positions between (i) those evaluation consumers that prefer having all the possible measures as field data for enabling deep result analysis and promote data sharing among community members [105] (e.g., people from academia), and (ii) those adopting a more pragmatical viewpoint that ask for an small set of meaningful and representative scores to characterise, rank and compare evaluated systems [106] (e.g., people from industry). To cope with this goal REFRAHN relies on the notion of quality model, adopted from ISO/IEC 25000 standards [107], to formulate not only rigorous but also usable and flexible interpretation rules.…”

Section: Analysis and Interpretation Of Resultsmentioning

confidence: 99%

A survey of evaluation platforms for ad hoc routing protocols: A resilience perspective

et al. 2014

View full text Add to dashboard Cite

Section: Analysis and Interpretation Of Resultsmentioning

confidence: 99%

A survey of evaluation platforms for ad hoc routing protocols: A resilience perspective

et al. 2014

View full text Add to dashboard Cite

“…al. developed a dependability benchmark for several versions of Windows and Linux [Kanoun et al, 2005]. The benchmark is defined as a measure of the robustness of the OS's ability to withstand invalid API inputs.…”

Section: Other Techniques For Verification and Validation29mentioning

confidence: 99%

“…This is an important property for any workload used in a comparative purpose and is a well established approach in the benchmarking community, like the standard tests in SPEC [SPE]. A deterministic workload is key for achieving reproducibility, an important property identified for dependability benchmarks [Johansson, 2001;Kanoun et al, 2005]. As one still strives after using real-world workloads, an approach is to modify applications by removing sources of non-determinism, such as user inputs, by using specific user scenarios or use cases.…”

Section: Workloadmentioning

confidence: 99%

Robustness Evaluation of Operating Systems

Johansson

2008

Information Assurance

View full text Add to dashboard Cite

SummaryThe premise behind this thesis is the observation that Operating Systems (OS), being the foundation behind operations of computing systems, are complex entities and also subject to failures. Consequently, when they do fail, the impact is the loss of system service and the applications running thereon. While a multitude of sources for OS failures exist, device drivers are often identified as a prominent cause behind failures.In order to characterize the impact of driver failures, at both the OS and application levels, this thesis develops a framework for error propagation-based robustness profiling for an OS. The framework is first developed conceptually and then experimentally validated on a real OS, namely Windows CE .Net. The choice of Windows CE is driven by its representativeness for a multitude of OS's, as well as the ability to customize the OS components for particular needs.For experimental validation, fault injection is a prominent technique that can be used to simulate faults (or errors) in the system by inserting synthetic ones and study their effect. Three key questions with such a technique are where, what and when to inject faults. This thesis shows how injecting errors at the interface between drivers and the OS can be very effective in evaluating the effects driver faults can have.To quantify the OS's robustness, this thesis defines a series of error propagation measures, specifically tailored for device drivers. These measures allow for quantifying and comparing both individual services and device drivers on their susceptibility and diffusing abilities.This thesis compares three contemporary error models on their suitability for robustness evaluation. The classical bit-flip model is found to identify a higher number of severe failures in the system. It also identifies failures for more services than both other models, data type and fuzzing. However, its main drawback is that it requires substantially more injections than the other two. Fuzzing, even though not giving rise to as many failures is able to find new additional services with severe failures.A careful study of the injections performed with the bit-flip model shows that only a few bits are generally useful for identifying new services with robustness weaknesses. Consequently, a new composite model is proposed, combining the most effective bits of the bit-flip model with the fuzzing model's ability to identify new services, giving rise to new model without loss of important information and at the same time incurring a moderate number of injections.To answer the question of when to inject an error this thesis proposes a novel model of a driver's usage profile, focusing on high-level operations being carried out. It guides the injection of errors to instances when the driver is carrying out specific operations. Results from extensive fault injection experiments show that more service vulnerabilities can be discovered. Furthermore, a priori profiling of the drivers can show how effective the proposed approach will be. vi

show abstract

“…Doing this in a systematic and repeatable way is essential when different evaluators need to make a fair comparison of their 4 results, so the methodology relies on a mathematical formalism. Second, defining our methodology in such a way it may satisfy the conflicting positions between (i) those evaluation consumers that prefer having all the possible measures as field data for enabling deep result analysis and promote data sharing among community members [7] (e.g., people from academia), and (ii) those adopting a more pragmatical viewpoint that ask for an small set of meaningful and representative scores to characterize, rank and compare evaluated systems [8] (e.g., people from industry). To cope with this goal we rely on the notion of quality model, adopted from ISO/IEC 25000 standards [9], to formulate not only rigorous but also usable and flexible interpretation rules.…”

Section: Introductionmentioning

confidence: 99%

Multi-criteria analysis of measures in benchmarking: Dependability benchmarking as a case study

Friginal

Martínez

Andrés

et al. 2016

Journal of Systems and Software

View full text Add to dashboard Cite

Benchmarks enable the comparison of computer-based systems attending to a variable set of criteria, such as dependability, security, performance, cost and/or power consumption. Despite its difficulty, the multi-criteria analysis of results remains today a subjective process rarely addressed in an explicit way in existing benchmarks. It is thus not surprising that industrial benchmarks only rely on the use of a reduced set of easy-to-understand measures, specially when considering complex systems.This is a way to keep the process of result interpretation straightforward and unambiguous.However, it limits at the same time the richness and depth of the analysis process. This is why the academia prefers to characterize complex systems with a wider set of measures. Marrying the requirements of industry and academia in a single proposal remains a challenge today. This paper addresses this question by reducing the uncertainty of the analysis process using quality (score-based) models. At measure definition time, these models make explicit (i) which are the requirements imposed to each type of measure, that may vary from one context of use to another, and (ii) which is the type, and intensity, of the relation between considered measures. At measure analysis time, they provide a consistent, straightforward and unambiguous method to interpret resulting measures. The methodology and its practical use are illustrated through three different case studies from the dependability benchmarking domain, which usually Preprint submitted to Elsevier May 13, 2015 consider several different criteria including both performance and dependability ones.Although the proposed approach is limited to dependability benchmarks in this document, its usefulness for any type of benchmark seems quite evident attending to the general formulation of the provided solution.

show abstract

Benchmarking the Dependability of Windows and Linux Using PostMark™ Workloads

Cited by 29 publications

References 14 publications

A survey of evaluation platforms for ad hoc routing protocols: A resilience perspective

A survey of evaluation platforms for ad hoc routing protocols: A resilience perspective

Robustness Evaluation of Operating Systems

Multi-criteria analysis of measures in benchmarking: Dependability benchmarking as a case study

Contact Info

Product

Resources

About