A trace-driven analysis of the UNIX 4.2 BSD file system

Ousterhout, John K.; Costa, Hervé Da; Harrison, David; Kunze, John; Kupfer, Mike; Thompson, James

doi:10.1145/323647.323631

Cited by 266 publications

(79 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Table 2 summarizes some studies founded on the understanding realistic system behavior. They fall into the categories of network systems [29,34,36,16,24,8], storage systems [35,16,32,13,19,10], and large scale data centers running computation paradigms including MapReduce [8,39,33,10,30,11]. Network and storage subsystems are key components for MapReduce.…”

Section: Prior Workmentioning

confidence: 99%

Design Insights for MapReduce from Diverse Production Workloads

Chen¹,

Alspaugh²,

Katz³

2012

View full text Add to dashboard Cite

In this paper, we analyze seven MapReduce workload traces from production clusters at Facebook and at Cloudera customers in e-commerce, telecommunications, media, and retail. Cumulatively, these traces comprise over a year's worth of data logged from over 5000 machines, and contain over two million jobs that perform 1.6 exabytes of I/O. Key observations include input data forms up to 77% of all bytes, 90% of jobs access KB to GB sized files that make up less than 16% of stored bytes, up to 60% of jobs re-access data that has been touched within the past 6 hours, peak-to-median job submission rates are 9:1 or greater, an average of 68% of all compute time is spent in map, task-seconds-per-byte is a key metric for balancing compute and data bandwidth, task durations range from seconds to hours, and five out of seven workloads contain map-only jobs. We have also deployed a public workload repository with workload replay tools so that the researchers can systematically assess design priorities and compare performance across diverse MapReduce workloads.

show abstract

Section: Prior Workmentioning

confidence: 99%

Design Insights for MapReduce from Diverse Production Workloads

Chen¹,

Alspaugh²,

Katz³

2012

View full text Add to dashboard Cite

show abstract

“…In our research w e d o n o t i n vestigate read/write le access patterns, because most les are opened for either reading or writing, with few les updated 5,12]. We expect this to be especially true for the large les used in scienti c applications.…”

Section: Workload Modelmentioning

confidence: 99%

“…These applications typically push the leading edge of computing technology, such a s m ultiprocessors, placing tremendous demands on both CPU and I/O systems. Most le caching studies have examined generalpurpose workloads (e.g., 16]), where les are much smaller 12,5 ]. The parallel environment a n d w orkload raise a number of questions: Are caches useful for parallel scienti c applications using parallel le systems?…”

Section: Introductionmentioning

confidence: 99%

Caching and Writeback Policies in Parallel File Systems

Kotz

Ellis

1993

Journal of Parallel and Distributed Computing

View full text Add to dashboard Cite

“…Measurements have shown that write sharing (i.e. two or more users writing the same file) happens only infrequently [125]. Thus, occasional conflicts may be tolerable if the availability of the data can be improved by relaxing the consistency model.…”

Section: Foundations and Related Workmentioning

confidence: 99%

Self-Organizing Infrastructures for Ambient Services

Herrmann

Kommunikation in Verteilten Systemen (KiVS)

View full text Add to dashboard Cite

In the last two decades, the advent of wireless networking technology and the achievements in the miniaturization of electronic devices have set new trends in distributed computing. This ultimately enables a new paradigm for embedding mobile users in intelligent environments that support them in their interactions with their local physical surrounding. This vision is called Ambient Intelligence (AmI). There are still many challenges ahead on the way towards the realization of AmI. One question that is central to the entire concept is: How can we render AmI systems self-organizing such that they can indeed disappear in our environment without creating a massive administrative problem?In this thesis, we propose a model for a dedicated AmI infrastructure that supports the user in his interaction with his physical environment and with external entities. This infrastructure is called Ad hoc Service Grid (ASG) and provides wireless services in a decentralized and self-organizing fashion. We identify three distinct problems associated with self-organized service provisioning in the ASG model and propose algorithms and protocols that solve them. The problem that is at the center of our work is the self-organized replication and distribution of arbitrary services in an ASG. A set of algorithms is presented that solves this problem in a completely distributed way. The two other problems we tackle are the discovery and lookup of dynamically distributed service replicas and the reconciliation among a dynamic group of replicas. Together, the mechanisms we propose lay the foundation for a general AmI software platform. We will derive the architecture of such a Serviceware from these mechanisms. Detailed experimental results are presented that show the validity of our concepts and identify ways for tailoring our algorithms and protocols to the requirements of specific applications. Furthermore, we propose a new general model and a classification methodology for self-organizing software systems. We employ this model to evaluate our own solutions.The focus of the research work presented in this thesis is on the global-scale interactions in an AmI system. We call this the macro-level of interactions to separate it from the focus of most current research projects. These projects concentrate more on adaptations at the micro-level, pertaining to the internal structures of specific applications and services. Our work complements these efforts by providing solutions for structuring AmI systems externally, for example by distributing a group of service replicas within an ASG network.

show abstract

A trace-driven analysis of the UNIX 4.2 BSD file system

Cited by 266 publications

References 10 publications

Design Insights for MapReduce from Diverse Production Workloads

Design Insights for MapReduce from Diverse Production Workloads

Caching and Writeback Policies in Parallel File Systems

Self-Organizing Infrastructures for Ambient Services

Contact Info

Product

Resources

About