Implementation of a Linux log-structured file system with a garbage collector

Jambor, Martin; Hrubý, Tomáš; Taus, Jan; Krchak, Kuba; Holub, Viliam

doi:10.1145/1228291.1228299

Cited by 10 publications

(3 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The read performance is comparable to that of a traditional LFS. Jambor et al [2007] is similar to that of Seltzer et al [1993]. The authors discuss the implementation of their LFS.…”

Section: Wang Et Al the Goal Ofmentioning

confidence: 52%

Online reorganization of databases

Sockut

Iyer

2009

ACM Comput. Surv.

View full text Add to dashboard Cite

In practice, any database management system sometimes needs reorganization, that is, a change in some aspect of the logical and/or physical arrangement of a database. In traditional practice, many types of reorganization have required denying access to a database (taking the database offline) during reorganization. Taking a database offline can be unacceptable for a highly available (24-hour) database, for example, a database serving electronic commerce or armed forces, or for a very large database. A solution is to reorganize online (concurrently with usage of the database, incrementally during users' activities, or interpretively). This article is a tutorial and survey on requirements, issues, and strategies for online reorganization. It analyzes the issues and then presents the strategies, which use the issues. The issues, most of which involve design trade-offs, include use of partitions, the locus of control for the process that reorganizes (a background process or users' activities), reorganization by copying to newly allocated storage (as opposed to reorganizing in place), use of differential files, references to data that has moved, performance, and activation of reorganization. The article surveys online strategies in three categories of reorganization. The first category, maintenance, involves restoring the physical arrangement of data instances without changing the database definition. This category includes restoration of clustering, reorganization of an index, rebalancing of parallel or distributed data, garbage collection for persistent storage, and cleaning (reclamation of space) in a logstructured file system. The second category involves changing the physical database definition; topics include construction of indexes, conversion between B + -trees and linear hash files, and redefinition (e.g., splitting) of partitions. The third category involves changing the logical database definition. Some examples are changing a column's data type, changing the inheritance hierarchy of object classes, and changing a relationship from one-to-many to many-to-many. The survey encompasses both research and commercial implementations, and this article points out several open research topics. As highly available or very large databases continue to become more common and more important in the world economy, the importance of online reorganization is likely to continue growing.

show abstract

“…The read performance is comparable to that of a traditional LFS. Jambor et al [2007] is similar to that of Seltzer et al [1993]. The authors discuss the implementation of their LFS.…”

Section: Wang Et Al the Goal Ofmentioning

confidence: 52%

Online reorganization of databases

Sockut

Iyer

2009

ACM Comput. Surv.

View full text Add to dashboard Cite

show abstract

“…ScaleFS's on-disk file system provides concurrent I/O using standard striping techniques. It could be further extended using more sophisticated techniques from file systems such as LinuxLFS [21], BTRFS [38], TABLEFS [36], NoFS [7], and XFS [40] to increase disk performance.…”

Section: Related Workmentioning

confidence: 99%

Scaling a file system to many cores using an operation log

Bhat

Eqbal

Clements

et al. 2017

Proceedings of the 26th Symposium on Operating Systems Principles

View full text Add to dashboard Cite

It is challenging to simultaneously achieve multicore scalability and high disk throughput in a file system. For example, even for commutative operations like creating different files in the same directory, current file systems introduce cache-line conflicts when updating an in-memory copy of the on-disk directory block, which limits scalability. ScaleFS is a novel file system design that decouples the in-memory file system from the on-disk file system using per-core operation logs. This design facilitates the use of highly concurrent data structures for the in-memory representation, which allows commutative operations to proceed without cache conflicts and hence scale perfectly. ScaleFS logs operations in a per-core log so that it can delay propagating updates to the disk representation (and the cache-line conflicts involved in doing so) until an fsync. The fsync call merges the per-core logs and applies the operations to disk. ScaleFS uses several techniques to perform the merge correctly while achieving good performance: timestamped linearization points to order updates without introducing cache-line conflicts, absorption of logged operations, and dependency tracking across operations. Experiments with a prototype of ScaleFS show that its implementation has no cache conflicts for 99% of test cases of commutative operations generated by Commuter, scales well on an 80-core machine, and provides on-disk performance that is comparable to that of Linux ext4. † Now at VMware. ‡ Now at Apple. § Now at Google.

show abstract

“…For applications dealing with data, file system component of the OS is critical. Many file systems [23], [24] are proposed, implemented and merged into Linux distributions.…”

Section: Linux and File Systemsmentioning

confidence: 99%

OS-Caused Large JVM Pauses: Investigations and Solutions

Zhuang¹,

Tran²,

Ramachandra³

et al. 2016

JSW

View full text Add to dashboard Cite

For customer-facing Java applications (e.g., online gaming and online chatting), ensuring low latencies is not just a preferred feature, but a must-have feature. Given the popularity and powerfulness of Java, a significant portion of today's backend services are implemented in Java. JVM (Java Virtual Machine) manages a heap space to hold application objects. The heap space can be frequently GC-ed (Garbage Collected), and applications can be occasionally stopped by JVM GC pauses. As a result, ensuring low JVM GC pauses is critical for business applications that have latency SLA (Service Level Agreements).In our production system, we found out that there are some (and large) JVM pauses cannot be explained by application-level activities and JVM activities during GC; instead, they are caused by OS mechanisms. In this work, we investigate the large JVM pauses caused by OS. We characterize various such problems into two scenarios. For both scenarios, we successfully root-cause the reasons and propose solutions. The findings can be used to mitigate the long JVM pauses and enhance JVM implementations. We share the knowledge and experiences in this writing.

show abstract

Implementation of a Linux log-structured file system with a garbage collector

Cited by 10 publications

References 10 publications

Online reorganization of databases

Online reorganization of databases

Scaling a file system to many cores using an operation log

OS-Caused Large JVM Pauses: Investigations and Solutions

Contact Info

Product

Resources

About