Thread-Shared Software Code Caches

Abstract

Software code caches are increasingly being used to amortize the runtime overhead of dynamic optimizers, simulators, emulators, dynamic translators, dynamic compilers, and other tools. Despite the now-widespread use of code caches, techniques for efficiently sharing them across multiple threads have not been fully explored. Some systems simply do not support threads, while others resort to thread-private code caches. Although thread-private caches are much simpler to manage, synchronize, and provide scratch space for, they simply do not scale when applied to many-threaded programs. Thread-shared code caches are needed to target server applications, which employ hundreds of worker threads all performing similar tasks. Yet, those systems that do share their code caches often have bruteforce, inefficient solutions to the challenges of concurrent code cache access: a single global lock on runtime system code and suspension of all threads for any cache management action. This limits the possibilities for cache design and has performance problems with applications that require frequent cache invalidations to maintain cache consistency.In this paper, we discuss the design choices when building thread-shared code caches and enumerate the difficulties of thread-local storage, synchronization, trace building, in-cache lookup tables, and cache eviction. We present efficient solutions to these problems that both scale well and do not require thread suspension. We evaluate our results in DynamoRIO, an industrial-strength dynamic binary translation system, on real-world server applications. On these applications our thread-shared caches use an order of magnitude less memory and improve throughput by up to four times compared to threadprivate caches.

Low overhead dynamic binary translation on ARM

“…This model has been shown to scale significantly better than thread-private code caches on multi-threaded applications [7,15]. Executing code in the code cache does not require any synchronization and MAMBO-X64 is able to perform code translation concurrently in multiple threads.…”

Fragment cache management for dynamic binary translators in embedded systems with scratchpad

“…Although more sophisticated eviction policies are possible, flushing the F$ is a simple one that is used to minimize the overhead of managing the F$ [5,11,27]. The problem with this approach is that fragments which are soon needed can be evicted, especially for a small F$.…”

“…Bruening et al investigated how to manage the F$ for multithreaded programs [5]. They also investigated how to bound the F$ and maintain consistency with self modifying code [4].…”

“…The management of the F$ has been widely studied for general-purpose systems and large fragment caches [2,3,4,5,11,21,23,27]. Bala et al suggested the strategy of flushing the F$ when it becomes full [2].…”

Comprehensive kernel instrumentation via dynamic binary translation

“…Lines 1-3 of HANDLE-INTERRUPT show these state-saving steps, which are common to all interrupt handling cases. Next, DRK gives the instrumentation client (if one exists) a chance to handle the interrupt (lines [4][5][6]. If the client has not fully handled the interrupt, DRK then decides to either compute native state and immediately dispatch or delay the interrupt.…”