The development of the HiPE system: design and experience report

Abstract

Abstract. The concurrent functional programming language Erlang has been designed to ease the development of large-scale distributed soft real-time control applications. So far, it has been used quite successfully in industry, both within Ericsson Telecom, where it was designed and developed, and by other companies. This "declarative language success-story" has taken place despite the fact that Erlang implementations are slow compared with implementations of other functional languages. Wanting to improve the performance aspects of publicly available Erlang implementations, which are based on emulators, we embarked on a project called HiPE (High-Performance Erlang) whose aim has been to develop an efficient just-in-time native code compiler for Erlang (called the HiPE system). Since its start in 1996, the system has gone through various (re-)design phases, partly due to implementation choices that did not turn out to be as promising as they appeared on paper, but mainly due to changes in Ericsson's Erlang system upon which the HiPE system is built. In this article, we describe how the HiPE system was developed, what it currently looks like, and its current performance. We critically examine design decisions that we took, and the main lessons learnt from implementing them. Finally, we also report on our experiences from trying to keep up with the concurrent development of Ericsson's base Erlang system. As such, this article both documents the HiPE system and can serve as possible guidance to anyone wishing to attempt a similar feat.

HiPE on AMD64

“…Each Erlang process has two stacks, one for interpreted code and one for native code. As explained in [10], this simplifies garbage collection and exception handling, since each stack contains only frames of a single type. Control flow between interpreted and native code, e.g.…”

“…The HiPE native code compiler [9,10] 1 has been developed with the aim of reducing this performance gap. It achieves this goal while allowing fine-grained, user-controlled compilation of Erlang functions or modules to native machine code.…”

High availability, elasticity, and strong consistency for massively parallel scans over relational data

“…Crescando/RB was running on Erlang/OTP 6 R14B02. Crescando/RB and the OTP libraries were compiled to native code, using the High-Performance Erlang Compiler (HiPE) [50]. The Crescando storage engine, written in C++, was compiled into a stand-alone executable using GCC 4.6.1.…”

Efficiently compiling a functional language on AMD64

“…In this section, to set the context of our work, we briefly describe the Erlang/OTP system and the HiPE native code compiler which is the basis of our work; refer to [9,13] for more detailed information.…”

“…Detecting the need for a REX prefix is easily done while the assembler is encoding an instruction: any use of a high register number (8)(9)(10)(11)(12)(13)(14)(15) or a 64-bit operation triggers it.…”