Considerations on efficient lexical analysis in the context of compiler design

Abstract

Each programming language needs a mechanism of translating source code into code understood by the computer, i.e., machine code or assembly. This is done by a compiler. A compiler is thus a fundamental tool for software development. Lexical analysis, as the first step of compilation, has an important role and should be performed in an optimal way. But not only compilers rely on lexical analysis. Its role lies nowadays also in pattern recognition or natural language processing. In this paper, we present some of the main aspects of lexical analysis and how to efficiently perform it. Our aim is to offer some guidelines in constructing a modern and efficient scanner (lexer), outlining the criteria, which should be considered, presenting some pitfalls, and comparing existent tools for compiler construction. The main contribution of the paper is to offer practical guidelines and recommendations, especially to graduate students and programmers, in constructing an efficient scanner (lexer), considering also the proper tools currently available.