F1 lightning

Abstract

The ongoing and increasing interest in HTAP (Hybrid Transactional and Analytical Processing) systems documents the intense interest from data owners in simultaneously running transactional and analytical workloads over the same data set. Much of the reported work on HTAP has arisen in the context of "greenfield" systems, answering the question "if we could design a system for HTAP from scratch, what would it look like?" While there is great merit in such an approach, and a lot of valuable technology has been developed with it, we found ourselves facing a different challenge: one in which there is a great deal of transactional data already existing in several transactional systems, heavily queried by an existing federated engine that does not "own" the transactional systems, supporting both new and legacy applications that demand transparent fast queries and transactions from this combination. This paper reports on our design and experiences with F1 Lightning, a system we built and deployed to meet this challenge. We describe our design decisions, some details of our implementation, and our experience with the system in production for some of Google's most demanding applications.

A survey on hybrid transactional and analytical processing

“…As a result, visibility delay is one of the most popular metrics evaluated by the previous papers [18, 34, 39, 78, 97, 111, Fig. 4 This diagram compares two isolation evaluation methods 150,175]. We show an example of visibility delay in Fig.…”

“…This architecture represents the road to building an HTAP database on the shoulder of existing row and column stores. Typical implementations include TiDB [78], Heatweave [123], and F1 Lightning [175], IBM BLU [137], and ByteHTAP [39].…”

“…Driven by the increasing connectivity between data generation (e.g., Online Transaction Processing, for short, OLTP) and data consumption (e.g., Online Analytical Processing, for short, OLAP), real-time analytics based on new data has attracted much interest from academia and industry [18,78,111,168,175]. The fundamental motivation behind this trend is that much information is most valuable when it first appears and is usually time-decayed [8,44,95].…”

The Distributed HTAP Architecture for Real-Time Analysis and Updating of Point Cloud Data

“…In subsequent HTAP research, Huang et al proposed the HTAP database TiDB, constructed through a solution that separately manages data for OLTP and data for OLAP using replication technology based on the Raft algorithm [6]. Yang et al further proposed the HTAP database F1 Lightning, constructed through a solution that merges the change logs (delta) of tables into one consistent view [7]. These studies were significant in that they built environments in which OLTP queries and OLAP queries did not interfere with each other in their contributions to HTAP workloads.…”

HTAP With Reactive Streaming ETL

“…Running transactional and analytical workloads efficiently on the same dataset is an open problem which attracts research and commercial interests (Yang et al, 2020). Referred to as Hybrid Transactional and Analytical Processing (HTAP), several approaches are proposed to tackle the ostensibly conflicting demands of preserving the performance of transactional workloads whilst at the same time running analytical queries efficiently on fresh data to facilitate time-critical business decisions.…”