Learning-based query optimization for multi-probe approximate nearest neighbor search

Zhang, Pengcheng; Yao, Bin; Gao, Chao; Wu, Bin; He, Xiao; Li, Feifei; Lu, Yuan‐Fei; Zhan, Chaoqun; Tang, Feilong

doi:10.1007/s00778-022-00762-0

Cited by 4 publications

(3 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our method can work well with these strategies. For more details on data segmentation strategies, please refer to [23,75].…”

Section: Evaluation On Billion-scale Datamentioning

confidence: 99%

“…Some purpose-built vector data management systems, such as Vearch [41] and Milvus [29,66], store and search largescale vector data efficiently by organizing data with segments for industrial applications. There are also some distributed solutions, e.g., Pyramid [21], LANNS [23], LEQAT [75]. They focus on data segmentation strategies or query dispatching optimizations.…”

Section: O Related Workmentioning

confidence: 99%

“…Each segment operates with limited storage capacity and computing resources [4,29], and an independent medium-sized index is constructed on each segment for autonomous searching. Leveraging specific data fragmentation strategies and a query coordinator allows us to search only a few segments of a machine during vector query execution [21,29,75]. This poses an open problem of efficient and accurate HVSS within a data segment in vector databases [6,29].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Starling: An I/O-Efficient Disk-Resident Graph Index Framework for High-Dimensional Vector Similarity Search on Data Segment

Wang,

Xu,

et al. 2024

Proc. ACM Manag. Data

View full text Add to dashboard Cite

High-dimensional vector similarity search (HVSS) is gaining prominence as a powerful tool for various data science and AI applications. As vector data scales up, in-memory indexes pose a significant challenge due to the substantial increase in main memory requirements. A potential solution involves leveraging disk-based implementation, which stores and searches vector data on high-performance devices like NVMe SSDs. However, implementing HVSS for data segments proves to be intricate in vector databases where a single machine comprises multiple segments for system scalability. In this context, each segment operates with limited memory and disk space, necessitating a delicate balance between accuracy, efficiency, and space cost. Existing disk-based methods fall short as they do not holistically address all these requirements simultaneously. In this paper, we present Starling, an I/O-efficient disk-resident graph index framework that optimizes data layout and search strategy within the segment. It has two primary components: (1) a data layout incorporating an in-memory navigation graph and a reordered disk-based graph with enhanced locality, reducing the search path length and minimizing disk bandwidth wastage; and (2) a block search strategy designed to minimize costly disk I/O operations during vector query execution. Through extensive experiments, we validate the effectiveness, efficiency, and scalability of Starling. On a data segment with 2GB memory and 10GB disk capacity, Starling can accommodate up to 33 million vectors in 128 dimensions, offering HVSS with over 0.9 average precision and top-10 recall rate, and latency under 1 millisecond. The results showcase Starling's superior performance, exhibiting 43.9x higher throughput with 98% lower query latency compared to state-of-the-art methods while maintaining the same level of accuracy.

show abstract

“…Our method can work well with these strategies. For more details on data segmentation strategies, please refer to [23,75].…”

Section: Evaluation On Billion-scale Datamentioning

confidence: 99%