TASTI: Semantic Indexes for Machine Learning-based Queries over Unstructured Data

Kang, Daniel; Guibas, John; Bailis, Peter; Hashimoto, Tatsunori; Zaharia, Matei

doi:10.1145/3514221.3517897

Cited by 14 publications

(6 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The text dataset is publicly available on Kaggle and contains 3M+ tweets between users and customer support Twitter accounts [23]. For each video dataset we generated proxy scores from TASTI embeddings that we created with a pre-trained ResNet-18 model [19,29]. Our oracle labels were computed using a Mask R-CNN model [18].…”

Section: Methodsmentioning

confidence: 99%

“…The planner wishes to know the per-frame average number of cars that pass through an intersection. The planner could submit the following continuous query to InQuest: where count(car) is computed using an objection detection DNN and proxy_count_cars could be computed via an embedding index for unstructured data [29]. In this setting, proxy_count_cars returns an estimate of the car count for every frame.…”

Section: Examplesmentioning

confidence: 99%

See 1 more Smart Citation

Accelerating Aggregation Queries on Unstructured Streams of Data

Russo,

Hashimoto,

Kang

et al. 2023

Proc. VLDB Endow.

Self Cite

View full text Add to dashboard Cite

Analysts and scientists are interested in querying streams of video, audio, and text to extract quantitative insights. For example, an urban planner may wish to measure congestion by querying the live feed from a traffic camera. Prior work has used deep neural networks (DNNs) to answer such queries in the batch setting. However, much of this work is not suited for the streaming setting because it requires access to the entire dataset before a query can be submitted or is specific to video. Thus, to the best of our knowledge, no prior work addresses the problem of efficiently answering queries over multiple modalities of streams. In this work we propose InQuest, a system for accelerating aggregation queries on unstructured streams of data with statistical guarantees on query accuracy. InQuest leverages inexpensive approximation models ("proxies") and sampling techniques to limit the execution of an expensive high-precision model (an "oracle") to a subset of the stream. It then uses the oracle predictions to compute an approximate query answer in real-time. We theoretically analyzed InQuest and show that the expected error of its query estimates converges on stationary streams at a rate inversely proportional to the oracle budget. We evaluated our algorithm on six real-world video and text datasets and show that InQuest achieves the same root mean squared error (RMSE) as two streaming baselines with up to 5.0x fewer oracle invocations. We further show that InQuest can achieve up to 1.9x lower RMSE at a fixed number of oracle invocations than a state-of-the-art batch setting algorithm.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Examplesmentioning

confidence: 99%

Accelerating Aggregation Queries on Unstructured Streams of Data

Russo,

Hashimoto,

Kang

et al. 2023

Proc. VLDB Endow.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Others help ensure reproducibility while iterating on different ideas [30,39,90]. With regards to validating changes in production systems, some researchers have studied CI (Continuous Integration) for ML and proposed preliminary solutions-for example, ease.ml/ci streamlines data management and proposes unit tests for overfitting [1], and some papers introduce tools to perform validation and monitoring in production ML pipelines [8,38,86]. Our work is complementary to existing literature on this tooling; we do not explicitly ask interviewees questions about tools, nor do we propose any tools.…”

Section: Software Engineering For MLmentioning

confidence: 99%

"We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning

Shankar,

Garcia,

Hellerstein

et al. 2024

Proc. ACM Hum.-Comput. Interact.

View full text Add to dashboard Cite

Organizations rely on machine learning engineers (MLEs) to deploy models and maintain ML pipelines in production. Due to models' extensive reliance on fresh data, the operationalization of machine learning, or MLOps, requires MLEs to have proficiency in data science and engineering. When considered holistically, the job seems staggering---how do MLEs do MLOps, and what are their unaddressed challenges? To address these questions, we conducted semi-structured ethnographic interviews with 18 MLEs working on various applications, including chatbots, autonomous vehicles, and finance. We find that MLEs engage in a workflow of (i) data preparation, (ii) experimentation, (iii) evaluation throughout a multi-staged deployment, and (iv) continual monitoring and response. Throughout this workflow, MLEs collaborate extensively with data scientists, product stakeholders, and one another, supplementing routine verbal exchanges with communication tools ranging from Slack to organization-wide ticketing and reporting systems. We introduce the 3Vs of MLOps: velocity, visibility, and versioning --- three virtues of successful ML deployments that MLEs learn to balance and grow as they mature. Finally, we discuss design implications and opportunities for future work.

show abstract

“…To do this, prior work has developed many different learning-to-approximate techniques that use lightweight models to approximate the reference model, using the reference model as supervision. This includes work in [12], [11], and more recently [13]. This is quite different than SeeSaw, where the goal is to optimize search queries only, without a reference model at all (i.e., SeeSaw finds new classes of objects for which no reference model is available.).…”

Section: Specifying Queries Over Image Datasetsmentioning

confidence: 99%

SeeSaw: interactive ad-hoc search over image databases

Moll¹,

Favela²,

Madden³

et al. 2022

Preprint

View full text Add to dashboard Cite

As image datasets become ubiquitous, the problem of ad-hoc searches over image data is increasingly important. In particular, many tasks, such as constructing datasets for training and testing object detectors, require finding ad-hoc objects or scenes within large image datasets. Existing approaches for searching image datasets rely on rigid categories or assume fully accurate models trained on the data are available ahead of time. In contrast, SeeSaw is a system for interactive ad-hoc searches in image datasets that does not assume a pre-defined set of categories in advance. SeeSaw users can start a search using text, and then provide feedback to SeeSaw in the form of box annotations on previous results. Using this input, SeeSaw refines the results it returns to help users locate images of interest in their data. Behind the scenes, SeeSaw uses several optimizations to transform the user's feedback into better results. We evaluate SeeSaw against a state of the art baseline that does not take advantage of user feedback and find SeeSaw can increase search quality metrics up 4x for the harder search queries, and significantly on almost all queries, even those where the baseline performs well.

show abstract

TASTI: Semantic Indexes for Machine Learning-based Queries over Unstructured Data

Cited by 14 publications

References 20 publications

Accelerating Aggregation Queries on Unstructured Streams of Data

Accelerating Aggregation Queries on Unstructured Streams of Data

"We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning

SeeSaw: interactive ad-hoc search over image databases

Contact Info

Product

Resources

About