FlexQuery: An online query system for interactive remote visual data exploration at large scale

Zou, Hongbo; Schwan, Karsten; Sławińska, M.; Wolf, Matt; Eisenhauer, Greg; Zheng, Fang; Dayal, Jai; Logan, Jeremy; Liu, Qing; Klasky, Scott; Bode, Tanja; Clark, Michael; Kinsey, M.

doi:10.1109/cluster.2013.6702635

Cited by 16 publications

(14 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another set of efforts is focused on building a querying system directly on the raw scientific data files. Typical examples include FastQuery [5,3], an array-based querying library based on FastBit [13] and FlexQuery [17]. These efforts do not perform reorganization of data based on patterns.…”

Section: Related Workmentioning

confidence: 99%

SDS

Dong

Byna

2013

Proceedings of the 8th Parallel Data Storage Workshop

View full text Add to dashboard Cite

Large-scale scientific applications typically write their data to parallel file systems with organizations designed to achieve fast write speeds. Analysis tasks frequently read the data in a pattern that is different from the write pattern, and therefore experience poor I/O performance. In this paper, we introduce a prototype framework for bridging the performance gap between write and read stages of data access from parallel file systems. We call this framework Scientific Data Services, or SDS for short. This initial implementation of SDS focuses on reorganizing previously written files into data layouts that benefit read patterns, and transparently directs read calls to the reorganized data. SDS follows a client-server architecture. The SDS Server manages partial or full replicas of reorganized datasets and serves SDS Clients' requests for data. The current version of the SDS client library supports HDF5 programming interface for reading data. The client library intercepts HDF5 calls using the HDF5 Virtual Object Layer (VOL) and transparently redirects them to the reorganized data. The SDS client library also provides a querying interface for reading part of the data based on user-specified selective criteria. We describe the design and implementation of the SDS client-server architecture, and evaluate the response time of the SDS Server and the performance benefits of SDS.

show abstract

Section: Related Workmentioning

confidence: 99%

SDS

Dong

Byna

2013

Proceedings of the 8th Parallel Data Storage Workshop

View full text Add to dashboard Cite

show abstract

“…DIRAQ [23] is a parallel data encoding and reorganization technique that supports range queries while data is being produced by scientific simulations. FlexQuery [40] is another proposed system to perform range queries to support visualization of data while the data is being generated. Su et al [32] implemented user-defined subsetting and aggregation operations on NetCDF data.…”

Section: Related Workmentioning

confidence: 99%

Parallel data analysis directly on scientific file formats

Blanas

Byna

et al. 2014

Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data

View full text Add to dashboard Cite

Scientific experiments and large-scale simulations produce massive amounts of data. Many of these scientific datasets are arrays, and are stored in file formats such as HDF5 and NetCDF. Although scientific data management systems, such as SciDB, are designed to manipulate arrays, there are challenges in integrating these systems into existing analysis workflows. Major barriers include the expensive task of preparing and loading data before querying, and converting the final results to a format that is understood by the existing post-processing and visualization tools. As a consequence, integrating a data management system into an existing scientific data analysis workflow is time-consuming and requires extensive user involvement.In this paper, we present the design of a new scientific data analysis system that efficiently processes queries directly over data stored in the HDF5 file format. This design choice eliminates the tedious and error-prone data loading process, and makes the query results readily available to the next processing steps of the analysis workflow. Our design leverages the increasing main memory capacities found in supercomputers through bitmap indexing and in-memory query execution. In addition, query processing over the HDF5 data format can be effortlessly parallelized to utilize the ample concurrency available in large-scale supercomputers and modern parallel file systems. We evaluate the performance of our system on a large supercomputing system and experiment with both a synthetic dataset and a real cosmology observation dataset. Our system frequently outperforms the relational database system that the cosmology team currently uses, and is more than 10× faster than Hive when processing data in parallel. Overall, by eliminating the data loading step, our query processing system is more effective in supporting in situ scientific analysis workflows.

show abstract

“…Social media also allows the library to enter into the space of the user, rather than waiting for the user to come to it. The library then begins actively seeking out conversations and participation and is able to speak with people it may otherwise not reach [14] [39]. The information and feedback that users provide is the "single best tool" that public libraries have to ensure that they remain relevant [2].…”

Section: Related Workmentioning

confidence: 99%

Understanding Library User Engagement Strategies through Large-Scale Twitter Analysis

Zou

Chen

Dey

2015

2015 IEEE First International Conference on Big Data Computing Service and Applications

Self Cite

View full text Add to dashboard Cite

Public libraries are increasingly using social media to connect their users in an innovative way. Librarians make use of social media with the purpose of "being part of their communities", or promoting libraries' services and events. Social media has become a significant platform for libraries to create their own participatory services emphasizing engagement with users. However, there has been little empirical investigation into the success of social media use by libraries. In this paper, we study the role of a recently popular social media, Twitter, in engaging users with a focus on public libraries. We use topicmodeling techniques to classify the library user engagement strategies into four categories -literature exhibits, engaging topic, community building, and library showcasing. These four engagement strategies are re-examined with the tweets collected from ten public libraries over three months. The tweets topic distribution of every library is discussed in the paper. Finally, the impacts of every strategy on user engagement have been evaluated by users feedback on every tweet. Through the data mining of public libraries' tweets, we aim to explore how user engagement strategies are used by the libraries on Twitter and suggest the best practices for libraries on social media initiatives to engage their users effectively.

show abstract

FlexQuery: An online query system for interactive remote visual data exploration at large scale

Cited by 16 publications

References 23 publications

SDS

SDS

Parallel data analysis directly on scientific file formats

Understanding Library User Engagement Strategies through Large-Scale Twitter Analysis

Contact Info

Product

Resources

About