NVST Data Archiving System Based on Fastbit Nosql Database

Liu, Yingbo; Wei, Feng; Ji, Kaifan; Deng, Hui; Dai, Wei; Liang, Bo

doi:10.5303/jkas.2014.47.3.115

Cited by 9 publications

(5 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Due to query-intensive nature of our experiments, this kind of indexing helps improve query performance. Liu et al (2014) reported good performance in index creation, query and storage space of this algorithm by comparing with a typical relational database (e.g., MySQL or PostgreSQL). The reader is referred to Liu et al (2014) for details of quantitative comparison between both database systems using high-resolution solar observations.…”

Section: Appendix 1 Database Systemmentioning

confidence: 92%

New Photometric Pipeline to Explore Temporal and Spatial Variability with KMTNet DEEP-South Observations

Chang¹,

Byun²,

Shin³

et al. 2018

Preprint

View full text Add to dashboard Cite

The DEEP-South photometric census of small Solar System bodies produces massive timeseries data of variable, transient or moving objects as a by-product. To fully investigate unexplored variable phenomena, we present an application of multi-aperture photometry and FastBit indexing techniques for faster access to a portion of the DEEP-South year-one data. Our new pipeline is designed to perform automated point source detection, robust high-precision photometry and calibration of non-crowded fields which have overlap with previously surveyed areas. In this paper, we show some examples of catalogbased variability searches to find new variable stars and to recover targeted asteroids. We discover 21 new periodic variables with period ranging between 0.1 and 31 days, including four eclipsing binary systems (detached, over-contact, and ellipsoidal variables), one white dwarf/M dwarf pair candidate, and rotating variable stars. We also recover astrometry (< ±1-2 arcsec level accuracy) and photometry of two targeted near-earth asteroids, 2006 DZ169 and 1996 SK, along with the small-(∼0.12 mag) and relatively largeamplitude (∼0.5 mag) variations of their dominant rotational signals in R-band.

show abstract

Section: Appendix 1 Database Systemmentioning

confidence: 92%

New Photometric Pipeline to Explore Temporal and Spatial Variability with KMTNet DEEP-South Observations

Chang¹,

Byun²,

Shin³

et al. 2018

Preprint

View full text Add to dashboard Cite

show abstract

“…At the end of the run, the DAQ updates this document with information such as the length of the run and the number of data files created. A full export of the DAQ settings at the start and at the end of the run is also attached to the run document 5 .…”

Section: Daq Integrationmentioning

confidence: 99%

“…The scale and the duration of the projects makes database approaches necessary for managing information about the detector components and the datasets. Projects have, for example, taken a database approach to managing the quality of detector components and their status [1,2,3], and to keeping track of datasets [4,5] and detector conditions [6]. In this work, we present a unified database approach to keeping track of detector hardware, response parameters, environment information, and datasets for the DEAP-3600 Dark Matter detector [7,8,9].…”

Section: Introductionmentioning

confidence: 99%

Database support of detector operation and data analysis in the DEAP-3600 Dark Matter experiment

Pollmann

Smith

2019

Eur. Phys. J. C

View full text Add to dashboard Cite

The DEAP-3600 detector searches for dark matter interactions on a 3.3 tonne liquid argon target. Over nearly a decade, from start of detector construction through the end of the data analysis phase, well over 200 scientists will have contributed to the project. The DEAP-3600 detector will amass in excess of 900 TB of data representing more than 10 10 particle interactions, a few of which could be from dark matter. At the same time, metadata exceeding 80 GB will be generated. This metadata is crucial for organizing and interpreting the dark matter search data and contains both structured and unstructured information.The scale of the data collected, the important role of metadata in interpreting it, the number of people involved, and the long lifetime of the project necessitate an industrialized approach to metadata management.We describe how the CouchDB and the Post-greSQL database systems were integrated into the DEAP detector operation and analysis workflows. This integration provides unified, distributed access to both structured (PostgreSQL) and unstructured (CouchDB) metadata at runtime of the data analysis software. It also supports operational and reporting requirements.

show abstract

“…It utilizes a compressed bitmap index, encoding, and binning (Wu 2005;Wu et al 2009). A new data archiving system for the NVST based on the Fastbit NoSQL database has been implemented, where the average number of records generated by NVST in one year is approximately 10 to 12 million (Liu et al 2014). A Distributed Data Storage System (DDSS) for real-time observational data storage based on the Lustre distributed file system was implemented on the NVST (Liu et al 2015).…”

Section: Related Workmentioning

confidence: 99%

High-performance Negative Database for Massive Data Management System of The Mingantu Spectral Radioheliograph

Shi

Wei

Deng

et al. 2017

PASP

Self Cite

View full text Add to dashboard Cite

As a dedicated synthetic aperture radio interferometer, the MingantU SpEctral Radioheliograph (MUSER), initially known as the Chinese Spectral RadioHeliograph (CSRH), has entered the stage of routine observation. More than 23 million data records per day need to be effectively managed to provide high performance data query and retrieval for scientific data reduction. In light of these massive amounts of data generated by the MUSER, in this paper, a novel data management technique called the negative database (ND) is proposed and used to implement a data management system for the MUSER. Based on the key-value database, the ND technique makes complete utilization of the complement set of observational data to derive the requisite information. Experimental results showed that the proposed ND can significantly reduce storage volume in comparison with a relational database management system (RDBMS). Even when considering the time needed to derive records that were absent, its overall performance, including querying and deriving the data of the ND, is comparable with that of an RDBMS. The ND technique effectively solves the problem of massive data storage for the MUSER, and is a valuable reference for the massive data management required in next-generation telescopes.

show abstract

NVST Data Archiving System Based on Fastbit Nosql Database

Cited by 9 publications

References 24 publications

New Photometric Pipeline to Explore Temporal and Spatial Variability with KMTNet DEEP-South Observations

New Photometric Pipeline to Explore Temporal and Spatial Variability with KMTNet DEEP-South Observations

Database support of detector operation and data analysis in the DEAP-3600 Dark Matter experiment

High-performance Negative Database for Massive Data Management System of The Mingantu Spectral Radioheliograph

Contact Info

Product

Resources

About