Common Data Format Archiving of Large-Scale Intelligent Transportation Systems Data for Efficient Storage, Retrieval, and Portability

Abstract

Intelligent transportation system (ITS) sensor networks, such as road weather information and traffic sensor networks, typically generate enormous amounts of data. As a result, archiving, retrieval, and exchange of ITS sensor data for planning and performance analysis are becoming increasingly difficult. An efficient ITS archiving system that is compact and exchangeable and allows efficient and fast retrieval of large amounts of data is essential. A proposal is made for a system that can meet the present and future archiving needs of large-scale ITS data. This system is referred to as common data format (CDF) and was developed by the National Space Science Data Center for archiving, exchange, and management of large-scale scientific array data. CDF is an open system that is free and portable and includes self-describing data abstraction. Archiving traffic data by using CDF is demonstrated, and its archival and retrieval performance is presented for the Minnesota Department of Transportation–s 30-s traffic data collected from about 4,000 loop detectors around Twin Cities freeways. For comparison of the archiving performance, the same data were archived by using a commercially available relational database, which was evaluated for its archival and retrieval performance. This result is presented, along with reasons that CDF is a good fit for large-scale ITS data archiving, retrieval, and exchange of data.

USID and Pycroscopy – Open Source Frameworks for Storing and Analyzing Imaging and Spectroscopy Data

“…By its definition, USID is an abstract data model and not accessible or tangible without its implementation into a file format. Therefore, we searched for file formats that would meet the Among the many file formats we tested including MathWorks' .MAT files 58 , XML 66 OME-TIFF files 52 , home-made binary files, common data format (CDF) 67 , net common data format (NetCDF) 68 , hierarchical data format (HDF5) 69 , Zarr 70 , etc., HDF5 was the only file format that satisfied all our requirements. In fact, HDF5 supports most popular data models such as NetCDF, NeXus 48 , Coherent X-Ray Imaging (CXI) data 71 , DREAM.3D model 55 Keeping the metadata and data within a single file, h5USID mitigates the need for a traditional relational database for tracking data processing via frameworks such as Sumatra 72 , ActivePapers 73 , or Madagascar 74 , though they can be added on if desired.…”

“…2(c 58 , R, python 59 , C++ 60,61 , C# 62 , Java 63 and perl 64 , Igor Pro, etc. ; and finally (i) compatible with cloud and high-performance computing by supporting parallel read and write 65 Among the many file formats we tested including MathWorks' .MAT files 58 , XML 66 OME-TIFF files 52 , home-made binary files, common data format (CDF) 67 , net common data format (NetCDF) 68 , hierarchical data format (HDF5) 69 , Zarr 70 , etc., HDF5 was the only file format that satisfied all our requirements. In fact, HDF5 supports most popular data models such as NetCDF, NeXus 48 , Coherent X-Ray Imaging (CXI) data 71 , DREAM.3D model 55 Traceability and reproducibility of the experiments as well as the data processing routines in h5USID are captured through comprehensive and consistent metadata stored as attributes, in addition to links created between the data source and the analysis results within the HDF5 file.…”