mzMLb: A Future-Proof Raw Mass Spectrometry Data Format Based on Standards-Compliant mzML and Optimized for Speed and Storage Requirements

Abstract

With ever-increasing amounts of data produced by mass spectrometry (MS) proteomics and metabolomics, and the sheer volume of samples now analyzed, the need for a common open format possessing both file size efficiency and faster read/write speeds has become paramount to drive the next generation of data analysis pipelines. The Proteomics Standards Initiative (PSI) has established a clear and precise XML representation for data interchange, mzML, receiving substantial uptake; nevertheless, storage and file access efficiency has not been the main focus. We propose an HDF5 file format 'mzMLb' that is optimised for both read/write speed and storage of the raw mass spectrometry data. We provide extensive validation of write speed, random read speed and storage size, demonstrating a flexible format that with or without compression is faster than all existing approaches in virtually all cases, while with compression, is comparable in size to proprietary vendor file formats. Since our approach uniquely preserves the XML encoding of the metadata, the format implicitly supports future versions of mzML and is straightforward to implement: mzMLb's design adheres to both HDF5 and NetCDF4 standard implementations, which allows it to be easily utilised by third parties due to their widespread programming language support. A reference implementation within the established ProteoWizard toolkit is provided. .

MSCAT: A Machine Learning Assisted Catalog of Metabolomics Software Tools

“…However, the file formats used in metabolomics do not provide a sufficient specification for this. The recent development of newer, more metadatarich file formats [48][49][50] for metabolomics tells us there is an increasingly urgent need to standardize the knowledge representation and metadata elements in this community to achieve better interoperability and information exchange. There is a similar problem being worked on in the healthcare space [51], and a similar effort is likely needed when dealing with large, diverse datasets as in metabolomics and multi-omics.…”

Spatially Resolved Mass Spectrometry at the Single Cell: Recent Innovations in Proteomics and Metabolomics

“…The universal datafile format developed for the MSI community, imzML, allows for data export into multiple image processing software, but it has been shown to be 3–4 times slower in write speed when compared to the HDF5 format . There are also several other research groups attempting to address this issue by exploring file format optimization for MS files. , However, these alternative formats have yet to gain traction with the broader community. Computational power continues to improve on basic lab computer workspaces, and cloud-based imaging processing (e.g., through Amazon Web Services) are becoming increasingly utilized for processing imaging data sets, , and as such, we anticipate these will be the key to handling such complicated data over the next decade.…”

Chemputation and the Standardization of Chemical Informatics

“… 124 For other analytical data types, no such requirement exists but universal data standards for other methods, the quality of which can be validated, would be beneficial to the whole community. FIDs from NMR spectrometers, 125 CSV files from spectrophotometers, mzML files from mass spectrometers, 126 and similar data from GC/HPLC systems should all be a minimum requirement for publication of results, which rely upon this data.…”