USID and Pycroscopy – Open Source Frameworks for Storing and Analyzing Imaging and Spectroscopy Data

Abstract

Materials science is undergoing profound changes due to advances in characterization instrumentation that have resulted in an explosion of data in terms of volume, velocity, variety and complexity. Harnessing these data for scientific research requires an evolution of the associated computing and data infrastructure, bridging scientific instrumentation with super-and cloudcomputing. Here, we describe Universal Spectroscopy and Imaging Data (USID), a data model capable of representing data from most common instruments, modalities, dimensionalities, and sizes. We pair this schema with the hierarchical data file format (HDF5) to maximize compatibility, exchangeability, traceability, and reproducibility. We discuss a family of community-driven, open-source, and free python software packages for storing, processing and visualizing data. The first is pyUSID which provides the tools to read and write USID HDF5 files in addition to a scalable framework for parallelizing data analysis. The second is Pycroscopy, which provides algorithms for scientific analysis of nanoscale imaging and spectroscopy modalities and is built on top of pyUSID and USID. The instrument-agnostic nature of USID facilitates the development of analysis code independent of instrumentation and task in Pycroscopy which in turn can bring scientific communities together and break down barriers in the age of openscience. The interested reader is encouraged to be a part of this ongoing community-driven effort to collectively accelerate materials research and discovery through the realms of big data.

py4DSTEM: A Software Package for Four-Dimensional Scanning Transmission Electron Microscopy Data Analysis

“…Maximizing the impact this new generation of STEM experiments will have on structural characterization research now requires that the computer processing methods which enable the various 4D-STEM characterization modalities are accessible to a broad and diverse segment of the scientific community. Fortunately, a new generation of open-source tools for electron scattering experiments is presently on the rise, such as hyperspy, pyXem, liberTEM, pycroscopy, ncempy, and others (de la Peña et al, 2019; Nord et al, 2019; Johnstone et al, 2019; Somnath et al, 2019; Clausen et al, 2020).…”

Materials science in the artificial intelligence age: high-throughput library generation, machine learning, and a pathway from correlations to the underpinning physics

“…Ensuring that data formats within individual laboratories and organizations are open, documented and standardized requires much work, but pays off in terms of efficiency gains in the long term. Towards this aim, a subset of the authors has created the universal spectral imaging data model (USID 181 ), while the crystallography community is well-versed with the CIF format. 196 Logging the correct metadata with each experiment is critical, and lab notebooks can be digitized to enable searchability and indexing.…”

Fast Pixelated Detectors in Scanning Transmission Electron Microscopy. Part I: Data Acquisition, Live Processing, and Storage

“…Multiple processing classes (blue, v) are implemented for scanning and imaging modes. (de la Peña et al, 2018;EMD authors, 2019;Somnath et al, 2019). It can be both read and written in a number of programming languages, including MatLab, C++, Python, Java, R, and Gatan Digital Micrograph through a third party plugin (Niermann, 2016).…”

“…Several other packages exist, like rigidRegistration for doing rigid image registration of atomic resolution image stacks (Savitzky et al, 2018), and wrappers for doing STEM simulations, like PyPrismatic (Ophus, 2017). Other packages for processing data from fast pixelated STEM detectors include py4DSTEM (Savitzky et al, 2019), LiberTEM (Clausen et al, 2019), pycroscopy (Somnath et al, 2019), and fpd (fpd devs, 2015).…”