The Brain Observatory Storage Service and Database (BossDB): A Cloud-Native Approach for Petascale Neuroscience Discovery

Hider, Robert; Kleissas, Dean; Gion, Timothy; Xenes, Daniel; Matelsky, Jordan; Pryor, Derek; Rodríguez, Luis M.; Johnson, Erik C.; Gray-Roncal, William; Wester, Brock A.

doi:10.3389/fninf.2022.828787

Cited by 25 publications

(21 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We demonstrate the capabilities of the NeuVue system with publicly-released EM imagery of the mouse visual cortex from both the "Minnie" (MICrONS Consortium et al 2021) and "Pinky" datasets (Dorkenwald et al 2019) generated on the MICrONS Program. For these datasets, the associated finealigned imagery, static segmentation, and 3D meshes are stored in BossDB as precompute data (Hider et al 2022, Maitin-Shepard 2020.…”

Section: Connectomics Data Sourcesmentioning

confidence: 99%

NeuVue: A Framework and Workflows for High-Throughput Electron Microscopy Connectomics Proofreading

Xenes

Kitchell

Rivlin

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

1.AbstractNeuVue is a software platform created for large-scale proofreading of machine segmentation and neural circuit reconstruction in high-resolution electron microscopy connectomics datasets. The NeuVue platform provides a robust web-based interface for proofreaders to collaboratively view, annotate, and edit segmentation and connectivity data. A backend queuing service organizes proofreader tasks into purpose-driven task types and increases proofreader throughput by limiting proofreader actions to simple, atomic operations. A collection of analytical dashboards, data visualization tools, and Application Program Interface (API) capabilities provide stakeholders real-time access to proofreading progress at an individual proofreader level as well as insights on task generation priorities. NeuVue is agnostic to the underlying data being proofread and improves upon the traditional proofreader experience through quality-of-life features that streamline complex editing operations such as splitting and merging objects in dense nanoscale segmentation.NeuVue heavily leverages cloud resources to enable proofreaders to simultaneously access and edit data on the platform. Production-quality features such as load-balancing, auto-scaling, and pre-deployment testing are all integrated into the platform’s cloud architecture. Additionally, NeuVue is powered by well-supported open-source connectomics tools from the community such as Neuroglancer, PyChunkedGraph, and Connectomics Annotation Versioning Engine (CAVE). The modular design of NeuVue facilitates easy integration and adoption of useful community tools to allow proofreaders to take advantage of the latest improvements in data visualization, processing, and analysis.We demonstrate our framework through proofreading of the mouse visual cortex data generated on the IARPA MICrONS Project. This effort has yielded over 40,000 proofreader edits across the 2 petavoxels of “Minnie” neuroimaging data. 44 unique proofreaders of various skill levels have logged a cumulative 3,740 proofreading hours, and we have been able to validate the improved connectivity of thousands of neurons in the volume. With sustained development on the platform, new integrated error detection and error correction capabilities, and continuous improvements to the proofreader model, we believe that the NeuVue framework can enable high-throughput proofreading for large-scale connectomics datasets of the future.

show abstract

Section: Connectomics Data Sourcesmentioning

confidence: 99%

NeuVue: A Framework and Workflows for High-Throughput Electron Microscopy Connectomics Proofreading

Xenes

Kitchell

Rivlin

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The OCP cluster also supports simple metadata queries and is widely used as the data backend in different connectomics applications [VPF * 18]. The brain observatory storage service and database (BossDB) [HKG * 22] is a newer generation cloud‐based data access system for petascale Neuroscience imaging datasets. Their spatial database implements a storage hierarchy that manages data transfer between affordable, persistent data storage and an in‐memory data storage like Redis.…”

Section: Visual Exploration and Analysismentioning

confidence: 99%

A Survey of Visualization and Analysis in High‐Resolution Connectomics

Beyer

Troidl

Boorboor³

et al. 2022

Computer Graphics Forum

View full text Add to dashboard Cite

The field of connectomics aims to reconstruct the wiring diagram of Neurons and synapses to enable new insights into the workings of the brain. Reconstructing and analyzing the Neuronal connectivity, however, relies on many individual steps, starting from high‐resolution data acquisition to automated segmentation, proofreading, interactive data exploration, and circuit analysis. All of these steps have to handle large and complex datasets and rely on or benefit from integrated visualization methods. In this state‐of‐the‐art report, we describe visualization methods that can be applied throughout the connectomics pipeline, from data acquisition to circuit analysis. We first define the different steps of the pipeline and focus on how visualization is currently integrated into these steps. We also survey open science initiatives in connectomics, including usable open‐source tools and publicly available datasets. Finally, we discuss open challenges and possible future directions of this exciting research field.

show abstract

“…MossDB was designed to be used alongside spatial datastores like BossDB (4) or CloudVolume (5), and graph databases like Neo4j (6) or Grand (discussed in further depth below). Because the datasets in question can span hundreds to thousands of terabytes, it is not realistic to store copies of even small subsets of the data.…”

Section: Contributionsmentioning

confidence: 99%

Scalable graph analysis tools for the connectomics community

Matelsky

Johnson

Wester

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Neuroscientists now have the opportunity to analyze synaptic resolution connectomes that are larger than the memory on single consumer workstations. As dataset size and tissue diversity have grown, there is increasing interest in conducting comparative connectomics research, including rapidly querying and searching for recurring patterns of connectivity across brain regions and species. There is also a demand for algorithm reuse --- applying methods developed for one dataset to another volume. A key technological hurdle is enabling researchers to efficiently and effectively query these diverse datasets, especially as the raw image volumes grow beyond terabyte sizes. Existing community tools can perform such queries and analysis on smaller scale datasets, which can fit locally in memory, but the path to scaling remains unclear. Existing solutions such as neuPrint or FlyBrainLab enable these queries for specific datasets, but there remains a need to generalize algorithms and standards across datasets. To overcome this challenge, we present a software framework for comparative connectomics and graph discovery to make connectomes easy to analyze, even when larger-than-RAM, and even when stored in disparate datastores. This software suite includes visualization tools, a web portal, a connectivity and annotation query engine, and the ability to interface with a variety of data sources and community tools from the neuroscience community. These tools include MossDB (an immutable datastore for metadata and rich annotations); Grand (for prototyping larger-than-RAM graphs); GrandIso-Cloud (for querying existing graphs that exceed the capabilities of a single workstation); and Motif Studio (for enabling the public to query across connectomes). These tools interface with existing frameworks such as neuPrint, graph databases such as Neo4j, and standard data analysis tools such as Pandas or NetworkX. Together, these tools enable tool and algorithm reuse, standardization, and neuroscience discovery.

show abstract

The Brain Observatory Storage Service and Database (BossDB): A Cloud-Native Approach for Petascale Neuroscience Discovery

Cited by 25 publications

References 25 publications

NeuVue: A Framework and Workflows for High-Throughput Electron Microscopy Connectomics Proofreading

NeuVue: A Framework and Workflows for High-Throughput Electron Microscopy Connectomics Proofreading

A Survey of Visualization and Analysis in High‐Resolution Connectomics

Scalable graph analysis tools for the connectomics community

Contact Info

Product

Resources

About