Community-based benchmarking improves spike rate inference from two-photon calcium imaging data

Abstract

In recent years, two-photon calcium imaging has become a standard tool to probe the function of neural circuits and to study computations in neuronal populations 1, 2 . However, the acquired signal is only an indirect measurement of neural activity due to the comparatively slow dynamics of fluorescent calcium indicators 3 . Different algorithms for estimating spike trains from noisy calcium measurements have been proposed in the past 4-8 , but it is an open question how far performance can be improved. Here, we report the results of the spikefinder 2 . CC-BY-NC 4.0 International license peer-reviewed) is the author/funder. It is made available under a The copyright holder for this preprint (which was not . http://dx.doi.org/10.1101/177956 doi: bioRxiv preprint first posted online Aug. 18, 2017; challenge, launched to catalyze the development of new spike inference algorithms through crowd-sourcing. We present ten of the submitted algorithms which show improved performance compared to previously evaluated methods. Interestingly, the top-performing algorithms are based on a wide range of principles from deep neural networks to generative models, yet provide highly correlated estimates of the neural activity. The competition shows that benchmark challenges can drive algorithmic developments in neuroscience.We organized the spikefinder challenge to crowd-source the problem of developing algorithms for inferring spike rates from calcium signals. Such challenges have been used successfully in machine learning and computer vision to drive algorithmic developments 9 . We used a benchmark data set consisting of 92 recordings from 73 neurons 6 , acquired in the primary visual cortex and the retina of mice using different calcium indicators (OGB-1 and GCaMP6s), different scanning methods, and different sampling rates. For this data, calcium imaging has been performed simultaneously with electrophysiological recordings allowing to evaluate the performance of spike inference algorithms on ground truth data. Importantly, all data has been acquired at a zoom factor typically used during population imaging experiments, ensuring that all benchmark results reflect performance under the typical use-case conditions.For the challenge, we split the data into a training and a test set (see Table 1). For the training data, we made both the calcium and the spike traces publicly available, but kept the spike traces secret for the test data. Additionally, the publicly available data sets provided by the GENIE project 10 were available as training data. This allowed participants to adjust their models on the 3 . CC-BY-NC 4.0 International license peer-reviewed) is the author/funder. It is made available under a The copyright holder for this preprint (which was not . http://dx.doi.org/10.1101/177956 doi: bioRxiv preprint first posted online Aug. 18, 2017; training data set, while avoiding overfitting to the specific benchmark data set. Participants could upload predictions for the spike rate generated by their algorithm on a dedicate...

CalmAn: An open source tool for scalable Calcium Imaging data Analysis

“…Neural activity deconvolution is performed using sparse non-negative deconvolution ( Vogelstein et al, 2010; Pnevmatikakis et al, 2016 ) and implemented with both the near-online OASIS algorithm ( Friedrich et al, 2017b ) and an efficient convex optimization framework ( Pnevmatikakis et al, 2016 ). The algorithm is competitive to the state of the art according to recent benchmarking studies ( Berens et al, 2017 ). Prior to deconvolution, the traces are detrended to remove non-stationary effects, e.g., photo-bleaching.…”

“…Unsupervised methods can be either deterministic, such as sparse non-negative deconvolution ( Vogelstein et al, 2010; Pnevmatikakis et al, 2016) that give a single estimate of the deconvolved neural activity, or probabilistic, that aim to also characterize the uncertainty around these estimates (e.g., ( Pnevmatikakis et al, 2013; Deneuxet al, 2016 )). A recent community benchmarking effort ( Berens et al, 2017 ) characterizes the similarities and differences of various available methods.…”

CosMIC: A Consistent Metric for Spike Inference from Calcium Imaging

“…In light of this problem, a recent benchmarking study tested a range of algorithms on a wide array of imaging data (Berens et al, 2017). Although informative, the study, which relied heavily on the spike train correlation to assess algorithm performance, may not provide the full picture.…”

“…In practise, however, it is rare for a spike detection algorithm to produce an estimate that is negatively correlated with the ground truth (Berens et al, 2017). Moreover, an estimate with maximal negative correlation is equally as informative as one with maximal positive correlation.…”

CosMIC: A Consistent Metric for Spike Inference from Calcium Imaging

“…The STC takes values in the range [−1, 1]. In practice, however, it is rare for a spike detection algorithm to produce an estimate that is negatively correlated with the ground truth (Berens et al, 2017). Moreover, an estimate with maximal negative correlation is equally as informative as one with maximal positive correlation.…”