CosMIC: A Consistent Metric for Spike Inference from Calcium Imaging

Reynolds, Stephanie; Schultz, Simon R.; Dragotti, Pier Luigi

doi:10.1101/238592

Cited by 4 publications

(3 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The best-fit parameters depend strongly on how we evaluate the match between true and inferred spike trains. The Pearson correlation coefficient between the true and inferred spike train is a common choice (Brown et al, 2004; Paiva et al, 2010; Theis et al, 2016; Reynolds et al, 2018; Berens et al, 2018), typically with both trains convolved with a Gaussian kernel to allow for timing errors. However, we find that choosing parameters to maximise the correlation coefficient can create notable errors.…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

On the use of calcium deconvolution algorithms in practical contexts

Evans

Petersen

Humphries

2019

Preprint

View full text Add to dashboard Cite

1Calcium imaging is a powerful tool for capturing the simultaneous activity of large 2 populations of neurons. Here we determine the extent to which our inferences of neu-3 ral population activity, correlations, and coding depend on our choice of whether and 4 how we deconvolve the calcium time-series into spike-driven events. To this end, we 5 use a range of deconvolution algorithms to create nine versions of the same calcium 6 imaging data obtained from barrel cortex during a pole-detection task. Seeking suit-7 able values for the deconvolution algorithms' parameters, we optimise them against 8 ground-truth data, and find those parameters both vary by up to two orders of mag-9 nitude between neurons and are sensitive to small changes in their values. Applied to 10 the barrel cortex data, we show that a substantial fraction of the processing methods 11 fail to recover simple features of population activity in barrel cortex already estab-12 lished by electrophysiological recordings. Raw calcium time-series contain an order of 13 magnitude more neurons tuned to features of the pole task; yet there is also qualitative 14 disagreement between deconvolution methods on which neurons are tuned to the task. 15 Finally, we show that raw and processed calcium time-series qualitatively disagree on 16 the structure of correlations within the population and the dimensionality of its joint 17 activity. Collectively, our results show that properties of neural activity, correlations, 18 and coding inferred from calcium imaging are highly sensitive to the choice of if and 19 how spike-evoked events are recovered. We suggest that quantitative results obtained 20 from population calcium-imaging be verified across multiple processed forms of the 21 calcium time-series. 22 28 Pnevmatikakis et al., 2016; Friedrich et al., 2017; Keemink et al., 2018; Giovannucci et al., 29 2019). As somatic calcium is proportional to the release of spikes, so we wish to use these 30 fluorescence time-series as a proxy for spiking activity in large, identified populations of 31 neurons. But raw calcium fluorescence is slow on the time-scale of spikes, nonlinearly 32 related to spiking, and contains noise from a range of sources. 33 These issues have inspired a wide range of deconvolution algorithms (Theis et al., 2016; 34 Berens et al., 2018; Stringer and Pachitariu, 2018), which attempt to turn raw somatic cal-35 1 cium into something more closely approximating spikes. Deconvolution algorithms them-36 selves range in complexity from simple deconvolution with a fixed kernel of the calcium 37 response (Yaksi and Friedrich, 2006), through detecting spike-evoked calcium events (Jew-38 ell and Witten, 2018; Pachitariu et al., 2016), to directly inferring spike times (Vogelstein 39 et al., 2010; Lütcke et al., 2013; Deneux et al., 2016). This continuum of options raises 40 the further question of the extent to which we should process the raw calcium signals. We 41 address here the question facing any systems neuroscientist using...

show abstract

Section: Resultsmentioning

confidence: 99%

“…By contrast, the Error Rate metric (Deneux et al, 2016; Victor and Purpura, 1996) resulted in excellent recovery of ground-truth spike trains. Other recently developed methods for comparing spike-trains based on information theory (Theis et al, 2016) or fuzzy set theory (Reynolds et al, 2018), may also be appropriate.…”

Section: Discussionmentioning

confidence: 99%

On the use of calcium deconvolution algorithms in practical contexts

Evans

Petersen

Humphries

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…F 1 -scores combine false positives and negatives 11 but are difficult to compare across datasets when the baseline spike rates vary (which is the case for our database). Other metrics try to combine the strengths of the correlation measure with a sensitivity to the correct number of spikes 78 but are less intuitive.…”

Section: Methodsmentioning

confidence: 99%

Database and deep learning toolbox for noise-optimized, generalized spike inference from calcium imaging

Rupprecht

Carta

Hoffmann

et al. 2020

Preprint

View full text Add to dashboard Cite

Calcium imaging is a key method to record patterns of neuronal activity across populations of identified neurons. Inference of temporal patterns of action potentials (‘spikes’) from calcium signals is, however, challenging and often limited by the scarcity of ground truth data containing simultaneous measurements of action potentials and calcium signals. To overcome this problem, we compiled a large and diverse ground truth database from publicly available and newly performed recordings. This database covers various types of calcium indicators, cell types, and signal-to-noise ratios and comprises a total of >20 hours from 225 neurons. We then developed a novel algorithm for spike inference (CASCADE) that is based on supervised deep networks, takes advantage of the ground truth database, infers absolute spike rates, and outperforms existing model-based algorithms. To optimize performance for unseen imaging data, CASCADE retrains itself by resampling ground truth data to match the respective sampling rate and noise level. As a consequence, no parameters need to be adjusted by the user. To facilitate routine application of CASCADE we developed systematic performance assessments for unseen data, we openly release all resources, and we provide a user-friendly cloud-based implementation.

show abstract

Community-based benchmarking improves spike rate inference from two-photon calcium imaging data

Berens

Freeman

Deneux

et al. 2017

Preprint

View full text Add to dashboard Cite

In recent years, two-photon calcium imaging has become a standard tool to probe the function of neural circuits and to study computations in neuronal populations 1, 2 . However, the acquired signal is only an indirect measurement of neural activity due to the comparatively slow dynamics of fluorescent calcium indicators 3 . Different algorithms for estimating spike trains from noisy calcium measurements have been proposed in the past 4-8 , but it is an open question how far performance can be improved. Here, we report the results of the spikefinder 2 . CC-BY-NC 4.0 International license peer-reviewed) is the author/funder. It is made available under a The copyright holder for this preprint (which was not . http://dx.doi.org/10.1101/177956 doi: bioRxiv preprint first posted online Aug. 18, 2017; challenge, launched to catalyze the development of new spike inference algorithms through crowd-sourcing. We present ten of the submitted algorithms which show improved performance compared to previously evaluated methods. Interestingly, the top-performing algorithms are based on a wide range of principles from deep neural networks to generative models, yet provide highly correlated estimates of the neural activity. The competition shows that benchmark challenges can drive algorithmic developments in neuroscience.We organized the spikefinder challenge to crowd-source the problem of developing algorithms for inferring spike rates from calcium signals. Such challenges have been used successfully in machine learning and computer vision to drive algorithmic developments 9 . We used a benchmark data set consisting of 92 recordings from 73 neurons 6 , acquired in the primary visual cortex and the retina of mice using different calcium indicators (OGB-1 and GCaMP6s), different scanning methods, and different sampling rates. For this data, calcium imaging has been performed simultaneously with electrophysiological recordings allowing to evaluate the performance of spike inference algorithms on ground truth data. Importantly, all data has been acquired at a zoom factor typically used during population imaging experiments, ensuring that all benchmark results reflect performance under the typical use-case conditions.For the challenge, we split the data into a training and a test set (see Table 1). For the training data, we made both the calcium and the spike traces publicly available, but kept the spike traces secret for the test data. Additionally, the publicly available data sets provided by the GENIE project 10 were available as training data. This allowed participants to adjust their models on the 3 . CC-BY-NC 4.0 International license peer-reviewed) is the author/funder. It is made available under a The copyright holder for this preprint (which was not . http://dx.doi.org/10.1101/177956 doi: bioRxiv preprint first posted online Aug. 18, 2017; training data set, while avoiding overfitting to the specific benchmark data set. Participants could upload predictions for the spike rate generated by their algorithm on a dedicate...

show abstract

CosMIC: A Consistent Metric for Spike Inference from Calcium Imaging

Cited by 4 publications

References 36 publications

On the use of calcium deconvolution algorithms in practical contexts

On the use of calcium deconvolution algorithms in practical contexts

Database and deep learning toolbox for noise-optimized, generalized spike inference from calcium imaging

Community-based benchmarking improves spike rate inference from two-photon calcium imaging data

Contact Info

Product

Resources

About