The brain represents and reasons probabilistically about complex stimuli and motor actions using a noisy, spike-based neural code. A key building block for such neural computations, as well as the basis for supervised and unsupervised learning, is the ability to estimate the surprise or likelihood of incoming high-dimensional neural activity patterns. Despite progress in statistical modeling of neural responses and deep learning, current approaches either do not scale to large neural populations or cannot be implemented using biologically realistic mechanisms. Inspired by the sparse and random connectivity of real neuronal circuits, we present a model for neural codes that accurately estimates the likelihood of individual spiking patterns and has a straightforward, scalable, efficient, learnable, and realistic neural implementation. This model’s performance on simultaneously recorded spiking activity of >100 neurons in the monkey visual and prefrontal cortices is comparable with or better than that of state-of-the-art models. Importantly, the model can be learned using a small number of samples and using a local learning rule that utilizes noise intrinsic to neural circuits. Slower, structural changes in random connectivity, consistent with rewiring and pruning processes, further improve the efficiency and sparseness of the resulting neural representations. Our results merge insights from neuroanatomy, machine learning, and theoretical neuroscience to suggest random sparse connectivity as a key design principle for neuronal computation.

4The brain represents and reasons probabilistically about complex stimuli and motor actions 5 using a noisy, spike-based neural code. A key building block for such neural computations, as 6 well as the basis for supervised and unsupervised learning, is the ability to estimate the surprise 7 or likelihood of incoming high-dimensional neural activity patterns. Despite progress in statistical 8 modeling of neural responses and deep learning, current approaches either do not scale to large 9 neural populations or cannot be implemented using biologically realistic mechanisms. Inspired by 10 the sparse and random connectivity of real neuronal circuits, we present a new model for neural 11 codes that accurately estimates the likelihood of individual spiking patterns and has a straightfor-12 ward, scalable, efficiently learnable, and realistic neural implementation. This model's performance 13 on simultaneously recorded spiking activity of >100 neurons in the monkey visual and prefrontal 14 cortices is comparable or better than that of current models. Importantly, the model can be learned 15 using a small number of samples, and using a local learning rule that utilizes noise intrinsic to neu-16 ral circuits. Slower, structural changes in random connectivity, consistent with rewiring and pruning 17 processes, further improve the efficiency and sparseness of the resulting neural representations. 18 Our results merge insights from neuroanatomy, machine learning, and theoretical neuroscience to 19 suggest random sparse connectivity as a key design principle for neuronal computation. 20 The majority of neurons in the central nervous system know about the external world only by observ-21 ing the activity of other neurons. Neural circuits must therefore learn to represent information and 22 33 an architecture designed for a particular task will typically not support other computations, as done 34 *Co-corresponding authors in the brain. Lastly, top-down models relate to neural data on a qualitative level, falling short of 35 reproducing the detailed statistical structure of neural activity across large neural populations. In 36 contrast, bottom-up approaches grounded in probabilistic modeling, statistical physics, or deep neu-37 ral networks, can yield concise and accurate models of the joint activity of the neural population in 38 an unsupervised fashion [15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27]. Unfortunately, these 39 models are difficult to relate to the mechanistic aspects of neural circuit operation or computation, 40 because they use architectures and learning rules that are non-biological or non-scalable. 41 A neural circuit that would learn to estimate the probability of its inputs would merge these two 42 approaches: rather than implementing particular tasks or extracting specific stimulus features, com-43 puting the likelihood of the input gives a universal 'currency' for the neural computation of different 44 circuits. Such circuit could be used and reused by the brain as a recurring m...

We studied the fine temporal structure of spiking patterns of groups of up to 100 simultaneously recorded units in the prefrontal cortex of monkeys performing a visual discrimination task. We characterized the vocabulary of population activity patterns using 10 ms time bins and found that different sets of population activity patterns (codebooks) are used in different task epochs and that spiking correlations between units play a large role in defining those codebooks. Models that ignore those correlations fail to capture the population codebooks in all task epochs. Further, we show that temporal sequences of population activity patterns have strong history-dependence and are governed by different transition probabilities between patterns and different correlation time scales, in the different task epochs, suggesting different computational dynamics governing each epoch. Together, the large impact of spatial and temporal correlations on the dynamics of the population code makes the observed sequences of activity patterns many orders of magnitude more likely to appear than predicted by models that ignore these correlations and rely only on the population rates. Surprisingly, however, models that ignore these correlations perform quite well for decoding behavior from population responses. The difference of encoding and decoding complexity of the neural codebook suggests that one of the goals of the complex encoding scheme in the prefrontal cortex is to accommodate simple decoders that do not have to learn correlations.

Active Learning is concerned with the question of how to identify the most useful samples for a Machine Learning algorithm to be trained with. When applied correctly, it can be a very powerful tool to counteract the immense data requirements of Artificial Neural Networks. However, we find that it is often applied with not enough care and domain knowledge. As a consequence, unrealistic hopes are raised and transfer of the experimental results from one dataset to another becomes unnecessarily hard.In this work we analyse the robustness of different Active Learning methods with respect to classifier capacity, exchangeability and type, as well as hyperparameters and falsely labelled data. Experiments reveal possible biases towards the architecture used for sample selection, resulting in suboptimal performance for other classifiers. We further propose the new "Sum of Squared Logits" method based on the Simpson diversity index and investigate the effect of using the confusion matrix for balancing in sample selection.

No abstract