This paper analyzes the behavior of stackaugmented recurrent neural network (RNN) models. Due to the architectural similarity between stack RNNs and pushdown transducers, we train stack RNN models on a number of tasks, including string reversal, contextfree language modelling, and cumulative XOR evaluation. Examining the behavior of our networks, we show that stack-augmented RNNs can discover intuitive stack-based strategies for solving our tasks. However, stack RNNs are more difficult to train than classical architectures such as LSTMs. Rather than employ stack-based strategies, more complex networks often find approximate solutions by using the stack as unstructured memory.

No abstract

No abstract

No abstract

1AbstractBrain activity as measured with functional magnetic resonance imaging (fMRI) gives the illusion of intractably high dimensionality, rife with collection and biological noise. Non-linear dimensionality reductions like PCA, UMAP, tSNE, and PHATE have proven useful for high-throughput biomedical data, but have not been extensively used in fMRI, which is known to reflect the redundancy and co-modulation of neural population activity. Here we take the manifold-geometry preserving method PHATE and extend it for use in brain activity timeseries data in a method we call temporal PHATE (T-PHATE). We observe that in addition to the intrinsically lower dimensionality of fMRI data, it also has significant autocorrelative structure that we can exploit to faithfully denoise the signal and learn brain activation manifolds. We empirically validate T-PHATE on three fMRI tasks and show that T-PHATE manifolds improve visualization fidelity, stimulus feature classification, and neural event segmentation. T-PHATE demonstrates impressive improvements over previous cutting-edge approaches to understanding the nature of cognition from fMRI and bodes potential applications broadly for high-dimensional datasets of temporally-diffuse processes.