A survey on the application of recurrent neural networks to statistical language modeling

Mulder, Wim De; Bethard, Steven; Moens, Marie‐Francine

doi:10.1016/j.csl.2014.09.005

Cited by 222 publications

(121 citation statements)

References 34 publications

Supporting

Mentioning

116

Contrasting

Unclassified

Order By: Relevance

“…A better option might be to extend the approach presented here in the direction of convolutional or recurrent neural networks. Artificial recurrent neural networks have been applied successfully to sound recognition problems [60], and it is well known that feedback projections are common features of the auditory pathway. Developing recurrent versions of the NRFs introduced here is therefore likely to be important, particularly if we hope to develop successful models of higher order auditory cortical neurons.…”

Section: Discussionmentioning

confidence: 99%

Network Receptive Field Modeling Reveals Extensive Integration and Multi-feature Selectivity in Auditory Cortical Neurons

et al. 2016

View full text Add to dashboard Cite

Cortical sensory neurons are commonly characterized using the receptive field, the linear dependence of their response on the stimulus. In primary auditory cortex neurons can be characterized by their spectrotemporal receptive fields, the spectral and temporal features of a sound that linearly drive a neuron. However, receptive fields do not capture the fact that the response of a cortical neuron results from the complex nonlinear network in which it is embedded. By fitting a nonlinear feedforward network model (a network receptive field) to cortical responses to natural sounds, we reveal that primary auditory cortical neurons are sensitive over a substantially larger spectrotemporal domain than is seen in their standard spectrotemporal receptive fields. Furthermore, the network receptive field, a parsimonious network consisting of 1–7 sub-receptive fields that interact nonlinearly, consistently better predicts neural responses to auditory stimuli than the standard receptive fields. The network receptive field reveals separate excitatory and inhibitory sub-fields with different nonlinear properties, and interaction of the sub-fields gives rise to important operations such as gain control and conjunctive feature detection. The conjunctive effects, where neurons respond only if several specific features are present together, enable increased selectivity for particular complex spectrotemporal structures, and may constitute an important stage in sound recognition. In conclusion, we demonstrate that fitting auditory cortical neural responses with feedforward network models expands on simple linear receptive field models in a manner that yields substantially improved predictive power and reveals key nonlinear aspects of cortical processing, while remaining easy to interpret in a physiological context.

show abstract

Section: Discussionmentioning

confidence: 99%

Network Receptive Field Modeling Reveals Extensive Integration and Multi-feature Selectivity in Auditory Cortical Neurons

et al. 2016

View full text Add to dashboard Cite

show abstract

“…[26], [27], [28]), object recognition (e.g. [29]), speech modeling [30], and in medical image analysis (e.g. [31], [32], [33], [34]).…”

Section: Methodsmentioning

confidence: 99%

A Recurrent CNN for Automatic Detection and Classification of Coronary Artery Plaque and Stenosis in Coronary CT Angiography

Zreik

Hamersvelt

Wolterink

et al. 2019

IEEE Trans. Med. Imaging

224

150

View full text Add to dashboard Cite

Various types of atherosclerotic plaque and varying grades of stenosis could lead to different management of patients with coronary artery disease. Therefore, it is crucial to detect and classify the type of coronary artery plaque, as well as to detect and determine the degree of coronary artery stenosis. This study includes retrospectively collected clinically obtained coronary CT angiography (CCTA) scans of 163 patients. In these, the centerlines of the coronary arteries were extracted and used to reconstruct multi-planar reformatted (MPR) images for the coronary arteries. To define the reference standard, the presence and the type of plaque in the coronary arteries (no plaque, non-calcified, mixed, calcified), as well as the presence and the anatomical significance of coronary stenosis (no stenosis, nonsignificant i.e. < 50% luminal narrowing, significant i.e. ≥ 50% luminal narrowing) were manually annotated in the MPR images by identifying the start-and end-points of the segment of the artery affected by the plaque. To perform automatic analysis, a multi-task recurrent convolutional neural network is applied on coronary artery MPR images. First, a 3D convolutional neural network is utilized to extract features along the coronary artery. Subsequently, the extracted features are aggregated by a recurrent neural network that performs two simultaneous multiclass classification tasks. In the first task, the network detects and characterizes the type of the coronary artery plaque. In the second task, the network detects and determines the anatomical significance of the coronary artery stenosis. The network was trained and tested using CCTA images of 98 and 65 patients, respectively. For detection and characterization of coronary plaque, the method achieved an accuracy of 0.77. For detection of stenosis and determination of its anatomical significance, the method achieved an accuracy of 0.80. The results demonstrate that automatic detection and classification of coronary artery plaque and stenosis are feasible. This may enable automated triage of patients to those without coronary plaque and those with coronary plaque and stenosis in need for further cardiovascular workup.

show abstract

“…Liu et al developed a recursive recurrent neural network ( R 2 NN ) to fit the end‐to‐end decoding process for an application of statistical machine translation. Mulder et al presented a survey that covers the applications of RNN in statistical language modeling and introduced some recent significant extensions of RNN to overcome the long training period and the constraints on the context words of RNN. Socher and Lin developed a max‐margin structure based on RNN as a syntactic parser to predict natural language sentences from a typical dataset of Penn Treebank.…”

Section: Introductionmentioning

confidence: 99%

Deep recurrent neural network‐based residual control chart for autocorrelated processes

Chen

2019

Quality & Reliability Eng

View full text Add to dashboard Cite

With the growth of automation in process industries, there is correlation in the process variables. Deep learning has achieved many great successes in image and visual analysis. This paper concentrates on developing a deep recurrent neural network (RNN) model to characterize process variables at vary time lags, and then a residual chart is developed to detect mean shifts in autocorrelated processes. The experiment results indicate that the RNN‐based residual chart outperforms other typical methods (eg, autoregressive [AR]‐based control chart, back propagation network [BPN]‐based residual chart). This paper provides guideline for deep learning technique employed as an effective tool in autocorrelated process control.

show abstract

A survey on the application of recurrent neural networks to statistical language modeling

Cited by 222 publications

References 34 publications

Network Receptive Field Modeling Reveals Extensive Integration and Multi-feature Selectivity in Auditory Cortical Neurons

Network Receptive Field Modeling Reveals Extensive Integration and Multi-feature Selectivity in Auditory Cortical Neurons

A Recurrent CNN for Automatic Detection and Classification of Coronary Artery Plaque and Stenosis in Coronary CT Angiography

Deep recurrent neural network‐based residual control chart for autocorrelated processes

Contact Info

Product

Resources

About