Combining Textual and Speech Features in the NLI Task Using State-of-the-Art Machine Learning Techniques

Abstract

We summarize the involvement of our CEMI team in the "NLI Shared Task 2017", which deals with both textual and speech input data. We submitted the results achieved by using three different system architectures; each of them combines multiple supervised learning models trained on various feature sets. As expected, better results are achieved with the systems that use both the textual data and the spoken responses. Combining the input data of two different modalities led to a rather dramatic improvement in classification performance. Our best performing method is based on a set of feed-forward neural networks whose hidden-layer outputs are combined together using a softmax layer. We achieved a macro-averaged F1 score of 0.9257 on the evaluation (unseen) test set and our team placed first in the main task together with other three teams.

Native Language Identification With Classifier Stacking and Ensembles

“…In the top band of seven systems (accuracies 86.5% to 88.2%), three of the best systems from each team (Cimino and Dell'Orletta 2017;Goutte and Léger 2017;Li and Zou 2017) used the architectures described later in this paper, with the same kinds of base machine learners, but incorporating some variations: for instance, Cimino and Dell'Orletta (2017) additionally used a sentence-level classifier. In the second band of three systems (accuracies 85.4% to 86.4%), all used ensembles (Chan et al 2017;Ircing et al 2017;Oh et al 2017). Interestingly, this band included the highest-ranked system to incorporate deep learning (Oh et al 2017), supporting our earlier observation that a state-of-the-art deep learning architecture for this kind of classification task has not yet been identified.…”

A Report on the 2017 Native Language Identification Shared Task

“…CEMI (Ircing et al, 2017) use a Logistic Regression meta-classifier to achieve their best essay-only results. The meta-classifier is trained on the outputs of several base classifiers, which are trained on TF-IDF weighted word unigrams, word bigrams, character n-grams and POS n-grams.…”

“…Each team's best system is briefly described below, ordered by rankings. CEMI (Ircing et al, 2017) attained their best result with an ensemble consisting of a SGD classifier trained on transcript word features and a feedforward neural network trained on the i-vector features. The final prediction is selected via softmax combination.…”

“…CEMI (Ircing et al, 2017) obtain their best results using a neural network based meta-classifier. They use several isolated feed-forward neural network models, each trained on one feature type.…”