Unsupervised Ensemble of Ranking Models for News Comments Using Pseudo Answers

Abstract

Ranking comments on an online news service is a practically important task, and thus there have been many studies on this task. Although ensemble techniques are widely known to improve the performance of models, there is little types of research on ensemble neuralranking models. In this paper, we investigate how to improve the performance on the comment-ranking task by using unsupervised ensemble methods. We propose a new hybrid method composed of an output selection method and a typical averaging method. Our method uses a pseudo answer represented by the average of multiple model outputs. The pseudo answer is used to evaluate multiple model outputs via ranking evaluation metrics, and the results are used to select and weight the models. Experimental results on the comment-ranking task show that our proposed method outperforms several ensemble baselines, including supervised one.

Proceedings of the Ninth International Workshop on Natural Language Processing for Social Media

“…• PostEval: Select the most promising output (or model) per article with a continuous version of majority voting (Kobayashi, 2018), where the similarity of two outputs was calculated with NDCG. • WeightEval: Use the weighted average of the top-k promising outputs (Fujita et al, 2020), where k was chosen with the validation set. This method is a hybrid of output selection (PostEval) and output average (NormAve), where NDCG was used as a similarity function for selecting and weighting.…”

A Case Study of In-House Competition for Ranking Constructive Comments in a News Service

“…• PostEval: Select the most promising output (or model) per article with a continuous version of majority voting (Kobayashi, 2018), where the similarity of two outputs was calculated with NDCG. • WeightEval: Use the weighted average of the top-k promising outputs (Fujita et al, 2020), where k was chosen with the validation set. This method is a hybrid of output selection (PostEval) and output average (NormAve), where NDCG was used as a similarity function for selecting and weighting.…”