A Word-Scale Probabilistic Latent Variable Model for Detecting Human Values

Abstract

This paper describes a probabilistic latent variable model that is designed to detect human values such as justice or freedom that a writer has sought to reflect or appeal to when participating in a public debate. The proposed model treats the words in a sentence as having been chosen based on specific values; values reflected by each sentence are then estimated by aggregating values associated with each word. The model can determine the human values for the word in light of the influence of the previous word. This design choice was motivated by syntactic structures such as noun+noun, adjective+noun, and verb+adjective. The classifier based on the model was evaluated on a test collection containing 102 manually annotated documents focusing on one contentious political issue-Net neutrality, achieving the highest reported classification effectiveness for this task. We also compared our proposed classifier with human second annotator. As a result, the proposed classifier effectiveness is statistically comparable with human annotators.

Toward automating detection of human values in the nuclear power debate

“…We note that LVM consistently yields higher recall than precision, whereas for SVM the opposite is true. These results are comparable to those reported in Takayama et al () for the value honor, the one category in those experiments that had a comparable number (317) of positive training examples.…”

An Automatic Dictionary Extraction and Annotation Method Using Simulated Annealing for Detecting Human Values

“…So far, we have only confirmed that our proposed method is applicable to the net neutrality corpus, however we have a plan to apply our method to another corpus that is related to different topic. Takayama et al [22] have achieved slightly but statistically significantly better method in terms of classification effectiveness (F 1 =0.7251 in 102-fold document crossvalidation for the net neutrality corpus), which is adopted a probabilistic latent variables model. We plan to explore other methods to see if we can generate a better values-dictionary.…”

“…Thus they improved SVM's effectiveness by expanding words features with associative words that extracted based on statistical similarity from unlabeled text and/or from hand-crafted thesaurus. In addition, there are several brand new approaches to model natural language text using probabilities [1], [10], [22]. However, these probabilistic approaches tend to focus on improving classification effectiveness in quantitative view.…”

“…As the interannotator agreement in Takayama et al [21], [22] indicates, well trained and well qualified people will sometimes make different judgments about the same sentence. To see how our proposed classifier SA compares with human annotator on a per-category basis, we ran experiments with the 20 documents (2,430 sentences) annotated by a second annotator with sufficient agreement with the first annotator as described in Takayama et al [21], [22] (These annotation by the second annotator is also available at the web page: http://nlp.inf.kyushu-u.ac.jp/values.html).…”

Understanding the Role of Human Values in the Spread of Misinformation

“…This can be achieved at the word, phrase, sentence, multiple sentence, paragraph, or document level (Fleischmann et al, 2009). Takayama et al (2014) applied a word-scale probabilistic latent variable model for detecting human values, using a combination of literature on human values to annotate their training data, including a metainventory conducted by Cheng and Fleischmann (2010) for the purposes of easing automatic detection of human values in relation to Net Neutrality. Their resulting classifier achieves a level of accuracy similar to human annotators.…”