An Automatic Dictionary Extraction and Annotation Method Using Simulated Annealing for Detecting Human Values

Abstract

Abstract-This paper studies a method for identifying word unigrams and word bigrams that are associated with one or more human values such as freedom or innovation. The key idea is to deterministically associate values with word choices, thus permitting values reflected by sentences to be assigned using dictionary lookup. This approach works nearly as well on average as the most accurate existing methods, and at close to the best results that can be achieved by a second human annotator, but the principal contribution of the new method is that the basis for the system's classification decisions are more easily interpreted by social scientists. The new method is based using a Monte Carlo algorithm with simulated annealing to efficiently explore the space for optimal assignments of human values to unigrams and bigrams. Results are reported on an annotated test collection of prepared statements from witnesses at public hearings on the topic of net neutrality. The results include accuracy comparisons with the previously reported approach.

Integrating human values in software development using a human values dashboard

“…The unbalanced nature of the dataset was also found in previous studies of human values analysis in text documents (e.g. (Ishita et al 2010;Takayama et al 2013;Takayama et al 2014)). Future work could expand the datasets for specific values by targeting specific types of applications.…”

“…This performance limitations also happened in prior studies on the detection of human values in text documents. These studies initially reported low performance (F1 score of 0.45 (Ishita et al 2010)), but a series of studies later in the following years (Takayama et al 2013;Takayama et al 2014;Takayama et al 2015) resulting in better performance (F1 score of 0.74 (Takayama et al 2016)). These recent works demonstrated that classifying human values is not a trivial task.…”