Predicting Partial Orders: Ranking with Abstention

Abstract

Abstract. The prediction of structured outputs in general and rankings in particular has attracted considerable attention in machine learning in recent years, and different types of ranking problems have already been studied. In this paper, we propose a generalization or, say, relaxation of the standard setting, allowing a model to make predictions in the form of partial instead of total orders. We interpret such kind of prediction as a ranking with partial abstention: If the model is not sufficiently certain regarding the relative order of two alternatives and, therefore, cannot reliably decide whether the former should precede the latter or the other way around, it may abstain from this decision and instead declare these alternatives as being incomparable. We propose a general approach to ranking with partial abstention as well as evaluation metrics for measuring the correctness and completeness of predictions. For two types of ranking problems, we show experimentally that this approach is able to achieve a reasonable trade-off between these two criteria.

One Tag to Bind Them All: Measuring Term Abstractness in Social Metadata

“…A synset can have multiple hypernyms, so that the graph is not a tree, but a directed acyclic graph. In order to allow for comparison with the other grounding datasets, we focussed on the noun subsumption network 6 . As it consists of several disconnected hierarchies, it is useful to add a fake top-level node subsuming all the roots of those hierarchies, making the graph fully connected and allowing a relative abstractness judgment between all contained pairs of nouns.…”

“…Hence we can use it as a "ground truth" to judge the quality of a given term abstractness measure S . We are interested how well O correlates to S ; picking up the idea of the gamma rank correlation [6], we define concordant and discordant pairs between S and S as follows: a pair of terms l and k is called concordant w.r.t. two partial orderings , * , if they agree on it, i.e.…”

“…The ranking function subgraph size naturally induces a partial order O subgraph size among the set of lexical items present in a core ontology O. In order to check how close each of our introduced term abstractness measures correlate, we computed the gamma correlation coefficient [6] between the two partial orders (see Eq. 7).…”

Retrieval of Semantic Workflows with Knowledge Intensive Similarity Measures

“…For both similarity measures, we compared the ordering of the cases with those of the human expert. As a measure of correctness we used the ranking measure proposed by Cheng et al [6]: They define the correctness and completness of an ordering with respect to a reference order * based on the concordance C of the two orders, i.e., by the number case pairs ordered equivalently by both orders and by the discordance D, i.e. by the number of case pairs for which both orders differ.…”

“…Workflow cases clearly belong to the class of highly structured case representations, for which graph-based representations are promising [8,6,[14][15][16]. A particular challenge is the similaritybased retrieval of semantic workflow descriptions, as workflow similarity enables to retrieve workflows that do not match the query exactly, but that are a good starting point for adaptation.…”