Active Sampling for Pairwise Comparisons via Approximate Message Passing and Information Gain Maximization

Mikhailiuk, Aliaksei; Wilmot, C J; Pérez-Ortiz, María; Yue, Dingcheng; Mantiuk, Rafał

doi:10.1109/icpr48806.2021.9412676

Cited by 31 publications

(21 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Unfortunately, the limitation is in the size of such matrix, since the number of possible comparisons is growing in 𝑂 (𝑁 2 ), 𝑁 being the number of stimuli: introducing efficiency for a subjective protocol. A lot of previous works have focused on active sampling solutions [3,7,17,18,31,42] and more recently with the work of [4,19,22,26,27,34,41]. Where the target is to select the most informative pairs and minimize experimental effort while maintaining accurate estimations and be robustness to bad annotator behavior (e.g.…”

Section: Subjective Methodologies For Perception Evaluationmentioning

confidence: 99%

Perception of video quality at a local spatio-temporal horizon

Pastor

Callet

2022

Proceedings of the 13th ACM Multimedia Systems Conference

View full text Add to dashboard Cite

This paper contains the research proposal of Andréas Pastor that was presented at the MMSys 2022 doctoral symposium. Encoding video for streaming on Internet has become a major topic to reduce the consumption of bandwidth and latency. At the same time, the human perception of distortions has been explored in multiple research projects, especially for distortions generated by Coder-DECoder (CODEC) algorithms. These algorithms operate in a rate-distortion optimization paradigm to efficiently compress video content. This optimization can be driven by metrics that are most of the time not based on the human perception, and more importantly, not tuned to reflect the local perception of distortions by human eyes.In this doctoral study, we proposed to work on the perception of localized distortion at a small temporal and spatial horizon. We present here the fundamental research questions and challenges in the domain with a focus on methods to collect perceptual judgments in subjective studies and metrics that can help us to derive an estimate of the perception of distortions by humans.

show abstract

Section: Subjective Methodologies For Perception Evaluationmentioning

confidence: 99%

Perception of video quality at a local spatio-temporal horizon

Pastor

Callet

2022

Proceedings of the 13th ACM Multimedia Systems Conference

View full text Add to dashboard Cite

show abstract

“…1) Active Sampling is a technique that chooses the pairs to compare based on which one gives the most amount of information [50]. This technique is basically a compromise between number of comparisons and accuracy of the ranking with respect to the ground truth (i.e.…”

Section: Annotation Set Creationmentioning

confidence: 99%

“…Moreover, active sampling allows to reduce this error much faster than random selection of pairs to compare. Several state-of-the-art techniques exist to perform this task, but ASAP [50] is the latest and fastest at the moment of writing. With this technique, we are guaranteed to reach at worse a 15% error within 1 3 n 2 comparisons.…”

Section: Annotation Set Creationmentioning

confidence: 99%

An architectural technical debt index based on machine learning and architectural smells

Sas¹,

Avgeriou²

2023

Preprint

View full text Add to dashboard Cite

A key aspect of technical debt (TD) management is the ability to measure the amount of principal accumulated in a system. The current literature contains an array of approaches to estimate TD principal, however, only a few of them focus specifically on architectural TD, and none of these are fully automated, freely available, and thoroughly validated. Moreover, a recent study has shown that many of the current approaches suffer from certain shortcomings, such as relying on hand-picked thresholds. In this paper, we propose a novel approach to estimate architectural technical debt principal based on machine learning and architectural smells to address such shortcomings. Our approach can estimate the amount of technical debt principal generated by a single architectural smell instance. To do so, we adopt novel techniques from Information Retrieval to train a learning-to-rank machine learning model that estimates the severity of an architectural smell and ensure the transparency of the predictions. Then, for each instance, we statically analyse the source code to calculate the exact number of lines of code creating the smell. Finally, we combine these two values to calculate the technical debt principal. To validate the approach, we conducted a case study and interviewed 16 practitioners, from both open source and industry, and asked them about their opinions on the TD principal estimations for several smells detected in their projects. The results show that for 71% of instances, practitioners agreed that the estimations provided were representative of the effort necessary to refactor the smell.

show abstract

“…Active SAmpling for Pairwise comparisons (ASAP) 16 is a state‐of‐the‐art active sampling algorithm based on information gain, that finds the best pairs to compare in ranking experiments. The use of ASAP in this paper supports the annotators' work since this algorithm minimises the number of comparisons needed to obtain a complete ranking.…”

Section: Motivation and Backgroundmentioning

confidence: 99%

“…Past active sampling solutions reduced the computational complexity with a sub‐optimal approach, only updating the probabilities of the pairs selected for the subsequent comparison, which might not converge to the best optimum. Instead, the ASAP algorithm reduces the overhead by using approximate message passing , and it only computes the information gain of the most informative pairs updating the probabilities of all the pairs, thus making it efficient and correct 16 …”

Section: Motivation and Backgroundmentioning

confidence: 99%

GitRanking: A ranking of GitHub topics for software classification using active sampling

et al. 2023

View full text Add to dashboard Cite

ContextGitHub is the world's most prominent host of source code, with more than 327M repositories. However, most of these repositories are not labelled or inadequately, making it harder for users to find relevant projects. Various proposals for software application domain classification over the past years have been proposed. However, these several of those approaches suffer from multiple issues, called antipatterns of software classification, that reduce their usability.ObjectiveIn this paper, we propose a new taxonomy in the GitHub ecosystem, called GitRanking, starting from a well‐structured data set, composed of curated repositories annotated with topics. The main objective is to create a baseline methodology for software classification that is expandable, hierarchical, grounded in a knowledge base, and free of antipatterns.MethodWe collected 121K topics from GitHub and used GitRanking to create a taxonomy of 301 ranked application domains. GitRanking (1) uses active sampling to ensure a minimal number of annotations to create the ranking; and (2) links each topic to Wikidata, reducing ambiguities and improving the reusability of the taxonomy. Furthermore, we adopt the conceived taxonomy in a classification task by considering a state‐of‐the‐art classifier.ResultsOur results show that GitRanking can effectively rank terms in a hierarchy according to how general or specific their meaning is. Furthermore, we show that GitRanking is a dynamically extensible method: it can currently accept further terms to be ranked, and with a minimum number of annotations (). Concerning the classification task, we show that the model achieves an F1‐score of 34%, with a precision of 54%.ConclusionThis paper is the first collective attempt at building a ground‐up taxonomy of software domains. Our vision is that our taxonomy, and its extensibility, can be used to better and more precisely label software projects.

show abstract

Active Sampling for Pairwise Comparisons via Approximate Message Passing and Information Gain Maximization

Cited by 31 publications

References 21 publications

Perception of video quality at a local spatio-temporal horizon

Perception of video quality at a local spatio-temporal horizon

An architectural technical debt index based on machine learning and architectural smells

GitRanking: A ranking of GitHub topics for software classification using active sampling

Contact Info

Product

Resources

About