Reputation Systems Evaluation Survey

Abstract

Various reputation systems have been proposed for a broad range of distributed applications, such as peer-to-peer, ad-hoc, and multiagent systems. Their evaluation has been mostly based on proprietary methods due to the lack of widely acceptable evaluation measures and methodologies. Differentiating factors in various evaluation approaches include the evaluation metrics, the consideration of the dynamic behavior of peers, the use of social networks, or the study of resilience to specific threat scenarios. The lack of a generally accepted common evaluation framework hinders the objective evaluation and comparison of different reputation systems. Aiming at narrowing the gap in the research area of objective evaluation of reputation systems, in this article, we study the various approaches to evaluating and comparing reputation systems, present them in a taxonomy, and analyze their strengths and limitations, with special focus on works suggesting a Common Evaluation Framework (CEF). We identify the challenges for a widely accepted CEF that enables testing and benchmarking of reputation systems, and we present the required properties for such a CEF; we also present an analysis of current CEF-related works in the context of the identified properties and our related proposals.

A hybrid trust model using reinforcement learning and fuzzy logic

“…It is often difficult to find suitable real‐world data set for comprehensive evaluation of trust models since the effectiveness of various trust models needs to be assessed under different environmental conditions and misbehaviors . Simulation‐based evaluations offer the possibility of experimentally evaluating different trust models based on the same set of environmental parameters and scenarios, which enable the use of performance measurements for comparisons . In trust modeling for MAS research field, most of the existing trust models are assessed using simulation or synthetic data .…”

“…27 Many surveys, reviews, and theoretical evaluation frameworks exist in the literature aiming to categorize and characterize trust and reputation models. 28 Some of them are based on online reputation systems, such as in the work of Jøsang et al, 29 and others are based on trust and reputation in peer-to-peer networks such as in the work of Koutrouli and Tsalgatidou 30 or vehicular ad hoc networks such as in the work of Zhang. 31 Some reviews focus on concrete aspects or functionalities such as attack and defense techniques.…”

“…5,7,8,27,[33][34][35][36][37] A recent review of reputation system evaluation approaches has been presented in the work of Koutrouli and Tsalgatidou. 28 This section highlights a selection of notable related models with a particular emphasis on their applicability in open MASs.…”

“…5 Simulation-based evaluations offer the possibility of experimentally evaluating different trust models based on the same set of environmental parameters and scenarios, which enable the use of performance measurements for comparisons. 28 In trust modeling for MAS research field, most of the existing trust models are assessed using simulation or synthetic data. 5 Experiments differ among various works.…”

“…Usually, they focus on the performance of trust models, but there is no agreed-on benchmarks to enable comparing different results. 28 One of the most popular simulation testbeds for trust models is the agent reputation and trust testbed proposed in the work of Fullam et al 61 However, even this testbed does not claim to be able to simulate all experimental conditions of interest. For this reason, many researchers design their own simulation environments when assessing the performance of their proposed trust models.…”