“…Interaction with experts would appear to be crucial for dealing with the above questions, etc. Experience suggests that it is highly advisable to consult a group or panel of experts for multiple gold-standard annotation rather than relying on a single expert [61] , [62] , [63] , [64] , [65] , [66] , [67] , [68] . The premises of this panel-based approach are as follows: - There are two or more individuals, each characterized by his or her own perceptions, attitudes, motivations, and personalities,
- who recognize the existence of a common problem, and
- attempt to reach a collective decision.
…”