Putting the crowd to work in a knowledge-based factory

Corney, Jonathan; Torres-Sánchez, Carmen; Jagadeesan, Ananda Prasanna; Yan, Xiu-Tian; Regli, William C.; Medellin, H.

doi:10.1016/j.aei.2010.05.011

Cited by 28 publications

(10 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The type of task is often referred to as a human intelligence task (HIT) [45][46]. Each HIT is a small portion of a large task, which is distributed among a large group of people known as workers, who have no contact with each other.…”

Section: B Experimentsmentioning

confidence: 99%

Semantic Retrieval of Trademarks Based on Conceptual Similarity

Anuar

Setchi

Lai

2016

IEEE Trans. Syst. Man Cybern, Syst.

View full text Add to dashboard Cite

Abstract-Trademarks are signs of high reputational value. Thus, they require protection. This paper studies conceptual similarities between trademarks, which occurs when two or more trademarks evoke identical or analogous semantic content. The paper advances the state-of-the-art by proposing a computational approach based on semantics that can be used to compare trademarks for conceptual similarity. A trademark retrieval algorithm is developed that employs natural language processing techniques and an external knowledge source in the form of a lexical ontology. The search and indexing technique developed uses similarity distance, which is derived using Tversky's theory of similarity. The proposed retrieval algorithm is validated using two resources: a trademark database of 1,400 disputed cases and a database of 378,943 company names. The accuracy of the algorithm is estimated using measures from two different domains: the R-precision score, which is commonly used in information retrieval, and human judgment/collective human opinion, which is used in human-machine systems.

show abstract

Section: B Experimentsmentioning

confidence: 99%

Semantic Retrieval of Trademarks Based on Conceptual Similarity

Anuar

Setchi

Lai

2016

IEEE Trans. Syst. Man Cybern, Syst.

View full text Add to dashboard Cite

show abstract

“…These are generous payment levels in comparison to other reported research studies which could as low as $0.01 [35], $0.10 [36] [35]. However, in consideration of the experiment's level of difficulty, the lowest rate was fixed as $0.15 [37] [38]. By choosing $1.00 as a maximum payment the task would by one of the best paid on the platform where only some translation jobs might be paid as much as $1.40 per hour [35] [39].…”

Section: 21mentioning

confidence: 98%

An evaluation methodology for crowdsourced design

Corney

Grant

2015

Advanced Engineering Informatics

View full text Add to dashboard Cite

This version is available at https://strathprints.strath.ac.uk/56037/ Strathprints is designed to allow users to access the research output of the University of Strathclyde. Unless otherwise explicitly stated on the manuscript, Copyright © and Moral Rights for the papers on this site are retained by the individual authors and/or other copyright owners. Please check the manuscript for details of any other licences that may have been applied. You may not engage in further distribution of the material for any profitmaking activities or any commercial gain. You may freely distribute both the url (https://strathprints.strath.ac.uk/) and the content of this paper for research or private study, educational, or not-for-profit purposes without prior permission or charge.Any correspondence concerning this service should be sent to the Strathprints administrator: strathprints@strath.ac.ukThe Strathprints institutional repository (https://strathprints.strath.ac.uk) is a digital archive of University of Strathclyde research outputs. It has been developed to disseminate open access research outputs, expose data about those outputs, and enable the management and persistent access to Strathclyde's intellectual output. ) have yet to be established for Crowdsourced Design (cDesign). As a contribution to the development of such a general model this paper proposes a cDesign framework to support the creation of crowdsourced design activities. Within the cDesign framework the effective evaluation of design quality is identified as a key component that not only enables the leveraging of a large, virtual workforce's creative activities but is also fundamental to almost all iterative optimisation processes. This paper reports an experimental investigation into two different Crowdsourced design evaluation approaches; free evaluation and 'Crowdsourced Design Evaluation Criteria' (cDEC). The results are benchmarked against a 'manual' evaluation carried out by a panel of experienced designers. The results suggest that the cDEC approach produces design rankings that correlate strongly with the judgements of an "expert panel". The paper concludes that cDEC assessment methodology demonstrates how Crowdsourcing can be effectively used to evaluate, as well as generate, new design solutions.

show abstract

“…Crowdsourcing is an open call task recently introduced in information retrieval studies and has been proven to produce fast and reliable results in a cost-effective way (Corney, 2010;Fadzli & Setchi, 2012;Snow et al, 2008). This task, commonly known as a human intelligence task (HIT), is a small portion of an even larger task distributed among a large group of workers without any apparent contact.…”

Section: Methodsmentioning

confidence: 99%

Multi-faceted assessment of trademark similarity

Setchi

Anuar

2016

Expert Systems with Applications

View full text Add to dashboard Cite

Highlights A novel method for the assessment of trademark similarity is proposed. The method blends together visual, semantic and phonetic similarity. It produces an aggregated score based on the individual assessments. Evaluation using information retrieval measures and human judgment.ACCEPTED MANUSCRIPT One of the most influential factors in this test is establishing similarity in appearance, meaning or sound. However, even though the trademark registration process suggests a multi-faceted similarity assessment, relevant research in expert systems mainly focuses on computing individual aspects of similarity between trademarks. Therefore, this paper contributes to the knowledge in this field by proposing a method, which, similar to the way people perceive trademarks, blends together the three fundamental aspects of trademark similarity and produces an aggregated score based on the individual visual, semantic and phonetic assessments. In particular, semantic similarity is a new aspect, which has not been considered by other researchers in approaches aimed at providing decision support in trademark similarity assessment. Another specific scientific contribution of this paper is the innovative integration, using a fuzzy engine, of three independent assessments, which collectively provide a more balanced and human-centered view on potential infringement problems. In addition, the paper introduces the concept of degree of similarity since the line between similar and dissimilar trademarks is not always easy to define especially when dealing with blending three very different assessments. The work described in the paper is evaluated using a database comprising 1,400 trademarks compiled from a collection of real legal cases of trademark disputes. The evaluation involved two experiments. The first ACCEPTED MANUSCRIPT A C C E P T E D M A N U S C R I P T experiment employed information retrieval measures to test the classification accuracy of the proposed method while the second used human collective opinion to examine correlations between the trademark scoring/rating and the ranking of the proposed method, and human judgment. In the first experiment, the proposed method improved the F-score, precision and accuracy of classification by 12.5%, 35% and 8.3%, respectively, against the best score computed using individual similarity. In the second experiment, the proposed method produced a perfect positive Spearman rank correlation score of 1.00 in the ranking task and a pairwise Pearson correlation score of 0.92 in the rating task. The test of significance conducted on both scores rejected the null hypotheses of the experiment and showed that both scores correlated well with collective human judgment. The combined overall assessment could add value to existing support systems and be beneficial for both trademark examiners and trademark applicants. The method could be further used in addressing recent cyberspace phenomena related to trademark infringement such as customer hijacking and cybersquatting.

show abstract

Putting the crowd to work in a knowledge-based factory

Cited by 28 publications

References 17 publications

Semantic Retrieval of Trademarks Based on Conceptual Similarity

Semantic Retrieval of Trademarks Based on Conceptual Similarity

An evaluation methodology for crowdsourced design

Multi-faceted assessment of trademark similarity

Contact Info

Product

Resources

About