Redcoat: A Collaborative Annotation Tool for Hierarchical Entity Typing

Abstract

We introduce Redcoat, a web-based annotation tool that supports collaborative hierarchical entity typing. As an annotation tool, Redcoat also facilitates knowledge elicitation by allowing the creation and continuous refinement of concept hierarchies during annotation. It aims to minimise not only annotation time but the time it takes for project creators to set up and distribute projects to annotators. Projects created using the web-based interface can be rapidly distributed to a list of email addresses. Redcoat handles the propagation of documents amongst annotators and automatically scales the annotation workload depending on the number of active annotators. In this paper we discuss these key features and outline Redcoat's system architecture. We also highlight Redcoat's unique benefits over existing annotation tools via a qualitative comparison.

PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts

“…Since then, many Webbased systems for annotating text have been developed (Stenetorp et al, 2012;Salgado et al, 2012;Wei et al, 2013;Yimam et al, 2013;Chen and Styler, 2013;Eckart de Castilho et al, 2016;Putra et al, 2020). Other systems support collaboration among multiple annotators (Yang et al, 2018;Stewart et al, 2019). More recently, many annotation systems have begun to incorporate learned models to improve workflow, using techniques such as ac- tive learning (Lin et al, 2019; and example recommendation (Lee et al, 2020;.…”

Adapting natural language processing for technical text

“…40 These NLP-based tools allow for the manual injection of real-world knowledge into the learning process by providing ontological information that can guide categorization and generalization. Two such tools, Nestor 41 and Redcoat, 42 allow for the tagging of short technical text, such as found in MWO descriptions, with annotations that facilitate processing. Machine learning systems can then use these tags as a signal to promote generalization by helping to mitigate the shallow heuristics and spurious correlations that could otherwise affect learning.…”

FAST: Fast Annotation tool for SmarT devices

“…We conclude this chapter with related work on methods to evaluate the contribution of annotation tools. A comparison at the functional level is often performed for tools with a large feature set, such as RedCoat (Stewart et al, 2019). Conversely, there are situations in which the performance of actual tasks are directly evaluated.…”