Agreement statistics play an important role in the evaluation of coding schemes for discourse and dialogue. Unfortunately there is a lack of understanding regarding appropriate agreement measures and how their results should be interpreted. In this article we describe the role of agreement measures and argue that only chance-corrected measures that assume a common distribution of labels for all coders are suitable for measuring agreement in reliability studies. We then provide recommendations for how reliability should be inferred from the results of agreement statistics.Since Jean Carletta (1996) exposed computational linguists to the desirability of using chance-corrected agreement statistics to infer the reliability of data generated by applying coding schemes, there has been a general acceptance of their use within the field. However, there are prevailing misunderstandings concerning agreement statistics and the meaning of reliability.Investigation of new dialogue types and genres has been shown to reveal new phenomena in dialogue that are ill suited to annotation by current methods and also new annotation schemes that are qualitatively different from those commonly used in dialogue analysis. Previously prescribed practices for evaluating coding schemes become less applicable as annotation schemes become more sophisticated. To compensate, we need a greater understanding of reliability statistics and how they should be interpreted. In this article we discuss the purpose of reliability testing, address certain misunderstandings, and make recommendations regarding the way in which coding schemes should be evaluated. Agreement, Reliability, and Coding SchemesAfter developing schemes for annotating discourse or dialogue, it is necessary to assess their suitability for the purpose for which they are designed. Although no statistical test can determine whether any form of annotation is worthwhile or how applications will benefit from it, we at least need to show that coders are capable of performing the annotation. This often means assessing reliability based on agreement between annotators applying the scheme. Agreement measures are discussed in detail in section 2.Much of the confusion regarding which agreement measures to apply and how their results should be interpreted stems from a lack of understanding of what it means to

No abstract

No abstract

Drug combination testing in the pharmaceutical industry has typically been driven by late-stage opportunistic strategies rather than by early testing to identify drug combinations for clinical investigation that may deliver improved efficacy. A rationale for combinations exists across a number of diseases in which pathway redundancy or resistance to therapeutics are evident. However, early assays are complicated by the absence of both assay formats representative of disease biology and robust infrastructure to screen drug combinations in a medium-throughput capacity. When applying drug combination testing studies, it may be difficult to translate a study design into the required well contents for assay plates because of the number of compounds and concentrations involved. Dispensing these plates increases in difficulty as the number of compounds and concentration points increase and compounds are subsequently rolled onto additional labware. We describe the development of a software tool, in conjunction with the use of acoustic droplet technology, as part of a compound management platform, which allows the design of an assay incorporating combinations of compounds. These enhancements to infrastructure facilitate the design and ordering of assay-ready compound combination plates and the processing of combinations data from high-content organotypic assays.

Dialogue Acts (DAs) which explicitly ensure mutual understanding are frequent in dialogues between cancer patients and health professionals. We present examples, and argue that this arises from the health-critical nature of these dialogues.