“…Using these methods, our team has annotated 318,676 dualadjudicated breaths for use in development of VM and PVA detection algorithms, which represents one of the largest existing collections of multi-reviewer annotated VWD. (1,7,15,17,39,40) Historically, VWD annotation has been performed either by hand, use of electronic spreadsheets, and most recently by the web-based ExpertLabeler system. (41) A major limitation of these methods is that classifications are performed by visual criteria alone which are subject to high inter-rater bias due to reviewer subjectivity and local practice pattern.…”