Performance of Multiple Pretrained BERT Models to Automate and Accelerate Data Annotation for Large Datasets

Tejani, Ali S.; Ng, Yee S.; Xi, Yin; Browning, Travis; Rayan, Jesse C.

doi:10.1148/ryai.220007

Cited by 20 publications

(9 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, performance of these models is often lower than what would be required clinically without additional feature engineering 13,15 or fine-tuning on thousands of manually-derived labels 14,16 specific to the task. This likely reflects the fact that medical report text has specific structure and meaning while comprising only a small proportion of the general language used to train these models.…”

Section: Many Published Methods For Extraction Of Multiple Values Use...mentioning

confidence: 99%

“…Machine learning has been used on clinical reports but not specifically for extraction of concepts from echocardiogram reports. Many have used various implementations of BERT (Bidirectional Encoder Representations from Transformers), an early large language model (LLM), to extract radiographic clinical findings 13 , mentions of devices 14 , study characteristics 15 , and result keywords 16 from radiology or pathology reports.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Mapping echocardiogram reports to a structured ontology: a task for statistical machine learning or large language models?

Subramaniam,

Rizvi,

Ramesh

et al. 2024

Preprint

View full text Add to dashboard Cite

BackgroundBig data has the potential to revolutionize echocardiography by enabling novel research and rigorous, scalable quality improvement. Text reports are a critical part of such analyses, and ontology is a key strategy for promoting interoperability of heterogeneous data through consistent tagging. Currently, echocardiogram reports include both structured and free text and vary across institutions, hampering attempts to mine text for useful insights. Natural language processing (NLP) can help and includes both non-deep learning and deep-learning (e.g., large language model, or LLM) based techniques. Challenges to date in using echo text with LLMs include small corpus size, domain-specific language, and high need for accuracy and clinical meaning in model results.MethodsWe tested whether we could map echocardiography text to a structured, three-level hierarchical ontology using NLP. We used two methods: statistical machine learning (EchoMap) and one-shot inference using the Generative Pre-trained Transformer (GPT) large language model. We tested against eight datasets from 24 different institutions and compared both methods against clinician-scored ground truth.ResultsDespite all adhering to clinical guidelines, there were notable differences by institution in what information was included in data dictionaries for structured reporting. EchoMap performed best in mapping test set sentences to the ontology, with validation accuracy of 98% for the first level of the ontology, 93% for the first and second level, and 79% for the first, second, and third levels. EchoMap retained good performance across external test datasets and displayed the ability to extrapolate to examples not initially included in training. EchoMap’s accuracy was comparable to one-shot GPT at the first level of the ontology and outperformed GPT at second and third levels.ConclusionsWe show that statistical machine learning can achieve good performance on text mapping tasks and may be especially useful for small, specialized text datasets. Furthermore, this work highlights the utility of a high-resolution, standardized cardiac ontology to harmonize reports across institutions.

show abstract

Section: Many Published Methods For Extraction Of Multiple Values Use...mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Mapping echocardiogram reports to a structured ontology: a task for statistical machine learning or large language models?

Subramaniam,

Rizvi,

Ramesh

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…A recent study has found that further pre‐training BERT on relevant, task‐specific content improves performance on classification tasks (Tejani et al., 2022). Therefore, we selected two further pre‐trained BERT models, Conversational BERT (Burtsev et al., 2018) and Bio‐clinical BERT (CliBERT) (Alsentzer et al., 2019).…”

Section: Methodsmentioning

confidence: 99%

Towards automated transcribing and coding of embodied teamwork communication through multimodal learning analytics

Zhao,

Gašević,

Swiecki

et al. 2024

Brit J Educational Tech

View full text Add to dashboard Cite

Effective collaboration and teamwork skills are critical in high‐risk sectors, as deficiencies in these areas can result in injuries and risk of death. To foster the growth of these vital skills, immersive learning spaces have been created to simulate real‐world scenarios, enabling students to safely improve their teamwork abilities. In such learning environments, multiple dialogue segments can occur concurrently as students independently organise themselves to tackle tasks in parallel across diverse spatial locations. This complex situation creates challenges for educators in assessing teamwork and for students in reflecting on their performance, especially considering the importance of effective communication in embodied teamwork. To address this, we propose an automated approach for generating teamwork analytics based on spatial and speech data. We illustrate this approach within a dynamic, immersive healthcare learning environment centred on embodied teamwork. Moreover, we evaluated whether the automated approach can produce transcriptions and epistemic networks of spatially distributed dialogue segments with a quality comparable to those generated manually for research objectives. This paper makes two key contributions: (1) it proposes an approach that integrates automated speech recognition and natural language processing techniques to automate the transcription and coding of team communication and generate analytics; and (2) it provides analyses of the errors in outputs generated by those techniques, offering insights for researchers and practitioners involved in the design of similar systems.Practitioner notesWhat is currently known about this topic Immersive learning environments simulate real‐world situations, helping students improve their teamwork skills. In these settings, students can have multiple simultaneous conversations while working together on tasks at different physical locations. The dynamic nature of these interactions makes it hard for teachers to assess teamwork and communication and for students to reflect on their performance. What this paper adds We propose a method that employs multimodal learning analytics for automatically generating teamwork‐related insights into the content of student conversations. This data processing method allows for automatically transcribing and coding spatially distributed dialogue segments generated from students working in teams in an immersive learning environment and enables downstream analysis. This approach uses spatial analytics, natural language processing and automated speech recognition techniques. Implications for practitioners Automated coding of dialogue segments among team members can help create analytical tools to assist in evaluating and reflecting on teamwork. By analysing spatial and speech data, it is possible to apply learning analytics advancements to support teaching and learning in fast‐paced physical learning spaces where students can freely engage with one another.

show abstract

“…The aim of these embedders is not to perform well on a single given property prediction task, but rather solely to produce a rich initial embedding of a molecule in latent space [ 88 ] and fully capture a lossless representation of the molecule. To this end, models are commonly trained in an unsupervised manner [ 89 ]. Unsupervised training involves unlabeled data; molecules without any specific property or endpoint being considered.…”

Section: Learned Representationsmentioning

confidence: 99%

“…Unsupervised training involves unlabeled data; molecules without any specific property or endpoint being considered. These models then have much more data available to them and if trained carefully, are more likely to converge on good parameters, enabling them to produce powerful, generalizable embeddings of new input molecules [ 89 ].…”

Section: Learned Representationsmentioning

confidence: 99%

From intuition to AI: evolution of small molecule representations in drug discovery

McGibbon,

Shave,

Dong

et al. 2023

Briefings in Bioinformatics

View full text Add to dashboard Cite

Within drug discovery, the goal of AI scientists and cheminformaticians is to help identify molecular starting points that will develop into safe and efficacious drugs while reducing costs, time and failure rates. To achieve this goal, it is crucial to represent molecules in a digital format that makes them machine-readable and facilitates the accurate prediction of properties that drive decision-making. Over the years, molecular representations have evolved from intuitive and human-readable formats to bespoke numerical descriptors and fingerprints, and now to learned representations that capture patterns and salient features across vast chemical spaces. Among these, sequence-based and graph-based representations of small molecules have become highly popular. However, each approach has strengths and weaknesses across dimensions such as generality, computational cost, inversibility for generative applications and interpretability, which can be critical in informing practitioners’ decisions. As the drug discovery landscape evolves, opportunities for innovation continue to emerge. These include the creation of molecular representations for high-value, low-data regimes, the distillation of broader biological and chemical knowledge into novel learned representations and the modeling of up-and-coming therapeutic modalities.

show abstract

Performance of Multiple Pretrained BERT Models to Automate and Accelerate Data Annotation for Large Datasets

Cited by 20 publications

References 18 publications

Mapping echocardiogram reports to a structured ontology: a task for statistical machine learning or large language models?

Mapping echocardiogram reports to a structured ontology: a task for statistical machine learning or large language models?

Towards automated transcribing and coding of embodied teamwork communication through multimodal learning analytics

From intuition to AI: evolution of small molecule representations in drug discovery

Contact Info

Product

Resources

About