Truth Discovery in Sequence Labels from Crowds

Sabetpour, Nasim; Kulkarni, Adithya; Xie, Sihong; Li, Qi

doi:10.1109/icdm51629.2021.00065

“…The more robust performance demonstrated by our Neural-Hidden-CRF relative to the SOTA neuralized HMM-based CHMM [18] largely showcases the effectiveness of our model in leveraging the global optimization perspective offered by the undirected graphical model. Further and more specifically, on the average F1 metric, Neural-Hidden-CRF outperforms the recently proposed AggSLC [35] by 4.81 points on CoNLL-03 (MTurk), and exceeds the SOTA method CHMM [18] by 2.80/2.23 points on the three WS datasets. It is also worth noting that the comparison methods [17,26,35,46] on CoNLL-03 (MTurk) dataset, apply either the same backbone (i.e., the GloVe 100-dimensional word embeddings along with BiLSMT-CRF in Nguyen et al [26]) as ours, or more advanced backbones (i.e., BERT-BiLSTM-CRF in Zhang et al [46], Efficient ELMO along with BiLSTM-CRF in Lan et al [17], BERT in AggSLC [35]) than ours.…”

Section: Results and Analysismentioning

confidence: 86%

“…All WSSL methods can be divided into probabilistic graphical model approach, deep learning model approach, and neuralized graphical model approach. (1) In probabilistic graphical model approach (and in addition to the HMM-based models [20,21,26,36,39]), Rodrigues et al [32] in early 2014 used a partially directed graph containing a CRF for modeling to solve the truth inference from crowdsourcing labels; (2) In deep learning model approach (and in addition to the "source-specific perturbation" methods [17,26,46]), other methods [17,[33][34][35] are either based on the end-to-end deep neural architecture [33], or the customized optimization objective along with coordinate ascent optimization technology [34,35], or the iterative solving framework similar to expectation-maximization algorithm [4]. However, all these methods do not have the advantages of the recently proposed neuralized HMM-based graphical models [18,19] and our Neural-Hidden-CRF in principled modeling for variants of interest and in harnessing the context information that provided by advanced deep learning models.…”

Section: Related Workmentioning

confidence: 99%

“…Specifically: (1) CoNLL-03 (MTurk) [32,33] is constructed on the well-established CoNLL-03 dataset [37] through introducing additional crowdsourcing annotations. [33], where "VW", "VW+B" and "MW" refer to three different ways of parameterizing weak source reliability; (iii) LSTM-Crowd [26], LSTM-Crowd-cat [26], Zhang et al [46], and CONNET [17]: These four methods, which apply the "source-specific perturbation" mentioned in Section 1, dominate the deep learning-based WSSL methods and show the competitive results [17]; (iv) OptSLA [34] and AggSLC [35]: They both follow the approach of constructing an optimization objective containing weak source weights, classifier parameters, latent ground truth, and iteratively updating them using a coordinate ascent algorithm; (v) CRF-MA [31]: This is a partial directed graphical model where the ground truth sequence is also modeled as a latent variable and each weak source's behavior pattern is modeled by a specific scalar;…”

Section: Experiments 31 Setupmentioning

confidence: 99%

See 1 more Smart Citation

Neural-Hidden-CRF: A Robust Weakly-Supervised Sequence Labeler

Chen

¹

,

Sun

²

,

Zhang

³

et al. 2023

Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

1

0

View full text Add to dashboard Cite

We propose a neuralized undirected graphical model called Neural-Hidden-CRF to solve the weakly-supervised sequence labeling problem. Under the umbrella of probabilistic undirected graph theory, the proposed Neural-Hidden-CRF embedded with a hidden CRF layer models the variables of word sequence, latent ground truth sequence, and weak label sequence with the global perspective that undirected graphical models particularly enjoy. In Neural-Hidden-CRF, we can capitalize on the powerful language model BERT or other deep models to provide rich contextual semantic knowledge to the latent ground truth sequence, and use the hidden CRF layer to capture the internal label dependencies. Neural-Hidden-CRF is conceptually simple and empirically powerful. It obtains new state-of-the-art results on one crowdsourcing benchmark and three weak-supervision benchmarks, including outperforming the recent advanced model CHMM by 2.80 F1 points and 2.23 F1 points in average generalization and inference performance, respectively.

show abstract

“…Given this concern, it is possible to estimate method performance in the context of label errors (Raykar et al 2009; Y. Yan et al 2014) or correct errored labels so traditional assessment metrics are more accurate (Sabetpour et al 2021; G. Zheng et al 2021).…”

Section: Discussionmentioning

confidence: 99%

Unsupervised Contrastive Peak Caller for ATAC-seq

Vu

¹

,

Zhang

²

,

Tuteja

³

et al. 2023

Preprint

1

0

View full text Add to dashboard Cite

The assay for transposase-accessible chromatin with sequencing (ATAC-seq) is a common assay to identify chromatin accessible regions by using a Tn5 transposase that can access, cut, and ligate adapters to DNA fragments for subsequent amplification and sequencing. These sequenced regions are quantified and tested for enrichment in a process referred to as "peak calling". Most unsupervised peak calling methods are based on simple statistical models and suffer from elevated false positive rates. Newly developed supervised deep learning methods can be successful, but they rely on high quality labeled data for training, which can be difficult to obtain. Moreover, though biological replicates are recognized to be important, there are no established approaches for using replicates in the deep learning tools, and the approaches available for traditional methods either cannot be applied to ATAC-seq, where control samples may be unavailable, or are post-hoc and do not capitalize on potentially complex, but reproducible signal in the read enrichment data. Here, we propose a novel peak caller that uses unsupervised contrastive learning to extract shared signals from multiple replicates. Raw coverage data are encoded to obtain low-dimensional embeddings and optimized to minimize a contrastive loss over biological replicates. These embeddings are passed to another contrastive loss for learning and predicting peaks and decoded to denoised data under an autoencoder loss. We compared our Replicative Contrastive Learner (RCL) method with other existing methods on ATAC-seq data, using annotations from ChromHMM genome and transcription factor ChIP-seq as noisy truth. RCL consistently achieved the best performance.

show abstract

“…Thus, observed recall is a contaminated measure of recall, and methods compared via observed recall (or F1 or PR curve) may not reveal their actual ranking. Given this concern, it is possible to estimate method performance in the context of label errors (Raykar et al 2009;Y Yan et al 2014) or correct errored labels so traditional assessment metrics are more accurate (Sabetpour et al 2021;G Zheng et al 2021). Alternatively, performance evaluation can be carried out with simulated data.…”

Section: Discussionmentioning

confidence: 99%

Unsupervised contrastive peak caller for ATAC-seq

Vu

¹

,

Zhang

²

,

Tuteja

³

et al. 2023

Genome Res.

1

0

View full text Add to dashboard Cite

The assay for transposase-accessible chromatin with sequencing (ATAC-seq) is a common assay to identify chromatin accessible regions by using a Tn5 transposase that can access, cut, and ligate adapters to DNA fragments for subsequent amplification and sequencing. These sequenced regions are quantified and tested for enrichment in a process referred to as "peak calling". Most unsupervised peak calling methods are based on simple statistical models and suffer from elevated false positive rates. Newly developed supervised deep learning methods can be successful, but they rely on high quality labeled data for training, which can be difficult to obtain. Moreover, though biological replicates are recognized to be important, there are no established approaches for using replicates in the deep learning tools, and the approaches available for traditional methods either cannot be applied to ATAC-seq, where control samples may be unavailable, or are post-hoc and do not capitalize on potentially complex, but reproducible signal in the read enrichment data. Here, we propose a novel peak caller that uses unsupervised contrastive learning to extract shared signals from multiple replicates. Raw coverage data are encoded to obtain low-dimensional embeddings and optimized to minimize a contrastive loss over biological replicates. These embeddings are passed to another contrastive loss for learning and predicting peaks and decoded to denoised data under an autoencoder loss. We compared our Replicative Contrastive Learner (RCL) method with other existing methods on ATAC-seq data, using annotations from ChromHMM genome and transcription factor ChIP-seq as noisy truth. RCL consistently achieved the best performance.

show abstract

Truth Discovery in Sequence Labels from Crowds

Cited by 9 publications

References 35 publications

Neural-Hidden-CRF: A Robust Weakly-Supervised Sequence Labeler

Neural-Hidden-CRF: A Robust Weakly-Supervised Sequence Labeler

Unsupervised Contrastive Peak Caller for ATAC-seq

Unsupervised contrastive peak caller for ATAC-seq

Contact Info

Product

Resources

About