Tracking tumor alteration in glioma through serum fibroblast activation protein combined with image

Yang, Xiaosong; Zhu, Peng; Xie, Rong-Xing; Chen, Pengfei; Liu, Hong; Cheng, Xiaoman; Zhu, Zhou; Peng, Xiaomin; Liu, Haibin; Yang, Qunying; Li, Junqi; Zhang, Ji

doi:10.21203/rs.3.rs-2476989/v1

Cited by 2 publications

(3 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In Section 6.4, we show extensive visual results comparison between our HQ-SAM and SAM on COCO [31], DIS-test [34], HR-SOD [50], NDD20 [40], DAVIS [33], and YTVIS [46]. Comparison with Adapter Tuning Strategy In Table 10, we also compare our efficient token adaptation strategy to the recent Adapter Tuning [47]. We introduce lightweight adapters to ViT layers of SAM's encoder for encoder tuning and identify that this strategy leads to overfitting and its zero-shot performance on COCO decreases from 33.3 to 29.6.…”

Section: Appendixmentioning

confidence: 99%

See 1 more Smart Citation

Mask Transfiner for High-Quality Instance Segmentation

Danelljan²,

Li³

et al. 2022

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

103

View full text Add to dashboard Cite

The recent Segment Anything Model (SAM) represents a big leap in scaling up segmentation models, allowing for powerful zero-shot capabilities and flexible prompting. Despite being trained with 1.1 billion masks, SAM's mask prediction quality falls short in many cases, particularly when dealing with objects that have intricate structures. We propose HQ-SAM, equipping SAM with the ability to accurately segment any object, while maintaining SAM's original promptable design, efficiency, and zero-shot generalizability. Our careful design reuses and preserves the pre-trained model weights of SAM, while only introducing minimal additional parameters and computation. We design a learnable High-Quality Output Token, which is injected into SAM's mask decoder and is responsible for predicting the high-quality mask. Instead of only applying it on mask-decoder features, we first fuse them with early and final ViT features for improved mask details. To train our introduced learnable parameters, we compose a dataset of 44K fine-grained masks from several sources. HQ-SAM is only trained on the introduced detaset of 44k masks, which takes only 4 hours on 8 GPUs. We show the efficacy of HQ-SAM in a suite of 9 diverse segmentation datasets across different downstream tasks, where 7 out of them are evaluated in a zero-shot transfer protocol. Our code and models will be released at https://github.com/SysCV/SAM-HQ.

show abstract

Section: Appendixmentioning

confidence: 99%

“…Comparison to Adapter Tuning[47] in SAM's encoder using ViT-L based SAM. For the COCO dataset, we use the SOTA detector FocalNet-DINO[51] trained on the COCO dataset as our box prompt generator.…”

mentioning

confidence: 99%

Mask Transfiner for High-Quality Instance Segmentation

Danelljan²,

Li³

et al. 2022

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

103

View full text Add to dashboard Cite

show abstract

“…Therefore, recent video-based approaches typically freeze the encoder and adopt the CLIP representations along with additional learnable components. These components include a transformer-based temporal module (Ju et al 2022), new cross-frame communication attention for video temporal modeling and a video-specific prompting technique (Ni et al 2022), textual or visual prompts (Wang et al 2021), a spatial adaptation, temporal adaptation, and joint adaptation module (Yang et al 2023), that are learned while keeping the CLIP backbone frozen or adapting the CLIP encoders as well. These are designed to adapt CLIP while learning them quickly.…”

Section: Introductionmentioning

confidence: 99%

ViLT-CLIP: Video and Language Tuning CLIP with Multimodal Prompt Learning and Scenario-Guided Optimization

Wang,

Liu,

Jiao

et al. 2024

AAAI

View full text Add to dashboard Cite

Pre-trained vision-language(V-L) models such as CLIP have demonstrated impressive Zero-Shot performance in many downstream tasks. Since adopting contrastive video-text pairs methods like CLIP to video tasks is limited by its high cost and scale, recent approaches focus on efficiently transferring the image-based CLIP to the video domain. A major finding is that fine-tuning the pre-trained model to achieve strong fully supervised performance leads to low zero shot, few shot, and base to novel generalization. Instead, freezing the backbone network to maintain generalization ability weakens fully supervised performance. Otherwise, no single prompt tuning branch consistently performs optimally. In this work, we proposed a multimodal prompt learning scheme that balances supervised and generalized performance. Our prompting approach contains three sections: 1) Independent prompt on both the vision and text branches to learn the language and visual contexts. 2) Inter-modal prompt mapping to ensure mutual synergy. 3) Reducing the discrepancy between the hand-crafted prompt (a video of a person doing [CLS]) and the learnable prompt, to alleviate the forgetting about essential video scenarios. Extensive validation of fully supervised, zero-shot, few-shot, base-to-novel generalization settings for video recognition indicates that the proposed approach achieves competitive performance with less commute cost.

show abstract

Tracking tumor alteration in glioma through serum fibroblast activation protein combined with image

Cited by 2 publications

References 31 publications

Mask Transfiner for High-Quality Instance Segmentation

Mask Transfiner for High-Quality Instance Segmentation

ViLT-CLIP: Video and Language Tuning CLIP with Multimodal Prompt Learning and Scenario-Guided Optimization

Contact Info

Product

Resources

About