Integrating Pose and Mask Predictions for Multi-person in Videos

Heo, Miran; Hwang, Sukjun; Oh, Seoung Wug; Lee, Joon-Young; Kim, Seon Joo

doi:10.1109/cvprw56347.2022.00299

Cited by 4 publications

(4 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The fact that videos can be represented by a set of vectors represented by object centers rather than pixel-level information as demonstrated in (Heo et al 2022(Heo et al , 2023, which encourages us to directly align spatial points on the embedding space to map the motion state of polyps. Therefore, we directly exploit e 2 to establish temporal consistency.…”

Section: Cross-wise Scale Alignmentmentioning

confidence: 99%

“…As shown in Figure 2, taking e 2 as input, we first model dynamic information as center-perceived polyp motion information. We also use the Transformer (Vaswani et al 2017) layers like (Heo et al 2022(Heo et al , 2023 to center polyp motion information. In addition, we introduce a pixel-decoder (Cheng et al 2022) to generate learnable per-position embedding bias for the embedding space to cope with the dramatic variations in VPS.…”

Section: Cross-wise Scale Alignmentmentioning

confidence: 99%

“…where m is the memory length, M is a combination of dot product and dimension summation, and ϕ is the Softmax function used to normalize the memory information. Then we use cross-attention to establish clip-wise center alignment between memory information and current information (Ke et al 2021;Han et al 2022;Heo et al 2022Heo et al , 2023. In this way, clip-wise center alignment enables our model to perceive the motion state of the entire video, thereby resisting dramatic variations in VPS.…”

Section: Cross-wise Scale Alignmentmentioning

confidence: 99%

See 2 more Smart Citations

An Embedding-Unleashing Video Polyp Segmentation Framework via Region Linking and Scale Alignment

Fang,

Guo,

Lin

et al. 2024

AAAI

View full text Add to dashboard Cite

Automatic polyp segmentation from colonoscopy videos is a critical task for the development of computer-aided screening and diagnosis systems. However, accurate and real-time video polyp segmentation (VPS) is a very challenging task due to low contrast between background and polyps and frame-to-frame dramatic variations in colonoscopy videos. We propose a novel embedding-unleashing framework consisting of a proposal-generative network (PGN) and an appearance-embedding network (AEN) to comprehensively address these challenges. Our framework, for the first time, models VPS as an appearance-level semantic embedding process to facilitate generate more global information to counteract background disturbances and dramatic variations. Specifically, PGN is a video segmentation network to obtain segmentation mask proposals, while AEN is a network we specially designed to produce appearance-level embedding semantics for PGN, thereby unleashing the capability of PGN in VPS. Our AEN consists of a cross-scale region linking (CRL) module and a cross-wise scale alignment (CSA) module. The former screens reliable background information against background disturbances by constructing linking of region semantics, while the latter performs the scale alignment to resist dramatic variations by modeling the center-perceived motion dependence with a cross-wise manner. We further introduce a parameter-free semantic interaction to embed the semantics of AEN into PGN to obtain the segmentation results. Extensive experiments on CVC-612 and SUN-SEG demonstrate that our approach achieves better performance than other state-of-the-art methods. Codes are available at https://github.com/zhixue-fang/EUVPS.

show abstract

Section: Cross-wise Scale Alignmentmentioning

confidence: 99%

Section: Cross-wise Scale Alignmentmentioning

confidence: 99%

Section: Cross-wise Scale Alignmentmentioning

confidence: 99%

See 1 more Smart Citation

An Embedding-Unleashing Video Polyp Segmentation Framework via Region Linking and Scale Alignment

Fang,

Guo,

Lin

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…VITA [6] first performs instance predictions in each frame using object queries, and second employs object decoder to associate instance predictions across frames. GenVIS [30] stores object queries in previous clips to guide feature learning in current clip. Moreover, GenVIS can perform video instance segmentation in online mode.…”

Section: A Video Instance Segmentationmentioning

confidence: 99%

Instance as Identity: A Generic Online Paradigm for Video Instance Segmentation

Zhu

Yang

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Open-vocabulary video instance segmentation strives to segment and track instances belonging to an open set of categories in a video. The vision-language model Contrastive Language-Image Pre-training (CLIP) has shown strong zero-shot classification ability in image-level open-vocabulary task. In this paper, we propose a simple encoder-decoder network, called CLIP-VIS, to adapt CLIP for open-vocabulary video instance segmentation. Our CLIP-VIS adopts frozen CLIP image encoder and introduces three modules, including class-agnostic mask generation, temporal topK-enhanced matching, and weighted open-vocabulary classification. Given a set of initial queries, class-agnostic mask generation employs a transformer decoder to predict query masks and corresponding object scores and mask IoU scores. Then, temporal topK-enhanced matching performs query matching across frames by using K mostly matched frames. Finally, weighted open-vocabulary classification first generates query visual features with mask pooling, and second performs weighted classification using object scores and mask IoU scores. Our CLIP-VIS does not require the annotations of instance categories and identities.The experiments are performed on various video instance segmentation datasets, which demonstrate the effectiveness of our proposed method, especially on novel categories. When using ConvNeXt-B as backbone, our CLIP-VIS achieves the AP and APn scores of 32.1% and 40.3% on validation set of LV-VIS dataset, which outperforms OV2Seg by 11.0% and 24.0% respectively. We will release the source code and models at https://github.com/zwq456/CLIP-VIS.git.

show abstract