Weakly Supervised Object Detection Using Proposal- and Semantic-Level Relationships

Zhang, Dingwen; Zeng, Wenyuan; Yao, Jieru; Han, Junwei

doi:10.1109/tpami.2020.3046647

Cited by 107 publications

(17 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Many studies combine CAMs [ 37 ] or Weakly Supervised Semantic Segmentation (WSSS) [ 38 ] to achieve better WSOD performances. The authors of [ 39 , 40 , 41 ] leverage the power of CAMs as segmentation proposals, [ 42 , 43 , 44 , 45 ] introduce a collaboration loop between the segmentation and detection branches, [ 46 ] proposes a cascaded convolutional neural network and [ 47 ] exploit segmentation properties, i.e., purity and completeness, to harvest tight boxes that take into account the surrounding context. Still, the actual methods cannot fully exploit CAMs as bounding box generators and require the use of external domain-dependent proposals or hybrid-annotated data.…”

Section: Related Workmentioning

confidence: 99%

Proposals Generation for Weakly Supervised Object Detection in Artwork Images

2022

View full text Add to dashboard Cite

Object Detection requires many precise annotations, which are available for natural images but not for many non-natural data sets such as artworks data sets. A solution is using Weakly Supervised Object Detection (WSOD) techniques that learn accurate object localization from image-level labels. Studies have demonstrated that state-of-the-art end-to-end architectures may not be suitable for domains in which images or classes sensibly differ from those used to pre-train networks. This paper presents a novel two-stage Weakly Supervised Object Detection approach for obtaining accurate bounding boxes on non-natural data sets. The proposed method exploits existing classification knowledge to generate pseudo-ground truth bounding boxes from Class Activation Maps (CAMs). The automatically generated annotations are used to train a robust Faster R-CNN object detector. Quantitative and qualitative analysis shows that bounding boxes generated from CAMs can compensate for the lack of manually annotated ground truth (GT) and that an object detector, trained with such pseudo-GT, surpasses end-to-end WSOD state-of-the-art methods on ArtDL 2.0 (≈41.5% mAP) and IconArt (≈17% mAP), two artworks data sets. The proposed solution is a step towards the computer-aided study of non-natural images and opens the way to more advanced tasks, e.g., automatic artwork image captioning for digital archive applications.

show abstract

Section: Related Workmentioning

confidence: 99%

Proposals Generation for Weakly Supervised Object Detection in Artwork Images

2022

View full text Add to dashboard Cite

show abstract

“…OIM [23] 尝试通过在空间图和外观图上引入信息, 引入空间局部上下文关系检测所有实例, 但没有对全局上下文关系和实例信息深度挖掘, 这限制性能进一步提高. WSOD2 [19] 利用自底向上的方法蒸馏目标区域, PSLR [22] 利用图卷积神经网络挖掘局部上下文和全局上下文信息, 提高检测到完整目标的概率. 但忽略最高得分框包含多个实例的情况, 在定位精度上明显低于本文算法.…”

Section: 对比实验unclassified

“…但忽略最高得分框包含多个实例的情况, 在定位精度上明显低于本文算法. C-WSL [78] , PCL [41] , C-MIL [43] , W2F [39] , wetectron [ 3 Comparison of the performance of the proposed approach and state-of-the-art methods on the VOC 2012 set OICR [16] PCL [41] MELM [17] W2F [39] C-WSL [78] C-MIL [43] C-MIDN [40] OIM+IR [23] PSLR [22] [19] C-MIDN* OIM+IR* wetectron [79]…”

Section: 对比实验mentioning

confidence: 99%

A weakly supervised object detection approach using point annotation

Yao¹,

Han²,

Zhang³

2022

Sci. Sin.-Inf.

View full text Add to dashboard Cite

相关工作2.1 弱监督目标检测当下弱监督目标检测的研究大多建模一个多实例学习, 将弱监督目标检测问题转化为多标签分类问题 [26∼34] . 多实例学习应用到弱监督目标检测的研究可分为多阶段弱监督目标检测算法 [35∼39] 和端到端弱监督目标检测算法 [15∼17, 19, 22, 40∼43] .WSDDN [15] 由检测分支和分类分支组成, 算法将检测得分和分类得分相乘得到候选框得分, 选择高置信度正样本. Kantorov等人 [43] 引入两种上下文感知模型, 即加法模型(Additive model)和对比模型(Contrastive model), 利用上下文信息改进WSDDN池化部分. 在WSDDN基础上, Tang等人 [16] 发现将图像级标签转化为实例级监督能有效提高分类精度, 提出在线精细化实例分类网络(Online instance classifier refinement, OICR). 多实例检测网络与在线精细化实例分类网络结合, 达到了更好的性能. 激活热力图(Class activation map, CAM)可定位目标位置 [43,44] , TS2C [35] 用CAM作为目标先验, 补充在线精细化实例分类网络的监督信息. C-MIDN [40] 由两个互补的多实例检测网络组成, 利用移除候选框挖掘不同的候选边界. C-MIL [42] 为缓解多实例学习中的非凸性问题, 将实例划分为不同子集, 在子集中定义一系列平滑损失函数逼近原损失函数. 考虑到每个类别中可能存在多个实例, PCL [41] 提出候选框聚类法, W2F [39] 提出PGE、PGA算法挖掘实例监督信息. MELM [17] 通过最小化局部、全局熵进行目标检测. WSOD2 [19] 采用自适应线性组合, 结合自下而上的目标线索和自上而下的类别置信度检测目标边界. Zigzag [38] 和Zhang等人 [24] 衡量图像中目标定位困难程度, 在训练过程中从易到难训练样本, 获得更好的检测效果.虽然当下已提出许多基于多实例学习的弱监督目标检测算法, 验证了多实例检测网络与在线精细化实例分类网络结合后的有效性, 但是在空间、类别和实例间丰富依赖关系上仍处于探索阶段. 本文提出了一种基于点标注的弱监督目标检测算法, 通过推理依赖关系弥补弱监督目标检测中监督信息的不足, 以达到更好的弱监督目标检测效果. 点标注在当前海量数据下, 弱监督学习试图从低成本的弱标注中学习目标特征, 例如: 图像级标注(Imagelevels annotations) [15,35,46,47] 、涂鸦标注(Scribble annotations) [48∼50] , 噪声标注(Noise annotations) [51,52] , 点标注(Point annotations) [25, 53∼58] 和其他标注 [59] 等. 弱标注与精确的边界框标注相比, 标注数据更容易获得并有较好的检测结果. 然而, 目前在对基于点标注的弱监督目标检测这一领域的探索上仍处于空白.基于点标注的计算机视觉研究中, Point-cut [53] 提出交互式的图像分割算法. 算法以标注点为种子, 计算超像素与种子间的颜色距离, 估计前景并抑制背景杂波. What's the Point [25] 在PASCAL VOC 2012数据集上手工为图片提供点标注, 将点标注纳入训练损失函数中, 提高图像语义分割的准确性. 基于What's the Point, LC-FCN [54] 进一步提出两种新的损失函数: Split-level loss和False-positive loss, 并设计线性分割法(Line-split method)和分水岭分割法(Watershed-split method)分割多实例区域. Laradji等

show abstract

“…Despite good performances on coarse-grained datasets (e.g., Animal with Attribute dataset [3]), these approaches gradually degenerate when dealing with fine-grained datasets (e.g., Caltech-UCSD Birds-200-2011 dataset [8]), since much more local discriminative information is required to distinguish these fine-grained classes. Several recent works [9], [10], [11], [12], [13] try to focus on discriminative visual feature learning, by introducing attention mechanism into zero-shot classification problem, such as the spatial and channel attention [9], region attention [10], [14]. However, there still exists significant semantic gap between the visual modality and the attribute modality in the existing passive attention mechanisms as these methods generate attention weights purely in the bottom-up forward passing manner, which lacks the necessary top-down guidance and semantic alignment for attending to the real attribute-correlation regions.…”

Section: Introductionmentioning

confidence: 99%

Hybrid Routing Transformer for Zero-Shot Learning

Cheng¹,

Wang²,

Wang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Zero-shot learning (ZSL) aims to learn models that can recognize unseen image semantics based on the training of data with seen semantics. Recent studies either leverage the global image features or mine discriminative local patch features to associate the extracted visual features to the semantic attributes. However, due to the lack of the necessary top-down guidance and semantic alignment for ensuring the model attending to the real attribute-correlation regions, these methods still encounter a significant semantic gap between the visual modality and the attribute modality, which makes their prediction on unseen semantics unreliable. To solve this problem, this paper establishes a novel transformer encoder-decoder model, called hybrid routing transformer (HRT). In HRT encoder, we embed an active attention, which is constructed by both the bottom-up and the top-down dynamic routing pathways to generate the attributealigned visual feature. While in HRT decoder, we use static routing to calculate the correlation among the attribute-aligned visual features, the corresponding attribute semantics, and the class attribute vectors to generate the final class label predictions. This design makes the presented transformer model a hybrid of 1) top-down and bottom-up attention pathways and 2) dynamic and static routing pathways. Comprehensive experiments on three widely-used benchmark datasets, namely CUB, SUN, and AWA2, are conducted. The obtained experimental results demonstrate the effectiveness of the proposed method.

show abstract

Weakly Supervised Object Detection Using Proposal- and Semantic-Level Relationships

Cited by 107 publications

References 55 publications

Proposals Generation for Weakly Supervised Object Detection in Artwork Images

Proposals Generation for Weakly Supervised Object Detection in Artwork Images

A weakly supervised object detection approach using point annotation

Hybrid Routing Transformer for Zero-Shot Learning

Contact Info

Product

Resources

About