Online Video Object Detection Using Association LSTM

Lu, Yongyi; Lu, Cewu; Tang, Chi-Keung

doi:10.1109/iccv.2017.257

Cited by 130 publications

(81 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Prediction of a time series using recurrence neural networks (RNN) has been primarily investigated in the context of natural language processing, [4][5][6][7][8] financial stock market prediction, 9,10 and computer vision problems including object recognition, 11,12 tracking, 13,14 and image caption. 15 Recently RNN has been rapidly extended to health-care applications and achieved a great success in electronic health records analysis, 16,17 disease progression analysis, 18,19 and analysis of tumor cell growth.…”

Section: Introductionmentioning

confidence: 99%

Toward predicting the evolution of lung tumors during radiotherapy observed on a longitudinal MR imaging study via a deep learning algorithm

Wang

Rimner

et al. 2019

Medical Physics

View full text Add to dashboard Cite

Purpose To predict the spatial and temporal trajectories of lung tumor during radiotherapy monitored under a longitudinal magnetic resonance imaging (MRI) study via a deep learning algorithm for facilitating adaptive radiotherapy (ART). Methods We monitored 10 lung cancer patients by acquiring weekly MRI‐T2w scans over a course of radiotherapy. Under an ART workflow, we developed a predictive neural network (P‐net) to predict the spatial distributions of tumors in the coming weeks utilizing images acquired earlier in the course. The three‐step P‐net consisted of a convolutional neural network to extract relevant features of the tumor and its environment, followed by a recurrence neural network constructed with gated recurrent units to analyze trajectories of tumor evolution in response to radiotherapy, and finally an attention model to weight the importance of weekly observations and produce the predictions. The performance of P‐net was measured with Dice and root mean square surface distance (RMSSD) between the algorithm‐predicted and experts‐contoured tumors under a leave‐one‐out scheme. Results Tumor shrinkage was 60% ± 27% (mean ± standard deviation) by the end of radiotherapy across nine patients. Using images from the first three weeks, P‐net predicted tumors on future weeks (4, 5, 6) with a Dice and RMSSD of (0.78 ± 0.22, 0.69 ± 0.24, 0.69 ± 0.26), and (2.1 ± 1.1 mm, 2.3 ± 0.8 mm, 2.6 ± 1.4 mm), respectively. Conclusion The proposed deep learning algorithm can capture and predict spatial and temporal patterns of tumor regression in a longitudinal imaging study. It closely follows the clinical workflow, and could facilitate the decision‐making of ART. A prospective study including more patients is warranted.

show abstract

Section: Introductionmentioning

confidence: 99%

Toward predicting the evolution of lung tumors during radiotherapy observed on a longitudinal MR imaging study via a deep learning algorithm

Wang

Rimner

et al. 2019

Medical Physics

View full text Add to dashboard Cite

show abstract

“…The official success criteria in all major object detection datasets and competitions [1,2,3] are based on AP. Popular still-image object detection [4,5,6,7], video object detection [8,9,10] and online video object detection [11,12] papers mainly report AP and mean-AP (mAP; explained in Section 3) arXiv:1807.01696v2 [cs.CV] 5 Jul 2018 results. AP not only enjoys such vast acceptance but it also appears to be unchallenged.…”

Section: Introductionmentioning

confidence: 99%

“…For instance, using a high-precision detector is common in visual tracking methods [13,14,15,16,17], while initializing the tracker, known as tracking by detection as faster response times are required. Also, in online video object detection, the current approach is to use a still-image object detector with a general threshold (e.g., Association-LSTM [12] uses SSD [4] detections with confidence score above 0.8). A desirable performance measure should help in setting an optimal confidence score threshold per class.…”

Section: Introductionmentioning

confidence: 99%

Localization Recall Precision (LRP): A New Performance Metric for Object Detection

Öksüz

Cam

Akbaş

et al. 2018

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Average precision (AP), the area under the recall-precision (RP) curve, is the standard performance measure for object detection. Despite its wide acceptance, it has a number of shortcomings, the most important of which are (i) the inability to distinguish very different RP curves, and (ii) the lack of directly measuring bounding box localization accuracy. In this paper, we propose "Localization Recall Precision (LRP) Error", a new metric which we specifically designed for object detection. LRP Error is composed of three components related to localization, false negative (FN) rate and false positive (FP) rate. Based on LRP, we introduce the "Optimal LRP", the minimum achievable LRP error representing the best achievable configuration of the detector in terms of recall-precision and the tightness of the boxes. In contrast to AP, which considers precisions over the entire recall domain, Optimal LRP determines the "best" confidence score threshold for a class, which balances the trade-off between localization and recall-precision. In our experiments, we show that, for state-of-the-art (SOTA) object detectors, Optimal LRP provides richer and more discriminative information than AP. We also demonstrate that the best confidence score thresholds vary significantly among classes and detectors. Moreover, we present LRP results of a simple online video object detector which uses a SOTA still image object detector and show that the class-specific optimized thresholds increase the accuracy against the common approach of using a general threshold for all classes. At https://github.com/cancam/LRP we provide the source code that can compute LRP for the PASCAL VOC and MSCOCO datasets. Our source code can easily be adapted to other datasets as well.Abstract. In this document, we provide supplementary material that were omitted in the original paper due to space constraints.

show abstract

“…Lu et al [34] tackle tracking by aggregating location and appearance features per frame and combining these across time using LSTMs. Sadeghian et al [51] also combine appearance features obtained by cropped detections with velocity and interaction information using a combination of LSTMs.…”

Section: Related Workmentioning

confidence: 99%

MOTS: Multi-Object Tracking and Segmentation

Voigtlaender

Krause

Ošep

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

557

508

View full text Add to dashboard Cite

This paper extends the popular task of multi-object tracking to multi-object tracking and segmentation (MOTS). Towards this goal, we create dense pixel-level annotations for two existing tracking datasets using a semi-automatic annotation procedure. Our new annotations comprise 65,213 pixel masks for 977 distinct objects (cars and pedestrians) in 10,870 video frames. For evaluation, we extend existing multi-object tracking metrics to this new task. Moreover, we propose a new baseline method which jointly addresses detection, tracking, and segmentation with a single convolutional network. We demonstrate the value of our datasets by achieving improvements in performance when training on MOTS annotations. We believe that our datasets, metrics and baseline will become a valuable resource towards developing multi-object tracking approaches that go beyond 2D bounding boxes. We make our annotations, code, and models available at https: //www.vision.rwth-aachen.de/page/mots.

show abstract

Online Video Object Detection Using Association LSTM

Cited by 130 publications

References 20 publications

Toward predicting the evolution of lung tumors during radiotherapy observed on a longitudinal MR imaging study via a deep learning algorithm

Toward predicting the evolution of lung tumors during radiotherapy observed on a longitudinal MR imaging study via a deep learning algorithm

Localization Recall Precision (LRP): A New Performance Metric for Object Detection

MOTS: Multi-Object Tracking and Segmentation

Contact Info

Product

Resources

About