Attention Augmented ConvLSTM for Environment Prediction

Lange, Bernard; Itkina, Masha; Kochenderfer, Mykel J.

doi:10.48550/arxiv.2010.09662

Cited by 2 publications

(3 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Lastly, it is expected that the performance can be further improved by applying attention to features that have greater influence on the future sequence using an attention block. In the future, we will also evaluate incorporating more recent ConvLSTM [25,42] and transformer-based modules (handles missing/noisy observations more naturally) [43,44] in our proposed seq2seq model along with evaluating these on a much larger 1000 patient lung radiotherapy cohort we are assembling.…”

Section: Discussionmentioning

confidence: 99%

“…Zhang et al [23] studied tumor growth prediction using ConvLSTM with cropped 32 × 32 patches of tumor CT, tumor contour, and intercellular volume fraction images. Even though the results achieved 0.80 dice overlap, their model suffered from the known blurring and limited long-term dependency ConvLSTM issues [25,26]. Moreover, the study focused on predicting a single timepoint from two earlier timepoints.…”

Section: Introduction and Purposementioning

confidence: 99%

See 1 more Smart Citation

Deformation Driven Seq2Seq Longitudinal Tumor and Organs-at-Risk Prediction for Radiotherapy

Lee,

Alam,

Jiang

et al. 2021

Preprint

View full text Add to dashboard Cite

Purpose: Radiotherapy presents unique challenges and clinical requirements for longitudinal tumor and organ-at-risk (OAR) prediction during treatment. The challenges include tumor inflammation/edema and radiation-induced changes in organ geometry, whereas the clinical requirements demand flexibility in input/output sequence timepoints to update the predictions on rolling basis and the grounding of all predictions in relationship to the pre-treatment imaging information for response and toxicity assessment in adaptive radiotherapy.Methods: To deal with the aforementioned challenges and to comply with the clinical requirements, we present a novel 3D sequence-to-sequence model based on Convolution Long Short Term Memory (ConvLSTM) that makes use of series of deformation vector fields (DVF) between individual timepoints and reference pre-treatment/planning CTs to predict future anatomical deformations and changes in gross tumor volume as well as critical OARs. High-quality DVF training data is created by employing hyper-parameter optimization on the subset of the training data with DICE coefficient and mutual information metric. We validated our model on two radiotherapy datasets: a publicly available head-and-neck dataset (28 patients with manually contoured pre-, mid-, and post-treatment CTs), and an internal non-small cell lung cancer dataset (63 patients with manually contoured planning CT and 6 weekly CBCTs).Results: The use of DVF representation and skip connections overcomes the blurring issue of ConvL-STM prediction with the traditional image representation. The mean and standard deviation of DICE for predictions of lung GTV at week 4, 5, and 6 were 0.83±0.09, 0.82±0.08, and 0.81±0.10, respectively, and for post-treatment ipsilateral and contralateral parotids, were 0.81±0.06 and 0.85±0.02.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introduction and Purposementioning

confidence: 99%

Deformation Driven Seq2Seq Longitudinal Tumor and Organs-at-Risk Prediction for Radiotherapy

Lee,

Alam,

Jiang

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Itkina et al proposed to use evidential occupancy grid and implement PredNet architecture for the prediction [16]. The approach is then carried forward to develop the double-pronged architecture [17] and attention-augmented ConvLSTM [18]. The latter work is able to make long-term predictions, however at turns the predictions still lose the scene structure.…”

Section: B Occupancy Grid Predictionmentioning

confidence: 99%

Allo-centric Occupancy Grid Prediction for Urban Traffic Scene Using Video Prediction Networks

Asghar

Rummelhard

Spalanzani

et al. 2022

2022 17th International Conference on Control, Automation, Robotics and Vision (ICARCV)

View full text Add to dashboard Cite

Prediction of dynamic environment is crucial to safe navigation of an autonomous vehicle. Urban traffic scenes are particularly challenging to forecast due to complex interactions between various dynamic agents, such as vehicles and vulnerable road users. Previous approaches have used egocentric occupancy grid maps to represent and predict dynamic environments. However, these predictions suffer from blurriness, loss of scene structure at turns, and vanishing of agents over longer prediction horizon. In this work, we propose a novel framework to make long-term predictions by representing the traffic scene in a fixed frame, referred as allo-centric occupancy grid. This allows for the static scene to remain fixed and to represent motion of the ego-vehicle on the grid like other agents'. We study the allo-centric grid prediction with different video prediction networks and validate the approach on the real-world Nuscenes dataset. The results demonstrate that the allo-centric grid representation significantly improves scene prediction, in comparison to the conventional ego-centric grid approach.

show abstract

Attention Augmented ConvLSTM for Environment Prediction

Cited by 2 publications

References 20 publications

Deformation Driven Seq2Seq Longitudinal Tumor and Organs-at-Risk Prediction for Radiotherapy

Deformation Driven Seq2Seq Longitudinal Tumor and Organs-at-Risk Prediction for Radiotherapy

Allo-centric Occupancy Grid Prediction for Urban Traffic Scene Using Video Prediction Networks

Contact Info

Product

Resources

About