Effect of Architectures and Training Methods on the Performance of Learned Video Frame Prediction

Yilmaz, M. Akin; Tekalp, A. Murat

doi:10.1109/icip.2019.8803624

Cited by 8 publications

(19 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In terms of network architecture, most previous methods use some form of recurrent convolutional encoder-decoder architecture [6,9]. There are also some methods that use 3D convolutions for handling temporal information [10].…”

Section: Frame Predictionmentioning

confidence: 99%

“…Several works formulate the frame prediction problem as a synthesis problem directly in the pixel domain [6,9], whereas others model similarity between successive frames by means of explicit transformations [11,12].…”

Section: Frame Predictionmentioning

confidence: 99%

“…Regarding optimization loss function, most works optimize l 1 or l 2 loss [6,12]. However, the resulting predicted frames may be blurry due to averaging effect.…”

Section: Frame Predictionmentioning

confidence: 99%

See 2 more Smart Citations

DFPN: Deformable Frame Prediction Network

Yilmaz

Tekalp

2021

2021 IEEE International Conference on Image Processing (ICIP)

Self Cite

View full text Add to dashboard Cite

Learned frame prediction is a current problem of interest in computer vision and video processing/compression. Although several deep network architectures have been proposed for learned frame prediction, to the best of our knowledge, there is no work based on using deformable convolutions for frame prediction. To this effect, we propose a deformable frame prediction network (DFPN) for task-oriented implicit motion modeling and next frame prediction. Experimental results demonstrate that the proposed DFPN model achieves state of the art results in next frame prediction in sequences with global motion. Our models and results are available at https://github.com/makinyilmaz/DFPN.

show abstract

Section: Frame Predictionmentioning

confidence: 99%

Section: Frame Predictionmentioning

confidence: 99%

See 1 more Smart Citation

DFPN: Deformable Frame Prediction Network

Yilmaz

Tekalp

2021

2021 IEEE International Conference on Image Processing (ICIP)

Self Cite

View full text Add to dashboard Cite

show abstract

“…LSTM is a widely applicable kind of RNN which contains feedback connections for both single data points and entire data sequences in deep learning [ 50 ]. The optimization task regarding accurate future image prediction has been a highlighted problem in artificial intelligence in recent several years [ 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 , 64 , 65 , 66 , 67 ]. Kalchbrenner et al have developed a video pixel network to predict the joint distribution of future image in pixel videos [ 60 ].…”

Section: Introductionmentioning

confidence: 99%

“…Xue et al proposed a cross convolutional network to synthesize future images in a probabilistic manner, based on auto-encoders of future maps and convolutional kernels, respectively, with the single input image and unknown motions [ 52 ]. Additionally, subsequent layers model [ 66 ], generative adversarial networks [ 56 ], CNNs [ 55 ], convolutional LSTM [ 68 ], and cubic LSTM [ 58 ] play significant roles in the prediction of future images.…”

Section: Introductionmentioning

confidence: 99%

Intelligent Calibration of Static FEA Computations Based on Terrestrial Laser Scanning Reference

Bao

Chen

et al. 2020

Sensors

View full text Add to dashboard Cite

The demand for efficient and accurate finite element analysis (FEA) is becoming more prevalent with the increase in advanced calibration technologies and sensor-based monitoring methods. The current research explores a deep learning-based methodology to calibrate FEA results. The utilization of monitoring reference results from measurements, e.g., terrestrial laser scanning, can help to capture the actual features in the static loading process. We learn the deviation sequence results between the standard FEA computations with the simplified geometry and refined reference values by the long short-term memory method. The complex changing principles in different deviations are trained and captured effectively in the training process of deep learning. Hence, we generate the FEA sequence results corresponding to next adjacent loading steps. The final FEA computations are calibrated by the threshold control. The calibration reduces the mean square errors of the FEA future sequence results significantly. This strengthens the calibration depth. Consequently, the calibration of FEA computations with deep learning can play a helpful role in the prediction and monitoring problems regarding the future structural behaviors.

show abstract