Towards Photographic Image Manipulation with Balanced Growing of Generative Autoencoders

Heljakka, Ari; Solin, Arno; Kannala, Juho

doi:10.1109/wacv45572.2020.9093375

Cited by 13 publications

(14 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Also, it is clear that the reconstructions are not sharp, and blurry objects are reconstructed. This is typical for a variational auto-encoder, and while many approaches exist to create sharper reconstructions (Makhzani et al, 2015 ; Heljakka et al, 2018 , 2020 ; Huang et al, 2018 ), we argue that this is not necessary for our case. As long as the generated observations are spatially correlated and the object properties such as size and color are correctly reconstructed, the generative model will be capable of working within the active inference framework.…”

Section: Discussionmentioning

confidence: 74%

Active Vision for Robot Manipulators Using the Free Energy Principle

et al. 2021

View full text Add to dashboard Cite

Occlusions, restricted field of view and limited resolution all constrain a robot's ability to sense its environment from a single observation. In these cases, the robot first needs to actively query multiple observations and accumulate information before it can complete a task. In this paper, we cast this problem of active vision as active inference, which states that an intelligent agent maintains a generative model of its environment and acts in order to minimize its surprise, or expected free energy according to this model. We apply this to an object-reaching task for a 7-DOF robotic manipulator with an in-hand camera to scan the workspace. A novel generative model using deep neural networks is proposed that is able to fuse multiple views into an abstract representation and is trained from data by minimizing variational free energy. We validate our approach experimentally for a reaching task in simulation in which a robotic agent starts without any knowledge about its workspace. Each step, the next view pose is chosen by evaluating the expected free energy. We find that by minimizing the expected free energy, exploratory behavior emerges when the target object to reach is not in view, and the end effector is moved to the correct reach position once the target is located. Similar to an owl scavenging for prey, the robot naturally prefers higher ground for exploring, approaching its target once located.

show abstract

Section: Discussionmentioning

confidence: 74%

Active Vision for Robot Manipulators Using the Free Energy Principle

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Whereas Shannon (1951) used prediction to exploit an observer's knowledge about regularities and thereby measure the redundancy in signals, unsupervised learning uses prediction and other objectives to bootstrap knowledge about statistical regularities into DNNs. Unsupervised DNNs have demonstrated impressive abilities to extract higherorder regularities, which often do map onto physical properties of scenes and objects (Heljakka et al, 2020;Higgins et al, 2017;Kingma & Welling, 2013;Lotter et al, 2020;Salimans et al, 2017;van den Oord et al, 2016van den Oord et al, , 2017 and may be more robust to noise and image perturbations than regularities learned via supervised training (Hendrycks et al, 2019). The time is now ripe to revisit redundancy reduction with these newly potent learning objectives, some of which we review here.…”

Section: Unsupervised Learning: Acquiring Deep Knowledge Through Proximal Objectivesmentioning

confidence: 99%

“…Autoencoders and their variants (Heljakka et al, 2020;Higgins et al, 2017;Kingma & Welling, 2013;van den Oord et al, 2017) embody this approach to learning. They are trained to encode large databases of images in highly compacted representations-sometimes just a dozen or so numbers per image, as opposed to tens of thousands for the raw pixels.…”

Section: Compressionmentioning

confidence: 99%

Learning About the World by Learning About Images

Storrs

Fleming

2021

Curr Dir Psychol Sci

View full text Add to dashboard Cite

One of the deepest insights in neuroscience is that sensory encoding should take advantage of statistical regularities. Humans’ visual experience contains many redundancies: Scenes mostly stay the same from moment to moment, and nearby image locations usually have similar colors. A visual system that knows which regularities shape natural images can exploit them to encode scenes compactly or guess what will happen next. Although these principles have been appreciated for more than 60 years, until recently it has been possible to convert them into explicit models only for the earliest stages of visual processing. But recent advances in unsupervised deep learning have changed that. Neural networks can be taught to compress images or make predictions in space or time. In the process, they learn the statistical regularities that structure images, which in turn often reflect physical objects and processes in the outside world. The astonishing accomplishments of unsupervised deep learning reaffirm the importance of learning statistical regularities for sensory coding and provide a coherent framework for how knowledge of the outside world gets into visual cortex.

show abstract

“…An autoencoder (N E , N D ), where N E and N D are feed-forward ANNs called the encoder and the decoder respectively, is a model whose goal is to compress (encode) its inputs x ∈ R n I to low-dimensional vectors l = N E (x) ∈ R n L (again, n L < n I ) such that approximate decompression (decoding, reconstruction) can be achieved: N D (l) is close to x. A generative autoencoder (such as in [21,32]) is an autoencoder whose decoder is additionally trained to sample from the original distribution D-thus, essentially, a generative autoencoder performs both the tasks of an autoencoder and a GAN. For a well-trained generative autoencoder, we may assume both l ∼ D L ⇒ N D (l) ∼ D and…”

Section: Generative Modelsmentioning

confidence: 99%

Metrics and methods for robustness evaluation of neural networks with generative models

Buzhinsky¹,

Nerinovsky²,

Tripakis³

2020

Preprint

View full text Add to dashboard Cite

Recent studies have shown that modern deep neural network classifiers are easy to fool, assuming that an adversary is able to slightly modify their inputs. Many papers have proposed adversarial attacks, defenses and methods to measure robustness to such adversarial perturbations. However, most commonly considered adversarial examples are based on p -bounded perturbations in the input space of the neural network, which are unlikely to arise naturally. Recently, especially in computer vision, researchers discovered "natural" or "semantic" perturbations, such as rotations, changes of brightness, or more high-level changes, but these perturbations have not yet been systematically utilized to measure the performance of classifiers. In this paper, we propose several metrics to measure robustness of classifiers to natural adversarial examples, and methods to evaluate them. These metrics, called latent space performance metrics, are based on the ability of generative models to capture probability distributions, and are defined in their latent spaces. On three image classification case studies, we evaluate the proposed metrics for several classifiers, including ones trained in conventional and robust ways. We find that the latent counterparts of adversarial robustness are associated with the accuracy of the classifier rather than its conventional adversarial robustness, but the latter is still reflected on the properties of found latent perturbations. In addition, our novel method of finding latent adversarial perturbations demonstrates that these perturbations are often perceptually small.

show abstract

Towards Photographic Image Manipulation with Balanced Growing of Generative Autoencoders

Cited by 13 publications

References 17 publications

Active Vision for Robot Manipulators Using the Free Energy Principle

Active Vision for Robot Manipulators Using the Free Energy Principle

Learning About the World by Learning About Images

Metrics and methods for robustness evaluation of neural networks with generative models

Contact Info

Product

Resources

About