LaTeRF: Label and Text Driven Object Radiance Fields

Enhancing image resolution of confocal fluorescence microscopy with deep learning

“…By manipulating the generators, VON can perform a range of tasks, including shape manipulation, appearance transfer, and novel view synthesis [157].Mirzaei et al proposed a reference-guided controllable inpainting method for neural radiance fields (NeRFs), which allows for the synthesis of novel views of a scene with missing regions. The method employs a reference image to guide the inpainting process, and a user interface that enables the user to adjust the degree of blending between the reference and the original NeRF [158].Yin et al introduced OR-NeRF, a novel pipeline that can remove objects from 3D scenes using point or text prompts on a single view. This pipeline leverages a points projection strategy, a 2D segmentation model, 2D inpainting methods, and depth supervision and perceptual loss to achieve better editing quality and efficiency than previous works [159].…”

“…Cohen-Bar et al [172] proposed a novel framework for synthesizing and manipulating 3D scenes from text prompts and object proxies. Finally, Mirzaei et al [173] proposed a novel method for reconstructing 3D scenes from multiview images by leveraging neural radiance fields (NeRF) to model the geometry and appearance of the scene, and introducing a segmentation network and a perceptual inpainting network to handle occlusions and missing regions. These methods represent significant progress towards the goal of enabling high-quality, user-driven 3D scene synthesis and editing.…”

SKED: Sketch-guided Text-based 3D Editing

“…The pivotal work of [45] introduced an efficient hash-based representation that allows NeRF optimizations to converge within seconds, effectively paving the way for interactive research directions on neural radiance fields. Recent works have ex-plored interactive editing of neural radiance fields through manipulation of appearance latents [38,48,50,8], by interacting with proxy representations [80,10], through segmented regions and masks [30,35,43,31] and text-based stylization [17,74,81,43,15].…”

Cycle-Consistent Generative Rendering for 2D-3D Modality Translation

“…Other models have utilized neural feature fields (NFFs), as opposed to "radiance" fields, where rendering is altered to output learned features instead. Some NFFs [67], [68] learn to produce the outputs of pretrained 2D feature extractors; similarly, several works have considered the use of languagerelated features [69], [70], [71] and other segmentation signals [72], [73], [74], [5] to embed semantics into the NFF. More closely related to our work are generative modelling NFFs that decode rendered features into images via generative adversarial networks [75], [76], [77] or diffusion models [78], [79], [80].…”

“…Neural rendering techniques [1] continue to grow in importance, particularly Neural Radiance Fields [2] (NeRFs), which achieve state-of-the-art performance in novel view synthesis and 3D-from-2D reconstruction. As a result, NeRFs have been utilized for a variety of applications, not only in content creation [3], [4], [5], [6], but also for many robotics tasks, including 6-DoF tracking [7], pose estimation [8], surface recognition [9] or reconstruction [10], motion planning [11], [12], [13], reinforcement learning [14], [15], tactile sensing [16], and data-driven simulation [17], [18]. However, slow rendering and the qualitative artifacts of NeRFs impede further use cases in production.…”