Monocular Image-Based 3-D Model Retrieval: A Benchmark

Abstract

Image-based virtual try-on aims to synthesize a naturally dressed person image with a clothing image, which revolutionizes online shopping and inspires related topics within image generation, showing both research significance and commercial potential. However, there is a big gap between current research progress and commercial applications and an absence of comprehensive overview of this field to accelerate the development. In this survey, we provide a comprehensive analysis of the state-of-the-art techniques and methodologies in aspects of pipeline architecture, person representation and key modules such as try-on indication, clothing warping and try-on stage. We propose a new semantic criteria with CLIP, and evaluate representative methods with uniformly implemented evaluation metrics on the same dataset. In addition to quantitative and qualitative evaluation of current open-source methods, we also utilize ControlNet to fine-tune a recent large image generation model (PBE) to show future potential of large-scale models on imagebased virtual try-on task. Finally, unresolved issues are highlighted and future research directions are prospected to identify key trends and inspire further exploration. The uniformly implemented evaluation metrics, dataset and collected methods will be made public available at https://github.com/little-misfit/Survey-Of-Virtual-Try-On.

Vision based 3D Object Detection using Deep Learning: Methods with Challenges and Applications towards Future Directions

“…Although 3D point cloud extracted from LiDAR provides superior performance, previously less effective alternative was monocular RGB images that were collected from single view RGB image as input for monocular 3D object detection [15]. Usage of monocular RGB images can facilitate domain adaptation for multimodal big data management and significantly aids for 3D model retrieval and classification when compared with multiview image sets [16]. Research in [4] used a single monocular image to generate class specific object proposals to run through standard CNN pipelines for 3D object detection.…”

“…However, fruitful experimentation on large variations of scales and distances can provide real time usage of their proposed method. There are several challenges exists for using monocular images for 3D object detection, i.e., significant information loss, object pose inconsistency, complex background of monocular images [16], accurate depth information [6]. Monocular images lose significant information during calibration such as multiview visual characteristics and spatial structure characteristics.…”

“…However, alternative point-cloud network structures, such as VoxelNet with sparse convolutions could be investigated as their backbone network. Research in [16] used ResNet-50 as the backbone for feature extraction followed by a fully connected neural network [45] with one or two hidden layers. The number of hidden nodes is tuned with 64, 100, 128, and 192 to generate the candidate results which cause additional processing due to too many layers.…”

“…Research in [47] used ResNet34 [48,49] with Feature Pyramid Network (FPN) [48] as backbone network. They replaced BatchNorm+ReLU layers with the synchronized version of InPlaceABN activated with LeakyReLU with negative slope 0.01 as proposed in [50] to • InceptionNet-v4 [98] • DenseNet-201 [16] • ResNet-18 [16] • ResNet-101 [3,19,50,53] • FPN [3,37,62] • RTM3D [97] • DLA-34 [15] • ResNet-50 [16,33,69] • DLA-34 [17,22,45,52,56] • ResNet34 [37] • DenseNet121 [38,60] • Darknet-53 [10] • ResNet-152 [10] • PointNet (v1) [11,14,67] • PointNet++ [11,14,67] • SDN [24,99] • ResNet34 [37,62,98] • FPN [62] www.ijacsa.thesai.org free up a significant amount of GPU memory for scaling up the batch size or input resolution. However, top down and bottom-up features fusion in feature pyramid network might create additional processing due to combine ResNet34 and FPN.…”

“…InceptionNet-v4 [49] was previously used for visual feature extraction, however, in case of 3D models for multiview mages, InceptionNet-v4 needs to be further investigated for 3D object detection overall pipeline. DenseNet-201 [16] was used to extract the visual features of monocular images and ResNet-18 [16] was for multiview feature extraction of 3-D models. However, combination two backbone network at two different phases, may increase overall computational overheads.…”