Toward Accurate and Realistic Virtual Try-on Through Shape Matching and Multiple Warps

Abstract

A virtual try-on method takes a product image and an image of a model and produces an image of the model wearing the product. Most methods essentially compute warps from the product image to the model image and combine using image generation methods. However, obtaining a realistic image is challenging because the kinematics of garments is complex and because outline, texture, and shading cues in the image reveal errors to human viewers. The garment must have appropriate drapes; texture must be warped to be consistent with the shape of a draped garment; small details (buttons, collars, lapels, pockets, etc.) must be placed appropriately on the garment, and so on. Evaluation is particularly difficult and is usually qualitative. This paper uses quantitative evaluation on a challenging, novel dataset to demonstrate that (a) for any warping method, one can choose target models automatically to improve results, and (b) learning multiple coordinated specialized warpers offers further improvements on results. Target models are chosen by a learned embedding procedure that predicts a representation of the products the model is wearing. This prediction is used to match products to models. Specialized warpers are trained by a method that encourages a second warper to perform well in locations where the first works poorly. The warps are then combined using a U-Net. Qualitative evaluation confirms that these improvements are wholesale over outline, texture shading, and garment details.

Shape Controllable Virtual Try-on for Underwear Models

“…Han et al [14] and Ren et al [35] adopted optical flow to get a more natural texture deformation effect. Li et al [28] tried to get better results by using multiple specialized warps. Ge et al [9] proposed a knowledge distillation method to produce images without human parsing.…”

“…The online virtual try-on system [14,15,17,18,20,28,31,34,35,37,47,51,53,55] has become a research hot-spot in recent years to accommodate the vast market demand. Virtual try-on applications Existing virtual try-on techniques can be divided into 3D virtual try-on methods and 2D image-based methods.…”

Dressing in Order: Recurrent Person Image Generation for Pose Transfer, Virtual Try-on and Outfit Editing

“…Generation of images of a given person with a desired garment on is a challenging task that requires both capturing the garment precisely and dressing it properly on the given human body. The simplest tryon methods are aimed at replacing a single garment with a new one [4,9,10,15,17,18,39,41]. Our work is more closely related methods that attempt to model all the garments worn by a person simultaneously, allowing users to achieve multiple garment try-on [28,29,32,35].…”

Correspondence Learning via Linearly-invariant Embedding

“…Computing correspondences between geometric objects is a widely investigated task. Its applications are countless: rigid and non-rigid registration methods are instrumental in engineering, medicine and biology [25,29,16] among other fields. Point cloud registration is important for range scan data, e.g., in robotics [19,52], but the problem can also be generalized to abstract domains like graphs [58,15].…”