Bringing Sketch Recognition into Your Hands

Abstract

The paper mentions that a flexible method of sketch recognition works consistently across a variety of software and hardware platforms, including mobile devices such as the Nintendo DS and iPhone. An example application employs this method to drive facial expressions.

Cartoonish sketch-based face editing in videos using identity deformation transfer

“…The following two sketch-based facial animation editing systems proposed recently are most related to our work. Nataneli et al [12] introduced an internal representation of sketch's semantics, while users have to draw sketch in some predefined regions. Miranda et al [8] built a sketching interface control system, which only allows users to draw strokes on a predefined set of areas corresponding to different face landmarks to avoid ambiguous conditions.…”

“…To evaluate the performance of sketch matching, we compare our method with a geometry-based algorithm described in [8] and another learning-based approach [12] which both achieve state-of-the-art performance. We use the stroke similarity measurement described in them to match strokes with landmarks, [12]. From top to bottom: two individual strokes, one stroke with noises, and a group of strokes with ambiguity (different colors represent different matched landmark group pairs).…”

“…However, as shown in the second case of Fig. 10, [8] is sensitive to noises since it fits landmarks to strokes via only geometry features; Nataneli et al [12] is able to handle this case due to pre-learned prior knowledge; our method can remove noises by taking the original appearance of the landmark group (the shape prior term in Eq. 7) into consideration.…”

“…(2) Since the face appearance depends on the pose of the actor as well as the identity, the influence of facial expression should be taken into account when applying changes to the identity. (3) Compared with previous sketch-based methods designed for static 2D images or 3D models [7,8,11,12], our framework has to further propagate the modifications from one frame to the whole video. In this process, we need to predict the modified face appearance in each frame while ensuring consistency and fidelity.…”

“…3(d) shows an example. Previous methods solve this problem by only allowing users to edit the face with pre-defined curves [7,11,14] or draw strokes in pre-ordered regions [8,12]. Instead, we introduce a robust mapping schema which enables users to draw strokes without certain constraints, while the only assumption we made here is that the stroke should be "clean", i.e., each stroke aims to edit just one target landmark group.…”