Supervoxel-Consistent Foreground Propagation in Video

Abstract

Abstract. A major challenge in video segmentation is that the foreground object may move quickly in the scene at the same time its appearance and shape evolves over time. While pairwise potentials used in graph-based algorithms help smooth labels between neighboring (super)pixels in space and time, they offer only a myopic view of consistency and can be misled by inter-frame optical flow errors. We propose a higher order supervoxel label consistency potential for semi-supervised foreground segmentation. Given an initial frame with manual annotation for the foreground object, our approach propagates the foreground region through time, leveraging bottom-up supervoxels to guide its estimates towards long-range coherent regions. We validate our approach on three challenging datasets and achieve state-of-the-art results.

Gamifying Video Object Segmentation

“…Fig. 6 shows the interaction times of users against the achieved segmentation quality for our method and [36], [41], [52]. Our approach yielded a reasonably accuracy over the 10 considered frames after approximately 50 seconds of playtime (i.e., about 20 game sessions) while the other approaches needed much more time.…”

“…In addition, superpixel segmentation is also largely employed as it allows us to reduce processing time ensuring at the same time spatio-temporal coherency among pixels. In some work [41], temporal linking between superpixels is done manually. Despite these methods are able to alleviate human effort for video annotation, at large scale they are ineffective and still time-consuming for human operators.…”

“…We also compared our method to existing interactive video segmentation ones [36], [41], [52] in terms of interaction times and accuracy. It has to be noted that for a single user the interaction time of our method depends only on TABLE 1 Average segmentation accuracy for the video categories employed in our game in terms of precision, recall and F-measure, when we employ only superclick extraction (first row) and when we refine the output segmentation by means of spatio-temporal linking between superclicks in consecutive frames (second row).…”

“…It has to be noted that for a single user the interaction time of our method depends only on TABLE 1 Average segmentation accuracy for the video categories employed in our game in terms of precision, recall and F-measure, when we employ only superclick extraction (first row) and when we refine the output segmentation by means of spatio-temporal linking between superclicks in consecutive frames (second row). the video length, i.e., there is a linear dependency between number of annotated frames and annotation times, while existing interactive methods [36], [41], [52] show a nonlinear (exponential-like) dependency. Nevertheless, using data generated by a single user in a single played game would result in poor accuracy performance (as discussed in the previous section), thus we consider, for comparison with the state of the art, the cumulative time spent by a single user in multiple game sessions (generally one frame is shown for 0.2 seconds in our game, thus if we select the clicks of one user in five game sessions, the interaction time would be of 1 second.).…”

“…Quantitatively, Table 7 reports the achieved segmentation accuracy (expressed by POM in percentage) over the 10 considered frames by all the comparing methods in two cases: a) within 50 seconds of annotation and b) the maximum achieved accuracy. Let us recall that the interaction time (50 seconds corresponding to about 20 game sessions) for our method is given by the playtime of a single user, and that the same value can be achieved by involving 20 users, in parallel, playing only one game session, i.e., with a very little human annotation effort as opposed to the existing other solutions [36], [41], [52].…”