Style Equalization: Unsupervised Learning of Controllable Generative Sequence Models

Abstract

Controllable generative sequence models with the capability to extract and replicate the style of specific examples enable many applications, including narrating audiobooks in different voices, auto-completing and auto-correcting written handwriting, and generating missing training samples for downstream recognition tasks. However, typical training algorithms for these controllable sequence generative models suffer from the training-inference mismatch, where the same sample is used as content and style input during training but different samples are given during inference. In this paper, we tackle the training-inference mismatch encountered during unsupervised learning of controllable generative sequence models. By introducing a style transformation module that we call style equalization, we enable training using different content and style samples and thereby mitigate the traininginference mismatch. To demonstrate its generality, we applied style equalization to text-to-speech and text-to-handwriting synthesis on three datasets. Our models achieve state-of-the-art style replication with a similar mean style opinion score as the real data. Moreover, the proposed method enables style interpolation between sequences and generates novel styles. 1 * work done during internship at Apple 1 Our synthesis results can be viewed at https://apple.github.io/ml-style-equalization

Data Incubation -- Synthesizing Missing Data for Handwriting Recognition

“…Online handwriting synthesis models [2,12,13,14,15,16] are sequence-to-sequence models that take input text (a sequence of characters) and output handwriting (a sequence of strokes). To improve downstream recognizers, it is important for the synthesized handwriting to contain few artifacts and a wide range of handwriting styles -from highly legible, printed-style handwriting to less legible, cursive handwriting.…”

“…We use the controllable generative model proposed in [2]. Given N real training samples, X = (x i , c i ) i=1...N , the generative model learns a distribution of handwriting samples, p(x|c, z), conditioned on the content c and the style z.…”

“…We model in an auto-regressive manner the output distribution p(∆x and c (generated by a linear layer). We refer to [2] for detailed description of the model. We trained two generative models: an English model using only the English training data and a multilingual model using training data from all six languages.…”

“…This recipe us to examine and understand the problems associated with the synthetic data. We build on the recently-proposed controllable generative model [2], which separately controls the content and the handwriting style. As we will demonstrate, utilizing data incubation significantly improves our multilingual handwriting recognizer, achieving a 66% reduction in Character Error Rate when compared to a recognizer trained with only original data.…”