Trace Controlled Text to Image Generation

Abstract

Figure 1. (a) ReCo extends pre-trained text-to-image models (Stable Diffusion [33]) with an extra set of input position tokens (in dark blue color) that represent quantized spatial coordinates. Combining position and text tokens yields the region-controlled text input, whichcan specify an open-ended regional description precisely for any image region. (b) With the region-controlled text input, ReCo can better control the object count/relationship/size properties and improve the T2I semantic correctness. We empirically observe that position tokens are less likely to get overlooked than positional text words, especially when the input query is complicated or describes an unusual scene.