The reconstruction of the polarization of a source in radio interferometry is a challenging calibration problem since the reconstruction strongly depends on the gains and leakages, which need to be inferred along with the image. This is particularly true for the Event Horizon Telescope (EHT) due to its small number of antennas, low signal-to-noise ratio, and large gain corruptions. To recover linear polarization, one has to either infer the leakages and gains together with the image structure or rely completely on calibration-independent closure quantities. While the first approach has been explored in very long baseline interferometry (VLBI) for a long time, the latter has been less studied for polarimetry. Closure traces are a recently proposed concept of closure quantities that, in contrast to closure phases and closure amplitudes, are independent of both gains and leakages and carry the relevant information about the polarization of the source. Here we explore how closure traces can be directly fitted to create an image, and we identify an imaging pipeline that succeeds in direct imaging from closure traces. Since closure traces have a number of inherent degeneracies, multiple local image modes that can fit the data are detected. Therefore, a multi-objective imaging technique is needed to correctly sample this multi-modality. Closure traces are not constraining enough for the current EHT configuration to recover an image directly, mainly due to the small number of antennas. For planned successors of the EHT, however (with a significantly larger number of antennas), this option will become feasible and will be competitive with techniques that use imaging with residual leakages.