We investigate the properties of an impurity particle interacting with a Fermi gas in a Cherninsulating state. The interaction leads to the formation of an exotic polaron, which consists of a coherent superposition of the topologically-trivial impurity and the surrounding topological cloud. We characterize this intriguing topologically-composite object by calculating its transverse (Hall) conductivity, using diagrammatic as well as variational methods. The "polaronic Hall conductivity", i.e., the transverse drag exerted by the dressing cloud on the impurity, is shown to exhibit a sharp jump from zero to a finite value whenever the surrounding cloud enters a topologically non-trivial state. In this way, the polaron partially inherits the topological properties of the Chern insulator through genuine interaction effects. This is also analysed at the microscopic level of wave functions, by identifying a "composite Berry curvature" for the polaron, which closely mimics the Berry curvature of the Chern insulator's band structure. Finally, we discuss how this interplay between topology and many-body correlations can be studied in cold-atom experiments, using available technologies.