Aims. We aim to explore the possible chemical signature of planet formation in the binary system HAT-P-4 by studying the trends of abundance vs. condensation temperature T c . The star HAT-P-4 hosts a planet detected by transits, while its stellar companion does not have any detected planet. We also study the lithium content, which might shed light on the problem of Li depletion in exoplanet host stars. Methods. We derived for the first time both stellar parameters and high-precision chemical abundances by applying a line-by-line full differential approach. The stellar parameters were determined by imposing ionization and excitation equilibrium of Fe lines, with an updated version of the FUNDPAR program, together with ATLAS9 model atmospheres and the MOOG code. We derived detailed abundances of different species with equivalent widths and spectral synthesis with the MOOG program. Results. The exoplanet host star HAT-P-4 is found to be ∼0.1 dex more metal rich than its companion, which is one of the highest differences in metallicity observed in similar systems. This could have important implications for chemical tagging studies. We rule out a possible peculiar composition for each star, such as is the case for λ Boötis and δ Scuti, and neither is this binary a blue straggler. The star HAT-P-4 is enhanced in refractory elements relative to volatile when compared to its stellar companion. Notably, the Li abundance in HAT-P-4 is greater than that of its companion by ∼0.3 dex, which is contrary to the model that explains the Li depletion by the presence of planets. We propose a scenario where at the time of planet formation, the star HAT-P-4 locked the inner refractory material in planetesimals and rocky planets, and formed the outer gas giant planet at a greater distance. The refractories were then accreted onto the star, possibly as a result of the migration of the giant planet. This explains the higher metallicity, the higher Li content, and the negative T c trend we detected. A similar scenario was recently proposed for the solar-twin star HIP 68468, which is in some aspects similar to HAT-P-4. We estimate a mass of at least M rock ∼ 10 M ⊕ locked in refractory material in order to reproduce the observed T c trends and metallicity.