Supermassive black holes (SMBHs) are found in most galactic nuclei. A significant fraction of these nuclei also contain a nuclear stellar cluster (NSC) surrounding the SMBH. In this paper, we consider the idea that the NSC forms first, from the merger of several stellar clusters that may contain intermediate-mass black holes (IMBHs). These IMBHs can subsequently grow in the NSC and form an SMBH. We carry out N-body simulations of the simultaneous merger of three stellar clusters to form an NSC, and investigate the outcome of simulated runs containing zero, one, two and three IMBHs. We find that IMBHs can efficiently sink to the centre of the merged cluster. If multiple merging clusters contain an IMBH, we find that an IMBH binary is likely to form and subsequently merge by gravitational wave emission. We show that these mergers are catalyzed by dynamical interactions with surrounding stars, which systematically harden the binary and increase its orbital eccentricity. The seed SMBH will be ejected from the NSC by the recoil kick produced when two IMBHs merge, if their mass ratio q ≳ 0.15. If the seed is ejected then no SMBH will form in the NSC. This is a natural pathway to explain those galactic nuclei that contain an NSC but apparently lack an SMBH, such as M33. However, if an IMBH is retained then it can seed the growth of an SMBH through gas accretion and tidal disruption of stars.