It is widely believed that the spin of black holes in X-ray binaries is mainly natal. A significant spin-up from accretion is not possible. If the secondary has a low mass, the black hole spin cannot change too much even if the black hole swallows the whole stellar companion. If the secondary has a high mass, its lifetime is too short to transfer the necessary amount of matter and spin the black hole up. However, while black holes formed from the collapse of a massive star with solar metallicity are expected to have low birth spin, current spin measurements show that some black holes in X-ray binaries are rotating very rapidly. Here we show that, if these objects are not the Kerr black holes of general relativity, the accretion of a small amount of matter (∼2 M ) can make them look like very fast-rotating Kerr black holes. Such a possibility is not in contradiction with any observation and it can explain current spin measurements in a very simple way.