We consider Shiraishi's metrics on the moduli space of extreme black holes.We interpret the simplification in the pattern of N-body interactions that he observed in terms of the recent picture of black holes in four and five dimensions as composites, made up of intersecting branes. We then show that the geometry of the moduli space of a class of black holes in five and nine dimensions is hyper-Kähler with torsion, and octonionic-Kähler with torsion, respectively. For this, we examine the geometry of point particle models with extended world-line supersymmetry and show that both of the above geometries arise naturally in this context. In addition, we construct a large class of hyper-Kähler with torsion and octonionic-Kähler with torsion geometries in various dimensions. We also present a brane interpretation of our results.