We study the effect of valence band spin-orbit interactions (SOI) on the acoustic phonon-assisted spin relaxation of holes confined in quantum dots (QDs). Heavy hole-light hole (hh-lh) mixing and all the spin-orbit terms arising from zinc-blende bulk inversion asymmetry (BIA) are considered on equal footing in a fully three-dimensional Hamiltonian. We show that hh-lh mixing and BIA have comparable contributions to the hole spin relaxation in self-assembled QDs, but BIA becomes dominant in gated QDs. Simultaneously accounting for both mechanisms is necessary for quantitatively correct results in quasi-two-dimensional QDs. The dependence of the hole spin relaxation on the QD geometry and spin splitting energy is drastically different from that of electrons, with a non-monotonic behavior which results from the interplay between SOI terms. Our results reconcile contradictory predictions of previous theoretical works and are consistent with experiments. For a qualitative understanding of the geometry dependence of 1/T h 1 , we rewrite the hole states, equation (14), as | h m = J z c m J z |φ m J z |3/2, J z , where |φ m J z = r c m J z ,r |r is the envelope function associated with the periodic Bloch function |3/2, J z . If we restrict to the diagonal components of H h-ph , the matrix element determining the relaxation rate becomes