We present a theory of valence holes as Luttinger spinor based qubits in p-doped self-assembled quantum dots within the four-band k · p formalism. The two-qubit levels are identified with the two chiralities of the doubly degenerate ground state. We show that single-qubit operations can be implemented with static magnetic field applied along the z axis ͑growth direction͒ for z operation and with magnetic field in the quantum dot plane, x direction, for x operation. The coupling of two dots and hence the double-qubit operations are shown to be sensitive to the orientation of the two quantum dots. For vertical qubit arrays, there exists an optimal qubit separation suitable for the voltage control of qubit-qubit interactions.