We investigate a two-electron double quantum dot with both spin and valley degrees of freedom as they occur in graphene, carbon nanotubes or silicon and regard the 16-dimensional space with one electron per dot as a four-qubit logic space. In the spin-only case, it is well known that the exchange coupling between the dots combined with arbitrary single-qubit operations is sufficient for universal quantum computation. The presence of valley degeneracy in the electronic band structure alters the form of the exchange coupling and, in general, leads to spin-valley entanglement. Here, we show that universal quantum computation can still be performed by exchange interaction and singlequbit gates in the presence of an additional (valley) degree of freedom. We present an explicit pulse sequence for a spin-only controlled-NOT consisting of the generalized exchange coupling and single-electron spin and valley rotations. We also propose state preparations and projective measurements with the use of adiabatic transitions between states with (1,1) and (0,2) charge distributions similar to the spin-only case, but with the additional requirement of controlling the spin and valley Zeeman energies by an external magnetic field. Finally, we demonstrate a universal two-qubit gate between a spin and a valley qubit, allowing universal gate operations on the combined spin and valley quantum register.