We study decoherence of an electron spin qubit in a quantum dot due to charge noise. We find that at the lowest order, the pure dephasing channel is suppressed for both 1/f charge noise and Johnson noise, so that charge noise leads to a pure relaxation channel of decoherence. Because of the weaker magnetic field dependence, the spin relaxation rate due to charge noise could dominate over phonon noise at low magnetic fields in a gate-defined GaAs or Si quantum dot or a InAs self-assembled quantum dot. Furthermore, in a large InAs self-assembled quantum dot, the spin relaxation rate due to phonon noise could be suppressed in high magnetic field, and the spin relaxation due to charge noise could dominate in both low and high magnetic field. Numerically, in a 1 Tesla field, the spin relaxation time due to typical charge noise is about 100 s in Si, 0.1 s in GaAs for a gate-defined quantum dot with a 1 meV confinement, and 10 µs in InAs self-assembled quantum dot with a 4 meV confinement.