We investigate the second-order optical response of graphene irradiated by two-color lights with frequencies
ω
and
3
ω
, focusing on the modulation by ellipticity and phase of the incident fields. Based on the analytical perturbation theory, it is found that the interplay between the second harmonic generation (SHG) (
ω
+
ω
) and difference frequency generation (DFG) (
3
ω
−
ω
) processes leads to nontrivial nonlinear optical properties. The interference between the two processes (SHG and DFG) results in different dependencies of the second-order response intensity
I
(
2
ω
)
on ellipticity, including monotonic, nonmonotonic change of
I
(
2
ω
)
with ellipticity, or even nearly independent of ellipticity under suitable conditions. The optimal incident angles for the highest
I
(
2
ω
)
are found, which can be tuned by changing the ellipticity, relative phase, or field strength of the incident light. In particular, a large output enhancement with respect to a small increment of the incident field can be achieved. We have found effective methods of modulating nonlinear optical processes, which may have applications in optical devices based on two-dimensional materials.