Context. Thanks to the Venus Express Mission, new data on the properties of Venus could be obtained, in particular concerning its rotation. Aims. In view of these upcoming results, the purpose of this paper is to determine and compare the major physical processes influencing the rotation of Venus and, more particularly, the angular rotation rate. Methods. Applying models already used for Earth, the effect of the triaxiality of a rigid Venus on its period of rotation are computed. Then the variations of Venus rotation caused by the elasticity, the atmosphere, and the core of the planet are evaluated. Results. Although the largest irregularities in the rotation rate of the Earth on short time scales are caused by its atmosphere and elastic deformations, we show that the irregularities for Venus are dominated by the tidal torque exerted by the Sun on its solid body. Indeed, as Venus has a slow rotation, these effects have a large amplitude of two minutes of time (mn). These variations in the rotation rate are greater than the one induced by atmospheric wind variations that can reach 25-50 s of time (s), depending on the simulation used. The variations due to the core effects that vary with its size between 3 and 20 s are smaller. Compared to these effects, the influence of the elastic deformation caused by the zonal tidal potential is negligible. Conclusions. As the variations in the rotation of Venus reported here are close to 3 mn peak to peak, they should influence past, present, and future observations, thereby providing further constraints on the planet's internal structure and atmosphere.