In a previous paper (Gavignaud et al. 2006, A&A, 457, 79), we presented the type-1 Active Galactic Nuclei (AGN) sample obtained from the first epoch data of the VIMOS-VLT Deep Survey (VVDS). The sample consists of 130 faint, broad-line AGN with redshift up to z = 5 and 17.5 < I AB < 24.0, selected on the basis of their spectra. The sample is thus free of the morphological and color selection biases, that lead to significant incompleteness in the optical surveys of faint AGN. In this paper we present the measurement of the Optical Luminosity Function up to z = 3.6 derived from this sample, we compare our results with previous results from brighter samples both at low and at high redshift and finally, through the estimate of the bolometric luminosity function, we compare them also with the results from X-ray and mid-IR selected samples. Our data, more than one magnitude fainter than previous optical surveys, allow us to constrain the faint part of the luminosity function up to high redshift. A comparison of our data with the 2dF sample at low redshift (1 < z < 2.1) shows that the VVDS data can not be well fitted with the PLE models derived by previous samples. Qualitatively, this appears to be due to the fact that our data suggest the presence of an excess of faint objects at low redshift (1.0 < z < 1.5) with respect to these models. By combining our faint VVDS sample with the large sample of bright AGN extracted from the SDSS DR3 (Richards et al. 2006b, AJ, 131, 2766 and testing a number of different evolutionary models, we find that the model which better represents the combined luminosity functions, over a wide range of redshift and luminosity, is a luminosity dependent density evolution (LDDE) model, similar to those derived from the major Xsurveys. Such a parameterization allows the redshift of the AGN space density peak to change as a function of luminosity and explains the excess of faint AGN that we find at 1.0 < z < 1.5. On the basis of this model we find, for the first time from the analysis of optically selected samples, that the peak of the AGN space density shifts significantly towards lower redshift going to lower luminosity objects. The position of this peak moves from z ∼ 2.0 for M B < −26.0 to z ∼ 0.65 for −22 < M B < −20. This result, already found in a number of X-ray selected samples of AGN, is consistent with a scenario of "AGN cosmic downsizing", in which the density of more luminous AGN, possibly associated to more massive black holes, peaks earlier in the history of the Universe (i.e. at higher redshift), than that of low luminosity ones, which reaches its maximum later (i.e. at lower redshift).