An optimized millimeter-wave digital controlled oscillator (DCO) in a 40-nm CMOS process is presented in this work. The coarse-tuning modules and medium-tuning modules of the DCO utilize modified binary-weighted digitally controlled transmission lines (DCTLs) to achieve a better compromise among smaller chip size, higher resonant frequency, better tuning resolution and lower phase noise. The tuning precision and die size of the medium tuning bank are improved without changing the binary coding rules by replacing the lowest-weight bit of the DCTLs with switched capacitors. In comparison with traditional DCTLs, the control bits of the coarse and medium tuning modules have been changed from 30 to 8, resulting in a 34.4% reduction in overall length (from 122$$\upmu$$
μ
m to 80$$\upmu$$
μ
m). In addition, the DCO’s fine-tuning modules are achieved using a binary-weighted switched capacitors array connected to the secondary winding of a low-coupling transformer, which enhances the DCO’s fine-tuning bank for better frequency resolution with less circuit complexity. The measured tuning range of the optimized DCO is 76-81GHz with a smaller die size of 0.12mm$$^2$$
2
. This results in an outstanding figure of merit ($$FoM_A$$
F
o
M
A
) of − 190.52dBc/Hz.