A combination of electron cooling and rf system is an effective method to compress the beam bunch length in storage rings. A simulation code based on multiparticle tracking was developed to calculate the bunched ion beam cooling process, in which the electron cooling, intrabeam scattering (IBS), ion beam space-charge field, transverse and synchrotron motion are considered. Meanwhile, bunched ion beam cooling experiments have been carried out in the main cooling storage ring (CSRm) of the Heavy Ion Research Facility in Lanzhou, to investigate the minimum bunch length obtained by the cooling method, and study the dependence of the minimum bunch length on beam and machine parameters. The experiments show comparable results to those from simulation. Based on these simulations and experiments, we established an analytical model to describe the limitation of the bunch length of the cooled ion beam. It is observed that the IBS effect is dominant for low intensity beams, and the spacecharge effect is much more important for high intensity beams. Moreover, the particles will not be bunched for much higher intensity beam. The experimental results in CSRm show a good agreement with the analytical model in the IBS dominated regime. The simulation work offers us comparable results to those from the analytical model both in IBS dominated and space-charge dominated regimes.