Optical chaos has attracted widespread attention for its complex dynamics. However, the time delay signature (TDS) caused by the external cavity mode reduces the complexity of optical chaos. We propose and numerically demonstrate the critical dispersion of chirped FBG (CFBG) for eliminating the TDS of laser chaos. In this work, the critical dispersion as the function of relaxation frequency and bandwidth of the optical spectrum is found through extensive dynamics simulations. It is shown that the TDS can be eliminated when the dispersion of CFBG is above this critical dispersion. In addition, the influence of dispersive feedback light and output light from a laser is investigated. These results offer crucial and quantitative guidance for designing chaotic semiconductor lasers without TDS.