In this paper, we design an optical chaotic secure communication system with multi-dimensional keys and a complex entropy source, which can allow secure communication working in different modes such as unicast, multicast, and broadcast. In the proposed scheme, amplified spontaneous emission (ASE) noise of the erbium-doped fiber amplifier (EDFA) with sufficient randomness is used as an entropy source to improve the security of the system. And by using the electro-optic delay feedback loops, the key spaces are introduced and the phase distortion is generated, then the data are masked in both phase and intensity fields. As a frequency-dependent group delay (FDGD) module, Gires-Tournois interferometer (G-T I) in the feedback loop can effectively hide the time delay signature (TDS) and achieve additional 10 36 key spaces. The influence of different key mismatches on decryption function is numerically investigated, and the results indicate that the broadband nature of ASE noise makes the TDS a very sensitive encryption key, which increases the difficulty of illegal third parties stealing information. And with the help of the wavelength converter and wavelength division multiplexer, secure communication in various modes is successfully realized, which greatly improves the communication efficiency and is suitable for various security places.