Analysis on the channel capacity of a coherent optical subcarrier multiplexing system using optical amplifiers is presented. The performance criterion, that is, the channel transport capability N, is derived in terms of carrier‐to‐noise ratio, input optical power, and optical amplifier gain. It is found that when the gain of the optical amplifier is large, the spontaneous emission noise limits the number of channels that can be transmitted in this system. As an example, when the gain is well above 20 dB and input optical power is −28 dBm, the maximum value of N is 46. In addition, the optimum phase modulation index is also presented. © 1995 John Wiley & Sons, Inc.