Although in-band full-duplex (IBFD) communication can theoretically double the spectral efficiency compared with conventional wireless communication technologies, it cannot be realized without reducing a high-power self-interference (SI) signal. Since the SI signal is affected by nonlinear distortion of radio-frequency (RF) circuits of its own terminal, cancellation systems capable of removing the nonlinear distorted SI have been actively developed. Detailed theoretical analysis based on the authors' past literature shows that for IBFD using a nonlinear self-interference canceller, the symbol error rate is improved when the power amplifier has certain distortion characteristics rather than being ideally linearized. However, it is not yet known what amplifier characteristics are best for the IBFD system. In this study, we theoretically analyze the self-interference signal to clarify the optimum amplifier characteristics for the IBFD system. An optimization problem to obtain an optimum transfer function from theoretical analysis is formulated as an Rayleigh quotient with constraints, and we introduce a numerical technique to solve the optimization problem. By numerical experiments, we show that the SI cancellation performance, signal-tointerference-distortion-plus-noise ratio (SIDNR), and amplifier's power efficiency are improved by the optimization.