An improved design of receiving filter for nonideally shaped modulated signals which provides better performance than existing schemes is proposed. By concerning both Inter-Symbol-Interference (ISI) and the degree of waveform mismatch between the transmitted signal and impulse response of the receiving filter, the exact expression of the Signal-to-Noise Ratio (SNR) loss that represents the performance degradation is derived, and the performance is compared with that of the ideally shaped signals' demodulation. The existing schemes such as root-raised-cosine (RRC) receiving filters don't perform well for non-ideally shaped situations, since large degrees of both ISI and waveform mismatch exist. To improve the performance and avoid the complexity of applying equalizer in receivers, the proposed scheme designs the impulse response of the receiving filter by optimizing the SNR loss to the minimum value, and it uses simulated annealing as the optimization algorithm. It is shown that the new method can achieve better performance than existing schemes, especially when the modulation order or the SNR is relatively high. Finally the conclusion is validated by simulation results.