The impact of optical receiver configuration on in-band crosstalk-induced penalty has been investigated in both theoretical and experimental analyses, for differential phase-shift keying (DPSK) and differential quadrature phase-shift keying (DQPSK) signals. Previously it has been shown that DPSK signals are ~6 dB more tolerant to in-band crosstalk than on-off keying (OOK) signals. However, we find that the tolerance difference between the two signals is reduced to ~3 dB when the decision threshold of the receiver is optimized to minimize the bit-error rate for each signal. Then we derive simple equations for the in-band crosstalk-induced penalty in DPSK and DQPSK signals with two different optical receiver configurations: balanced and single-ended direct-detection receivers. We confirm that the penalties obtained from our simple equations agree well with the measured results.