We consider the problem of designing demodulators for linear vector channels with memory that use reduced-size trellis descriptions for the received signal. We assume an overall iterative receiver, and use interference cancellation (IC) based on the soft information provided by the outer decoder, to mitigate the parts of the signal that are not covered by the reduced-size trellis description. In order to reach a trellis description, a linear filter is applied as front-end to compress the signal structure into a small trellis. This process requires three parameters to be designed: (i) the front-end filter, (ii) the feedback filter through which the IC is done, and (iii) a target response which specifies the trellis.Demodulators of this form have been studied before under then name channel shortening (CS), but the interplay between CS, IC and the trellis-search process has not been adequately addressed in the literature. In this paper, we analyze two types of CS demodulators that are based on the Forney and Ungerboeck detection models, respectively. The parameters are jointly optimized based on a generalized mutual information (GMI) function. We also introduce a third type of CS demodulator that is in general suboptimal, but has closed-form solutions. Moreover, signal to noise ratio (SNR) asymptotic properties are analyzed and we show that the third CS demodulator asymptotically converges to the optimal CS demodulator in the sense of GMI-maximization.
Index TermsChannel shortening, intersymbol interference, prefilter, front-end filter, feedback filter, target response, generalized mutual information, achievable information rate, Forney model, Ungerboeck model. The authors are with the Department of Electrical and Information Technology, Lund University, Lund, Sweden (email: firstname.lastname@eit.lth.se). This paper has been presented in part [53] at the IEEE 26th annual international symposium on personal, indoor and mobile radio communications (PIMRC), Hongkong, Sep. 2015.