Acoustic Simulations of Cochlear Implants in Human and Machine Hearing Research

Abstract

IntroductionCochlear implant is an instrument which can be implanted in the inner ear and can restore partial hearing to profoundly deaf people (Loizou, 1999a) (see Fig. 1). Acoustic simulations of cochlear implants are widely used in cochlear implant research. Basically, they are acoustic signals which simulate what the profoundly deaf people could hear when they wear cochlear implants. Useful conclusions can be deduced from the results of experiments performed with acoustic simulations of cochlear implants. There are two typical applications in cochlear implant research which use acoustic simulations of cochlear implants. In the first one, acoustic simulations of cochlear implants are used to define how many independent channels are needed in order to achieve high levels of speech understanding (Loizou et al., 1999). The second application of acoustic simulations in cochlear implants research is for determining the effect of electrode insertion depth on speech understanding (Baskent & Shannon, 2003;Dorman et al., 1997b). In this chapter, we review briefly these conventional applications of acoustic simulations in cochlear implants research and, on the other hand, introduce novel applications of acoustic simulations of cochlear implants, both in cochlear implants research and in other domains, such as automatic speech recognition (ASR) research. To this end, we present quantitative analyses on the fundamental frequency (F0) of the cochlear implant-like spectrally reduced speech (SRS) which are, essentially, acoustic simulations of cochlear implants (Loizou, 1999a). These analyses support the report of , which was based on subjective tests, about the difficulty of cochlear implant users in identifying speakers. Following the results of our analyses, the F0 distortion in state-of-the-art cochlear implants is large when the SRS, which is acoustic simulation of cochlear implant, is synthesized only from subband temporal envelopes (Do, Pastor & Goalic, 2010a). The analyses revealed also a significant reduction of F0 distortion when the frequency modulation is integrated in cochlear implant, as proposed by . Consequently, the results of such quantitative analyses, performed on relevant acoustic traits, could be exploited to conduct subjective studies in cochlear implant research. On the other hand, we investigate the automatic recognition of the cochlear implant-like SRS. Actually, state-of-the-art ASR systems rely on relevant spectral information, extracted from original speech signals, to recognize input speech in a statistical pattern recognition framework. We show that from certain SRS spectral resolution, it is possible to achieve (automatic) recognition performance as good as that attained with the original clean speech even though the cochlear implant-like SRS is synthesized only from subband temporal envelopes of the original clean speech

Temporal Filterbanks in Cochlear Implant Hearing and Deep Learning Simulations

“…Compared to band envelope features (Figure 4), Table 3 shows AM and FM lowered the error rate in GMM systems used during forced alignment to generate frame-level DNN training labels. These results expand upon [31] and provide additional evidence that FAME preserves more of the relevant detail compared to other carriers. Table 4 shows error rates for GMM systems trained and tested on an additive configuration of ⧠⧠ (MFCC + FAME) (Figure 12).…”

“…Many computational algorithms [9,27,28,29,30,31,35,40,41,42] have been inspired by auditory processing pathways in the human nervous system. Traditionally, the critical band theory is commonly accepted for baseline DNN features [35] due to the ability of MFCC and FBANK to allow better suppression of insignificant spectral variation in the higher-frequency bands.…”

“…In [30], cochlear implant speech synthesized from subband temporal envelope was shown to contain sufficient information to rival MFCC features in terms of accuracy. These acoustic simulations of cochlear implants [31] were subsequently proposed as general indicators to conduct useful subjective studies. In [32], the cross-disciplinary methods of cognitive science and machine learning were converged to promote the shared views of computational [33] foundations.…”

“…Designing a representation that both preserves relevant detail in the speech signal and also provides stability/invariance to distortions is a nontrivial task. Therefore, Figure 11 derives slowly varying amplitude modulation (AM) and frequency modulation (FM) from speech to design novel features (referred to as frequency amplitude modulation encoding "FAME") with different modulations, as proposed in [42] and computed in ASR [31,39]. In the FAME condition, the FM is smoothed in terms of both rate and depth and then modulated by the AM.…”

Simulated output of Continuous Interleaved Sampling and Frequency Amplitude Modulation Encoding Techniques used in Cochlear Implant

“…The AM pathway involves full-wave rectification followed by lowpass filtering at 500 Hz to obtain the slowly varying envelope. The FM pathway involves removal of each narrow band's center frequency followed by low-pass filtering of the FM bandwidth with a cutoff of 500 Hz (Fig 1 and 2 will be print at same place) [4,5].…”