Mitigating Cochlear Implant Stimulus Pulses Dominated by Reverberant Distortions

Cochlear implant (CI) users can experience considerable degradations in speech intelligibility in reverberant environments (Kressner et al., 2018). Reverberation occurs when sound reflects off of surfaces in an enclosure, with reverberant reflections arriving at the listener at the same time as the direct speech signal. CI speech processing transforms the amplitude of frequency envelopes into discrete stimulus pulses across electrode channels. When the speech envelope is degraded by the simultaneous arrival of reverberant reflections, speech intelligibility is degraded. Removing reverberant artifacts from either the reverberant speech signal (Hazrati et al., 2013b, 2013a; Hazrati and Loizou, 2013; Kokkinakis et al., 2011) or the reverberant stimulus pattern (Desmond, 2014; Desmond et al., 2014) can restore speech envelope structure and significantly recover speech intelligibility for CI users in reverberation. The mitigation strategies applied to the reverberant speech signal used a measure of local signal distortion to indicate speech-dominant portions of the reverberant signal. The measure of signal distortion required a priori knowledge of the clean speech signal or the reverberant room. A few studies proposed statistical signal processing techniques to estimate the measure of distortion (Hazrati et al., 2013b, 2013a; Hazrati and Loizou, 2013). Reverberant speech mitigated by these strategies resulted in improvements in speech intelligibility for CI users, indicating that a measure of reverberant distortion can identify portions of the reverberant signal that are detrimental to intelligibility for CI users. Unfortunately, mitigation prior to CI processing presents limitations for real-time feasibility, imposing a delay on stimulation at the electrode array. For CI users to receive benefit from a mitigation algorithm, the delay imposed by reverberation mitigation must be minimized to avoid audio-visual synchrony (Hay-McCutcheon et al., 2009). Further, since CI users are likely to encounter a number of reverberant scenarios in their daily lives, mitigation of reverberant stimuli must provide intelligibility improvements in a range of listening scenarios. The approaches in (Hazrati et al., 2013b, 2013a; Hazrati and Loizou, 2013) relied on heuristic condition-specific tuning of parameters, yielding algorithm performance that may not be robust to changes in reverberant conditions. An alternative approach to mitigating the effects of reverberation on intelligibility for CI users leveraged a machine learning algorithm to identify reverberant artifacts due to temporal masking within the CI stimulus (Desmond, 2014). The machine learning artifact detection algorithm harnessed a data-driven approach to capture the variability present in the training data and provide artifact detection in a variety of reverberant environment. The approach in (Desmond, 2014) led to improved intelligibility in CI users and demonstrated good generalization to reverberant conditions not contained in the training set. Although temporal masking-based mitigation can remove reverberant artifacts between phonemes, the reverberant distortions within phonemes imposed in highly reverberant listening scenarios can be responsible for significant degradations in intelligibility for CI users (Desmond et al., 2014; Kokkinakis and Loizou, 2011a). Given that a measure of reverberant distortion can identify reverberant speech components that are harmful to intelligibility for CI users both within phonemes and after phoneme termination, distortion-based mitigation strategies are likely to yield intelligibility improvements in a variety of reverberant listening scenarios when compared to temporal masking-based mitigation. We combine the objective of the distortion-based criterion with the data-driven approach of machine learning to enable improved intelligibility for CI users in a range of realistic reverberant listening scenarios. To minimize the delay imposed by the artifact detection algorithm, only causal information is used and removal of reverberant artifacts was implemented at the pulse level. To ensure that the distortion-based mitigation strategy results in the greatest intelligibility benefits, we conduct an intelligibility study to determine the tolerance of intelligibility to the amount of reverberant distortion present in each stimulus pulse. The tolerance for the amount of reverberant distortion present in the mitigated signal was varied and a balance between reverberant artifact removal and speech signal retention was identified. To assess the efficacy of reverberant artifact removal, the intelligibility of the mitigated stimulus pattern must be assessed. Although online intelligibility testing is the gold standard for intelligibility assessment, it can be costly and time-intensive, prohibiting the rapid development of reverberant artifact detection algorithms. This work extends previous analyses of offline intelligibility measures (Falk et al., 2015; Santos et al., 2013) to predict the performance of mitigated reverberant CI stimuli by including the most widely adopted offline measures used in the literature and introducing new speech intelligibility data for validation. The correlation of offline measure scores to online intelligibility results is reported, indicating offline measures that leverage speech features relevant to the intelligibility of mitigated reverberant CI stimuli, and adjustments to offline measure evaluation are proposed to improve measure performance when applied to reverberant CI-processed signals. After validating the performance of offline intelligibility measures, we investigate the use of various machine learning models with different parametric complexities for identifying CI stimulus pulses dominated by reverberant distortion. To minimize the delay imposed by the algorithms, only causal frames of information are used. To test the algorithm’s robustness to unseen reverberant conditions, the models are tested in reverberant environments not encountered during training. We test the speech intelligibility of the CI stimulus after mitigation by each artifact detection algorithm with normal hearing subjects using a simulation of CI perception. We find that artifact detection performance improves significantly when models are trained on speech material spoken in a reverberant environment that closely resembles the target reverberant environment. Additionally, we demonstrate a trade-off between the computational complexity of the model and the ability of the model to generalize to speech spoken in unseen reverberant environments. As it would be infeasible to train one machine learning model to mitigate the effects of reverberation for every room a CI user might encounter, we investigate characteristics of reverberant rooms that may indicate similarities in artifact addition across different reverberant environments. We demonstrate the feasibility of applying specialized artifact detection models, termed room-matching models, to improve artifact detection performance in a range of realistic reverberant listening scenarios. To determine similarities in reverberant listening scenarios, room-matching models leverage measures of acoustic reverberation, estimated for the target listening scenario. We find that the reverberation time and clarity index measures reflect trends in reverberant artifact addition, echoing the findings that these acoustic measures indicate trends in intelligibility for CI users (Badajoz-Davila et al., 2020; Kokkinakis et al., 2011; Kressner et al., 2018). Overall, the findings from this research project indicate a framework for identifying reverberant artifacts in cochlear implant stimuli, which has the potential to improve intelligibility outcomes for cochlear implant users in a variety of real-world reverberant listening conditions.