Automatic speech recognition to aid the hearing impaired: Prospects for the automatic generation of cued speech

R. M. Uchanski, L. A. Delhorne, A. K. Dix, L. D. Braida, C. M. Reed, N. I. Durlach

Research output: Contribution to journalArticlepeer-review

38 Scopus citations


Although great strides have been made in the development of automatic speech recognition (ASR) systems, the communication performance achievable with the output of current real-time speech recognition systems would be extremely poor relative to normal speech reception. An alternate application of ASR technology to aid the hearing impaired would derive cues from the acoustical speech signal that could be used to supplement speechreading. We report a study of highly trained receivers of Manual Cued Speech that indicates that nearly perfect reception of everyday connected speech materials can be achieved at near normal speaking rates. To understand the accuracy that might be achieved with automatically generated cues, we measured how well trained spectrogram readers and an automatic speech recognizer could assign cues for various cue systems. We then applied a recently developed model of audiovisual integration to these recognizer measurements and data on human recognition of consonant and vowel segments via speechreading to evaluate the benefit to speechreading provided by such cues. Our analysis suggests that with cues derived from current recognizers, consonant and vowel segments can be received with accuracies in excess of 80%. This level of performance is roughly equivalent to the segment reception accuracy required to account for observed levels of Manual Cued Speech reception. Current recognizers provide maximal benefit by generating only a relatively small number (three to five) of ene groups, and may not provide substantially greater aid to speechreading than simpler aids that do not incorporate discrete phonetic recognition. To provide guidance for the development of improved automatic cueing systems, we describe techniques for determining optimum cue groups for a given recognizer and speechreader, and estimate the cueing performance that might be achieved if the performance of current recognizers were improved.

Original languageEnglish
Pages (from-to)20-41
Number of pages22
JournalJournal of Rehabilitation Research and Development
Issue number1
StatePublished - 1994


  • Manual Cued Speech
  • automatic speech recognition
  • cueing performance
  • improved automatic cueing systems


Dive into the research topics of 'Automatic speech recognition to aid the hearing impaired: Prospects for the automatic generation of cued speech'. Together they form a unique fingerprint.

Cite this