ATTENTION: The software behind KU ScholarWorks is being upgraded to a new version. Starting July 15th, users will not be able to log in to the system, add items, nor make any changes until the new version is in place at the end of July. Searching for articles and opening files will continue to work while the system is being updated.
If you have any questions, please contact Marianne Reed at mreed@ku.edu .
Plain-to-clear speech video conversion for enhanced intelligibility
dc.contributor.author | Sachdeva, Shubam | |
dc.contributor.author | Ruan, Haoyao | |
dc.contributor.author | Hamarneh, Ghassan | |
dc.contributor.author | Behne, Dawn M. | |
dc.contributor.author | Jongman, Allard | |
dc.contributor.author | Sereno, Joan A. | |
dc.contributor.author | Wang, Yue | |
dc.date.accessioned | 2023-04-10T16:50:43Z | |
dc.date.available | 2023-04-10T16:50:43Z | |
dc.date.issued | 2023-01-28 | |
dc.identifier.citation | Sachdeva, S., Ruan, H., Hamarneh, G. et al. Plain-to-clear speech video conversion for enhanced intelligibility. Int J Speech Technol 26, 163–184 (2023). https://doi.org/10.1007/s10772-023-10018-z | en_US |
dc.identifier.uri | https://hdl.handle.net/1808/34078 | |
dc.description.abstract | Clearly articulated speech, relative to plain-style speech, has been shown to improve intelligibility. We examine if visible speech cues in video only can be systematically modified to enhance clear-speech visual features and improve intelligibility. We extract clear-speech visual features of English words varying in vowels produced by multiple male and female talkers. Via a frame-by-frame image-warping based video generation method with a controllable parameter (displacement factor), we apply the extracted clear-speech visual features to videos of plain speech to synthesize clear speech videos. We evaluate the generated videos using a robust, state of the art AI Lip Reader as well as human intelligibility testing. The contributions of this study are: (1) we successfully extract relevant visual cues for video modifications across speech styles, and have achieved enhanced intelligibility for AI; (2) this work suggests that universal talker-independent clear-speech features may be utilized to modify any talker’s visual speech style; (3) we introduce “displacement factor” as a way of systematically scaling the magnitude of displacement modifications between speech styles; and (4) the high definition generated videos make them ideal candidates for human-centric intelligibility and perceptual training studies. | en_US |
dc.publisher | Springer | en_US |
dc.rights | © The Author(s) 2023. This article is licensed under a Creative Commons Attribution 4.0 International License. | en_US |
dc.rights.uri | http://creativecommons.org/licenses/by/4.0/ | en_US |
dc.subject | Video speech synthesis | en_US |
dc.subject | Speech style | en_US |
dc.subject | Intelligibility | en_US |
dc.subject | AI lip reading | en_US |
dc.subject | Speech enhancement | en_US |
dc.title | Plain-to-clear speech video conversion for enhanced intelligibility | en_US |
dc.type | Article | en_US |
kusw.kuauthor | Jongman, Allard | |
kusw.kuauthor | Sereno, Joan A. | |
kusw.kudepartment | Linguistics | en_US |
dc.identifier.doi | 10.1007/s10772-023-10018-z | en_US |
dc.identifier.orcid | https://orcid.org/0000-0003-3862-3767 | en_US |
kusw.oaversion | Scholarly/refereed, publisher version | en_US |
kusw.oapolicy | This item meets KU Open Access policy criteria. | en_US |
dc.identifier.pmid | PMC10042924 | en_US |
dc.rights.accessrights | openAccess | en_US |