Investigation of the Effectiveness of Converted Speech Auditory Feedback in Low-Latency Real-Time Voice Conversion

Low-latency real-time voice conversion (VC) technique transforming voice characteristics while preserving linguistic content allows users to freely speak with desired target voices. This technique can instantaneously present auditory feedback of the converted speech to the users, enabling them to adapt their vocalization to the VC system. On the other hand, it is also possible that the converted auditory feedback causes any adverse effects, such as stuttering, similar to those caused by delayed auditory feedback (DAF). In this study, we investigate the effectiveness of the converted auditory feedback using eight participants in a real-time VC system with 77 ms end-to-end latency. We compare real-time (RT) feedback conditions where participants hear converted speech during utterance with recording (Rec) feedback conditions where participants hear their converted speech after utterance completion. Experimental results demonstrate that the converted auditory feedback yields statistically significant improvements in both speech naturalness (UT-MOS) and speaker similarity. Objective analysis reveals that users subconsciously increase their fundamental frequency (F0) variability to better match the VC model, while Character Error Rate (CER) analysis confirms no adverse DAF-like effects.