Textless Acoustic Model with Self-Supervised Distillation for Noise-Robust Expressive Speech-to-Speech Translation

Min-Jae Hwang, Ilia Kulikov, Benjamin Peloquin, Hongyu Gong, Peng-Jen Chen, and Ann Lee

Abstract

In this paper, we propose a textless acoustic model with a self-supervised distillation strategy for noise-robust expressive speech-to-speech translation (S2ST). Recently proposed expressive S2ST systems have achieved impressive expressivity preservation performances by cascading unit-to-speech (U2S) generator to the speech-to-unit translation model. However, these systems are vulnerable to the presence of noise in input speech, which is an assumption in real-world translation scenarios. To address this limitation, we propose a U2S generator that incorporates a DINO self-supervised training strategy into it's pretraining process. Because the proposed method captures noise-agnostic expressivity representation, it can generate qualified speech even in noisy environment. Objective and subjective evaluation results verified that the proposed method significantly improved the performance of the expressive S2ST system in noisy environments while maintaining competitive performance in clean environments.

Systems

We provide source speech as well as audio samples from four systems:
(1) PRETSSEL: We combined a Prosody UnitY2 and PRETSSEL model [1].
(2) PRETSSEL + Denoiser: We combined a Prosody UnitY2 and PRETSSEL with high-quality speech enhancement model. Specifically, we applied MetricGAN+ denoiser [3] to the input of PRETSSEL for removing noise components.
(3) DINO-PRETSSEL (proposed): We combined a Prosody UnitY2 and proposed DINO-PRETSSEL.

S2ST using Benchmarking Dataset
mExpresso English-to-Spanish [1]
Source PRETSSEL (conventional) PRETSSEL + Denoiser DINO-PRETSSEL (proposed)
Sample 1
Clean environment
Noisy environment
Sample 2
Clean environment
Noisy environment

mDRAL Spanish-to-English [1]
Source PRETSSEL (conventional) PRETSSEL + Denoiser DINO-PRETSSEL (proposed)
Sample 1
Clean environment
Noisy environment
Sample 2
Clean environment
Noisy environment

S2ST using Authors' Speech
English-to-Spanish
Source PRETSSEL (conventional) PRETSSEL + Denoiser DINO-PRETSSEL (proposed)
Sample 1
Clean environment
Noisy environment
Sample 2
Clean environment
Noisy environment

Spanish-to-English
Source PRETSSEL (conventional) PRETSSEL + Denoiser DINO-PRETSSEL (proposed)
Sample 1
Clean environment
Noisy environment
Sample 2
Clean environment
Noisy environment
Template based on Textless NLP and HiFi-GAN pages.
References
[1] Seamless Communication, “Seamless: Multilingual Expressive and Streaming Speech Translation,” arXiv, 2023.
[2] Jörgen Valk and Tanel Alumäe, “VoxLingua107: a dataset for spoken language recognition,” In Proc. IEEE SLT Workshop, 2021.
[3] Szu-Wei Fu et al., "MetricGAN+: An Improved Version of MetricGAN for Speech Enhancement," arXiv, 2021.