Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen,
Changhan Wang, Sravya Popuri, Yossi Adi, Juan Pino, Jiatao Gu, Wei-Ning Hsu
[paper]
We present a textless speech-to-speech translation (S2ST) system that can translate speech from one language into another language and can be built without the need of any text data. Different from existing work in the literature, we tackle the challenge in modeling multi-speaker target speech and train the systems with real-world S2ST data. The key to our approach is a self-supervised unit-based speech normalization technique, which finetunes a pre-trained speech encoder with paired audios from multiple speakers and a single reference speaker to reduce the variations due to accents, while preserving the lexical content. With only 10 minutes of paired data for speech normalization, we obtain on average 3.2 BLEU gain when training the S2ST model on the VoxPopuli S2ST dataset, compared to a baseline trained on un-normalized speech target. We also incorporate automatically mined S2ST data and show an additional 2.0 BLEU gain. To our knowledge, we are the first to establish a textless S2ST technique that can be trained with real-world data and works for multiple language pairs.
Below we provide audio samples from four systems:
(1) S2T+T2U: a cascaded system with a speech-to-text translation model trained with ASR decoded text of the S2ST data, followed by a text-to-unit speech synthesizer,
(2) S2UT (orig_unit): a direct speech-to-unit translation (S2UT) model trained with un-normalized target units,
(3) S2UT (tgt_spk): a direct S2UT model trained with un-normalized target units and target speaker embedding,
(4) S2UT (sp_norm): a direct S2UT model trained with target units normalized with a speech normalizer trained with 1-hr of data.
For each source audio, we provide text transcription in the source language and the reference text translation in the target language.
For each audio from the four systems, we provide ASR decoded text.
| Source (Spanish) | Predictions | ||||
|---|---|---|---|---|---|
| S2T+T2U | S2UT (orig_unit) | S2UT (tgt_spk) | S2UT (sp_norm) | ||
| 1. | |||||
| (Es) La plena financiación comunitaria es fundamental para nosotros. (En) full community funding is fundamental for us |
full funding is vital for us | foll community funding is fundamental for us | full community funding is vital to us | full community funding is fundamental for us | |
| 2. | |||||
| (Es) Señorías, Europa debe situar a los profesionales y a los consumidores en el corazón del Mercado Único. (En) ladies and gentlemen europe must put professionals and consumers at the heart of the single market |
ladies and gentlemen europe has to place consumers at the heart of the single market | ladies and gentlemen you must place professionals and consumers at the hot of the single knotted | upe has to situate professionals and consumers in the heart of the single marted | ladies and gentlemen europe needs to place professionals and consumers at the heart of the single market | |
| 3. | |||||
| (Es) Señora Presidenta, Comisario, estimados colegas, sabemos que sin jóvenes no hay futuro y también tenemos la certeza de que, con las altas tasas de desempleo juvenil, Europa puede ser insostenible. (En) madam president commissioner ladies and gentlemen we know that without young people there is no future and we are also sure that with high rates of youth unemployment europe may become unsustainable |
commissioner colligues we know that without young people there's no future and we're certain that with the high youth unemployment rates europe can be unsustainable | president cme colleagues we know that without young people there is no future we also have the certainty that with the lack of youth unemployment rates up can be unsustainable | oisiner we know that without young people without the future and we also have the certainty that with the lack of youth unemployment europe can be unsustainable | commissioner colleagues we know that without young people there is no future and we also feel that with the lack of youth unemployment rates europe cannot be sustainable | |
| 4. | |||||
| (Es) Cuando estaba con su padre, se encontraba siempre en lugares rodeado de diferentes instrumentos. (En) when he was with his father he was always in places surrounded by different instruments |
when he was with his father he was always surrounded by various instruments | when he was with his father he always surrounded by different instruments | when he was with his father when he was always surrounded by different instruments | when he was with his father he was always in places surrounded by different instruments | |
| 5. | |||||
| (Es) Sin embargo, en la práctica el lugar está aún habitado. (En) however in practice the place is still inhabited |
however in practice the place is still inhabited | nevertheless the place is still inhabited | nevertheless the place is still inhabited | but in practice the place is still inhabited | |
| 6. | |||||
| (Es) Estas montañas limitan la expansión de la ciudad, hacia el norte. (En) those mountains limit the expansion of the city to the north |
these mountains limit the expansion of the city to the northward | these mountains confine the exportion of the northward to the north | these mountains confined the expansion of the city to the north | these mountains limit the expansion of the city to the north | |
| Source (French) | Predictions | ||||
|---|---|---|---|---|---|
| S2T+T2U | S2UT (orig_unit) | S2UT (tgt_spk) | S2UT (sp_norm) | ||
| 1. | |||||
| (Fr) Je vous remercie, Monsieur le Commissaire, mais en démocratie, il est aussi un autre principe. (En) thank you commissioner but in democracy there is also another principle |
president commissioner but in democracy its also are principal | commissioner but in democracy is also our principle | commisiner but in democracy it is also our principle | commissioner but in democracy there is also our principle | |
| 2. | |||||
| (Fr) Les petits États insulaires ont par exemple des capacités très limitées. (En) small island states have very limited capabilities for example |
small island states have very limited capacities | small states have limited capacities | small states have limited capacities | small island states have very limited capacities for example | |
| 3. | |||||
| (Fr) Face à ces catastrophes, l ’ Union européenne doit montrer sa réactivité et exprimer sa solidarité. (En) in the face of these disasters the european union must demonstrate its responsiveness and express its solidarity |
faced with these catastrophes the european union has to show its reactivity expressing its solidarity | facing these catastrophes the e u has to show its reactivity expressing its solidality | faced with its disasters the eu must when a show its reactivity express its solidarity | in the face of these disasters the european union needs to show its reactiviity to express its solidarity | |
| 4. | |||||
| (Fr) Mais plusieurs interruptions furent provoqués à cause des conflits internes dans le pays. (En) but several interruptions were caused because of internal conflicts in the country |
but several indiruptions were caused by internal conflicts in the country | but many interruptions were provoked because of the internal conflicts in the country | but several indiructions were provoked because of the internal conflicts in the country | but several interruptions were provoked because of internal conflicts in the country | |
| 5. | |||||
| (Fr) Une extension du réseau n'est pas envisageable dans ces conditions. (En) an extension of the network is not possible under these conditions |
an extension of the network is not envisaged under these conditions | an extension of the network is not faced under those conditions | an extension of the network was not faced under these conditions | an extension of the network is not considered under these conditions | |
| 6. | |||||
| (Fr) Le compositeur Louis Varney, qui a étudié la musique avec lui, était son fils. (En) composer louis varney who studied music with him was his son |
louise vanished who had studied music with him was his son | louis came who studied the music with him was his son | the composer louis vaney who studied music with him was his son | the composer louis varney who studied music with him was his sign | |
| Source (English) | Predictions | ||||
|---|---|---|---|---|---|
| S2T+T2U | S2UT (orig_unit) | S2UT (tgt_spk) | S2UT (sp_norm) | ||
| 1. | |||||
| (En) We must use this occasion – it is the first year of a five-year Parliament – to look ahead. (Es) debemos aprovechar esta ocasión es el primer año de un parlamento de cinco años de duración para mirar hacia delante |
TENEMOS QUE UTILIZAR ESTA OCASIÓN ES EL PRIMER AÑO DE UN PARLAMENTO CINCO AÑOS PARA MIRAR HACIA ADELANTE | TENEMOS QUE UTIZAR ESTA OCASIÓN ES EL PRIMER AÑO DE UN PAGAMENTO PARA MIRAR HACIA DELANTE | TENEMOS QUE OTIZAR ESTA OCASIÓN ES EL PRIMER AÑO DEL PARGAMENTO CINCO AÑOS PARA MIRAR HACIA ADELANTE | TENEMOS QUE UTILIZAR ESTA OCASIÓN ES EL PRIMER AÑO DE UN PARLAMENTO DE CINCO AÑOS PARA MIRAR DELANTE | |
| 2. | |||||
| (En) More than 100 000 flights have been cancelled and around 12 million passengers have not been able to travel as planned. (Es) se han cancelado más de cien mil vuelos y unos doce millones de pasajeros no han podido viajar como tenían previsto |
CIEN MIL VUELOS HAN SIDO CANCELADOS EN LOS DOCE MILLONES DE PASAJEROS QUE NO HAN PODIDO VIAJAR COMO PLANTA | E MÁS DE CIEN MIL VOLUNTADES HAN SIDO ALCANSADOS EN UNOS DOCE MILLONES DE PASAJEROS NO HAN PODIDO VIAJAR COMO PLANIFICADO | E MÁS DE CIEN MIL VUELOS SE HAN ENCONTRADO EN DOCE MILLONES DE PASAJEROS NO HAN PODIDO VIAJAR EN SU PLANIFICACIÓN | MÁS DE CIEN MIL VUELOS HAN SIDO ENCANZADOS EN DOCE MILLONES DE PASAJEROS NO HAN PODIDO VIAJAR ESTE PLAN | |
| 3. | |||||
| (En) But how can we work more proactively towards the ultimate goal of improving global aviation safety? (Es) pero cómo podemos trabajar de manera más proactiva hacia el objetivo final de mejorar la seguridad de la aviación internacional |
PERO COMO PODEMOS TRABAJAR MÁS PROACTIVAMENTE PARA MEJORAR LA SEGURIDAD DE LA AVIACIÓN MUNDIAL | COMO PODEMOS TRABAJAR MÁS PARA MEJORAR LA SEGURIDAD DE LA AVIACIÓN AVIACIÓN | PERO COMO PODEMOS TRABAJAR MÁS PRODUCTIVAMENTE PARA PODER CONSEGUIR EL OBJETIVO ÚLTIMO SÍ MEJORAMOS LA SEGURIDAD DE LA AVIACIÓN | PERO COMO PODEMOS TRABAJAR DE MANERA MÁS ACTIVA HACIA EL OBJETIVO ÚLTIMO DE MEJORAR LA SEGURIDAD DE LA AVIACIÓN | |
| Source (English) | Predictions | ||||
|---|---|---|---|---|---|
| S2T+T2U | S2UT (orig_unit) | S2UT (tgt_spk) | S2UT (sp_norm) | ||
| 1. | |||||
| (En) I trust your good judgment in solving this procedural problem, and I am looking forward to hearing the opinion of the representatives of the Council and the Commission. (Fr) je m'en remets à votre bon jugement pour résoudre ce problème procédural et je me réjouis d'entendre l'opinion des représentants du conseil et de la commission |
JE SUIS PERSUADÉ QUE VOUS ÊTES UN BON JUGEMENT POUR RÉSOUDRE CE PROBLÈME ET JATTENDS AVEC IMPATIENCE LA VIE DES REPRÉSENTANTS DU CONSEIL ET DE LA COMMISSION | JE PENSE QUE VOTRE BONNE JUGEMENT RÉSOUDRE CE PROBLÈME PROCÉDURE ET JE ME RÉJOUIS DENTENDRE LAVIS DES REPRÉSENTANTS DU CONSEIL ET DE LA COMMISSION | JE PENSE QUE VOTRE BON JUGEMENT POUR RÉSOUDRE CE PROBLÈME PROCÉDURE ET JATTENDS AVEC IMPATIENCE LAVIS DES REPRÉSENTANTS DU CONSEIL ET DE LA COMMISSION | JE CONFIANCE À VOTRE BON JUGEMENT POUR RÉSOUDRE CE PROBLÈME DE PROCÉDURE ET JE ME RÉJOUIS DENTENDRE LAVIS DES REPRÉSENTANTS DU CONSEIL ET DE LA COMMISSION | |
| 2. | |||||
| (En) It is high time to support a tougher approach tackling the misuse of EU funds. (Fr) il est grand temps de soutenir une approche plus stricte pour s'attaquer aux détournements des fonds communautaires |
IL EST GRAND TEMPS DE SOUTENIR UNE APPROCHE PLUS STRICTE POUR LUTTER CONTRE LE MAUVAIS USAGE DES FONDS EUROPÉENS | IL EST GRAND TEMPS DE SOUTENIR UNE APPROCHE PLUS SUPPLÉMENTAIRE AVEC LA LUTTE CONTRE LE MAL USAGE DES FONDS | IL EST GRAND TEMPS DE SOUTENIR UNE APPROCHE PLUS DIFFICILE POUR LUTTER CONTRE LE MAUVAIS USAGE DES FONDS | IL EST GRAND TEMPS DE SOUTENIR UNE APPROCHE PLUS DIFFICILE POUR LUTTER CONTRE LE MAUVAIS USAGE DES FONDS EUROPÉENS | |
| 3. | |||||
| (En) We want to see energy poverty as a part of this debate. (Fr) nous voulons que la pauvreté énergétique fasse partie intégrante de ce débat |
NOUS VOULONS QUE LA PAUVRETÉ ÉNERGÉTIQUE FASSE PARTIE DE CE DÉBAT | NOUS SOUHAITONS QUE LA POLITIQUE ÉNERGÉTIQUE FASSE PARTIE DE CE DRAM | NOUS SOUHAITONS QUE LA POLITIQUE ÉNERGÉTIQUE DANS LE CADRE DE CE DÉBATNUS | NOUS VOULONS QUE LA PAUVRETÉ ÉNERGÉTIQUE SOIT DANS LE CADRE DE CE DÉBAT | |