Meta Debuts SeamlessM4T, the Swiss Army Knife of Translation Models

Meta首次亮相翻译模型的瑞士军刀：SeamlessM4T.

It recognizes speech (that is, automatically — as in automatic speech recognition). It translates speech into speech (or text), and text into text (or speech) — in 100+ languages. Meta’s new Massively Multilingual & Multimodal Machine Translation (SeamlessM4T) is the Swiss army knife of language models. Proud parent Meta introduced the new model in a blog post published on August 22, 2023.

SeamlessM4T可识别语音（即自动-如自动语音识别）。它将语音转换为语音（或文本），并将文本转换为文本（或语音），在100多种语言中。Meta新兴大规模多语言和多模态机器翻译（SeamlessM 4 T）是语言模型的瑞士军刀。以此为荣的母公司Meta在2023年8月22日发布的博客中介绍了这款新模型。

The SeamlessM4T launch follows a number of language technology announcements by Meta over the past 12 months. These include low resource massively multilingual MT in mid 2022, massively multilingual speech translation in May 2023, and multilingual speech model Voicebox in June 2023. The social media giant is spending considerable resources on tackling the language problem of its metaverse vision.

SeamlessM4T的发布是在Meta过去12个月发布了一系列语言技术之后。其中包括2022年年中的低资源大规模多语言MT，2023年5月的大规模多语言语音翻译，以及2023年6月的多语言语音模型Voicebox。这家社交媒体巨头正在投入大量资源来解决其虚拟世界愿景的语言问题。

On X, one observer described SeamlessM4T as “revolutionary” and called it a “game-changer.” Another gushed, “It’s not just a tool; it’s a step towards a world where everyone can be understood, regardless of language.”

在X上，一位观察者将SeamlessM4T描述为“一场革命”，并称其为“游戏规则改变者”。另一位则滔滔不绝地说：“它不仅仅是一个工具;这是迈向一个每个人都能被理解的世界的一步，无论语言如何。”

“The code switching support of SeamlessM4T is pretty cool!” shared a fan with a sense of humor. “It doesn’t do very well with my French or Japanese, but then again neither is very good.”

“SeamlessM4T的代码切换支持非常酷！“一个有幽默感的粉丝分享。“它和我的法语或日语搭配得不太好，但话说回来，这两种语言都不太好。”

One Dr. Hubertus Becker questioned the model’s reliability for critical translations, noting, “It’s concerning that an experimental demo can alter the meaning of input words.”

Hubertus Becker博士质疑该模型在关键翻译中的可靠性，并指出：“这是一个实验演示可以改变输入单词的含义。

Kalev Leetaru, reporting on SeamlessM4T’s performance in translating Weibo social media posts, cited inconsistent results.

Kalev Leetaru在报告SeamlessM4T翻译微博社交媒体帖子的表现时，引用了不一致的结果。

“For some posts it yields translations that compare favorably to both NMT and LLM translations, but with the added cost of having to use language-specific punctuation rules to split into sentences to translate a sentence at a time,” Leetaru explained. “For other posts, it yields subpar translations that can remove or truncate key details, suggesting promise but that it is not quite ready for production use.”

“对于一些帖子，它产生的翻译与NMT和LLM翻译相比都是有利的，但必须使用特定语言的标点规则来拆分成句子以一次翻译一个句子，”Leetaru解释说。对于其他帖子，它会生成可能会删除或截断关键细节的不合格翻译，这表明它很有前途，但还没有完全准备好用于生产。

Of course, the more than 60 authors behind the August 22, 2023 paper introducing SeamlessM4T, believe in what they dubbed “the first multilingual system” to translate from and into English for both speech and text.

当然，2023年8月22日介绍SeamlessM4T的论文背后的60多位作者相信他们所谓的“第一个多语言系统”，可以将语音和文本翻译成英语。

If the stats behind SeamlessM4T’s training seem somewhat disparate, that might be because the model required training in so many (formerly) separate and siloed tasks. Similarly, the number of languages handled by the model varies by task.

如果SeamlessM4T的训练背后的统计数据看起来有些不同，那可能是因为该模型需要在许多（以前）单独和孤立的任务中进行训练。同样，模型处理的语言数量因任务而异。

SeamlessM4T can provide automatic speech recognition (ASR) for almost 100 languages; speech-to-text (STT) translation for nearly 100 input and output languages; speech-to-speech translation and text-to-speech translation for nearly 100 input languages and 36 output languages (including English); and traditional “text” translation for close to 100 languages.

SeamlessM 4 T可提供近100种语言的自动语音识别（ASR）;语音到文本（STT）翻译近100种输入和输出语言;近100种输入语言和36种输出语言（包括英语）的语音转换和文本转换;和传统的“文本”翻译近100种语言。

According to the authors, Meta’s motivation for the new model was to work around the existing separate systems that can complete the above tasks — but generally perform well in only one modality per system.

根据作者的说法，Meta对新模型的动机是围绕现有的独立系统进行工作，这些系统可以完成上述任务，但通常每个系统只能在一种模式下表现良好。

SeamlessM4T, by contrast, reportedly achieves state-of-the-art results for all these languages while offering “multitask support” in a single model. The paper also asserts that SeamlessM4T outperforms its previous SOTA competitors, namely Whisper and AudioPaLM-2.

相比之下，据报道，SeamlessM 4 T在所有这些语言上都实现了最先进的结果，同时在单个模型中提供了“多任务支持”。该论文还断言，SeamlessM 4 T的性能优于其之前的SOTA竞争对手，即Whisper和AudioPaLM-2。

Meta has publicly released the contributions to its new model, and encourages researchers and developers to build on this first iteration.

Meta已经公开发布了对新模型的贡献，并鼓励研究人员和开发人员在第一次迭代的基础上进行开发。