First, let me say that this is impressive. And then let me pose some questions:
As a linguist, I would like to know more about the kinds of languages this works well with, or does not work well with. For example, half the world's languages are tone languages, and the way tones work varies greatly among these. Some just have high and low tones, while others are considerably more complicated; Thai has high, mid, low, rising and falling. Also, tone is relative, e.g. a man's high tone might be a woman's low tone. And some African languages have tones whose absolute frequencies vary across an utterance. So transcribing tone is a quite different problem from transcribing phonemes--and yet for many tone languages, the tone is crucial.
There are also rare(r) phonemes, like the clicks in many languages of southern Africa. Of course maybe they've already trained on some of these languages.
The HuggingFace demo says "Supported Languages[:]
For this public demo, we've restricted transcription to low-resource languages with error rates below 10%." That's unclear: 10% word error rate, or character/ phoneme error rate? The meta.com page refers to character error rate (CER); a 10% character error rate can imply a much higher word error rate (WER), since most words contain several characters/ phonemes. That said, there are ways to get around that, like using a dictionary to select among different paths through possible character sequences so you only get known words, and adding to that a morphological parser for languages that have lots of affixes (meaning not all the word forms will be in the dictionary--think walk, walks, walked, walking--only the first will be in most dictionaries.)
I agree that this is a very exciting and really crucial research and I'm glad there is funding for this.
But it's very strange that Hungarian is marked as "highly endangered" at https://aidemos.atmeta.com/omnilingualasr/language-globe
Highly endangered is supposed to mean "The language is used by grandparents and older generations; while the parent generation may still understand the language, they typically do not speak it to children or among themselves."
Then why is Hungarian marked as such? Obviously not true with 14 million active speakers and being the 20th in terms of the most language resources published on the Internet.
Additionally, the feedback mechanism seems also broken ("There was an error submitting your feedback. Please try again.")
The Ethnologue link in footnote 7 of the paper has utm_source=chatgpt.com at the end, so I suspect whoever was tasked with listing languages and determining their status thought this wasn't important enough to do it themselves and just had ChatGPT give them a list. FWIW, Ethnologue does say that Ghari is "Stable" https://www.ethnologue.com/language/gri/ Meanwhile Swedish is "Institutional," the highest possible level of vitality https://www.ethnologue.com/language/swe/
This seems like a massive improvement for openly available local ASR. Even the 300M model outperforms whisper-large-v3 according to the paper's benchmarks.
This model is actually expected to be bad for popular languages, just like previous MMS it is not accurate at all, it wins by supporting something rare well but never had good ASR accuracy even for Swedish etc. It is more a research thing than a real tool. Unlike Whisper.
And moreover, you can not tune those models for practical applications. The model is originally trained on very clean data, so lower layers are also not very stable for diverse inputs. To finetune you have to update the whole model, not just upper layers.
In section 5.7.5, they fine-tune for "11 low-resource languages, with between 5-10 hours of training data and at least 1 hour of validation splits." "CTC
fine-tuning takes ≈1 hour of walltime on 32 GPUs for the 300M scale." If that's too expensive, you also have the option of supplying additional context for the LLM-based model (section 5.5).
As for "very clean data," see section 5.7.4: "Omnilingual + OMSF ASR was intentionally curated to represent naturalistic (i.e., often noisy) audio conditions, diverse speaker identities, and spontaneous, expressive speech."
> Omnilingual ASR was designed as a community-driven framework. People around the world can extend Omnilingual ASR to new languages by using just a few of their own samples.
That's a dumb joke. Yes, I hope to look in detail at their performance on a couple of low-resource languages. Without lots of speakers and data, I think good metrics are hard to come by. I've found that in Meta's massively-multilingual TTS - what looks impressive at first glance, you can see performance is quite bad on smaller languages.
Unfortunately I don't read anything in the paper about improvements to timing/timestamping. In particular unclean word boundaries are hard with wav2vev2.
And their use of LLMs as part of the transcription process makes it likely that they trained the model to correct mispronounciations by the speaker. This lowers CER because the human transcription often corrects for mispronounciations as well, but reduces the ability of the model to actually transcribe what was said.
What I really want to know is how well these could work for non-human languages. No, not aliens, but chimpanzees, dolphins, bonobos. We have hundreds or thousands of hours of recordings.
I think linguistics, don't deem animals to have languages as you require human level intelligence to use and understand some of the features in human languages like communicating about things that are away from your current timespace location. Animals have communication systems
I'm not asserting that bonobos, for example, have as complex a language as humans, just that it would be interesting to understand what language that they do have.
"You haven't experienced Shakespeare until you've read him in the original Bonobo". :-)
It would be indeed. But it would not be a language. What animals have is called "communication system". A language can be seen as a type of communication system. Languages are complex by definition and require certain cognitive capabilities: ability to create new sentences based on a set of rules, spatial-temporal displacement, etc.
How hard is it to make TTS out of this? A few independent journalists from Belarus asked for TTS in their language, but I am no expert, was thinking about re-using Mozilla's work. What's the easiest way to get working TTS for a language?
As far as I understand, the MMS TTS models are trained from scratch (section 7.1 of [1]), they do not employ any SSL models. So the OmniASR SSL models are not useful here.
What might be interesting is the newly released OmniASR data, because the MMS data, which was used for the MMS TTS, was never released.
Also, the OmniASR can be used to transcribe some untranscribed speech to train a TTS on it.
Meta cheated with the mms models. That is they didn’t use a phonemeizsr step. This means they just won’t work or sound very strange. ASR data is usually not quite right for tts. But anyhow - not really answering your question but many of these languages already done in mms. Try them https://huggingface.co/spaces/willwade/sherpa-onnx-tts
Yes, there seems to be lots of mistakes and no easy way to mark it. Highly endangered: Malayalam (=35 million speakers), Hungarian (14 million), Uighur (11 million), or Swedish as endangered... These are quite obvious mistakes even for a layperson.
can't say for sure, but a lot of the UI (and text) is quite familiar. the history page is a near rip off which is a giveaway.
i believe the mit license should be distributed since it's almost certainly a derivative work.
"The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software."
I can't confirm if the license is infact distributed, since I would have to pay $50, which quite frankly I'm not going to do.
a bit sad to see a ui reskin claimed as original work. the reskin is totally fine, but I believe the license must be distributed. i believe in the proliferation of this software so im happy to see this overall (it's good enough someone wants to charge for it! that's a big win!) but it's just a bit of a shame how this project has gone about it imo.
First, let me say that this is impressive. And then let me pose some questions:
As a linguist, I would like to know more about the kinds of languages this works well with, or does not work well with. For example, half the world's languages are tone languages, and the way tones work varies greatly among these. Some just have high and low tones, while others are considerably more complicated; Thai has high, mid, low, rising and falling. Also, tone is relative, e.g. a man's high tone might be a woman's low tone. And some African languages have tones whose absolute frequencies vary across an utterance. So transcribing tone is a quite different problem from transcribing phonemes--and yet for many tone languages, the tone is crucial.
There are also rare(r) phonemes, like the clicks in many languages of southern Africa. Of course maybe they've already trained on some of these languages.
The HuggingFace demo says "Supported Languages[:] For this public demo, we've restricted transcription to low-resource languages with error rates below 10%." That's unclear: 10% word error rate, or character/ phoneme error rate? The meta.com page refers to character error rate (CER); a 10% character error rate can imply a much higher word error rate (WER), since most words contain several characters/ phonemes. That said, there are ways to get around that, like using a dictionary to select among different paths through possible character sequences so you only get known words, and adding to that a morphological parser for languages that have lots of affixes (meaning not all the word forms will be in the dictionary--think walk, walks, walked, walking--only the first will be in most dictionaries.)
Enquiring minds want to know!
I'm not an expert but the rule of thumb is to expect something like this:
https://xkcd.com/1838/
[dead]
I agree that this is a very exciting and really crucial research and I'm glad there is funding for this. But it's very strange that Hungarian is marked as "highly endangered" at https://aidemos.atmeta.com/omnilingualasr/language-globe Highly endangered is supposed to mean "The language is used by grandparents and older generations; while the parent generation may still understand the language, they typically do not speak it to children or among themselves." Then why is Hungarian marked as such? Obviously not true with 14 million active speakers and being the 20th in terms of the most language resources published on the Internet. Additionally, the feedback mechanism seems also broken ("There was an error submitting your feedback. Please try again.")
Finnish: "safe" – sounds right
South Estonian: "vulnerable" – sure, yeah
Karelian: "endangered" – seems correct
Swedish: also "endangered" – wat
Ghari (12k speakers): "safe" – :facepalm:
Are these really language-vulnerability ratings or did they just make a mapping from Trump's tariff rates?
The Ethnologue link in footnote 7 of the paper has utm_source=chatgpt.com at the end, so I suspect whoever was tasked with listing languages and determining their status thought this wasn't important enough to do it themselves and just had ChatGPT give them a list. FWIW, Ethnologue does say that Ghari is "Stable" https://www.ethnologue.com/language/gri/ Meanwhile Swedish is "Institutional," the highest possible level of vitality https://www.ethnologue.com/language/swe/
My new favourite mistake is Malayalam being highly endangered...
This seems like a massive improvement for openly available local ASR. Even the 300M model outperforms whisper-large-v3 according to the paper's benchmarks.
Not sure, I recorded 3 seconds of voice (a single sentence) and the hf demo misrecognized about half of the words.
This model is actually expected to be bad for popular languages, just like previous MMS it is not accurate at all, it wins by supporting something rare well but never had good ASR accuracy even for Swedish etc. It is more a research thing than a real tool. Unlike Whisper.
And moreover, you can not tune those models for practical applications. The model is originally trained on very clean data, so lower layers are also not very stable for diverse inputs. To finetune you have to update the whole model, not just upper layers.
In section 5.7.5, they fine-tune for "11 low-resource languages, with between 5-10 hours of training data and at least 1 hour of validation splits." "CTC fine-tuning takes ≈1 hour of walltime on 32 GPUs for the 300M scale." If that's too expensive, you also have the option of supplying additional context for the LLM-based model (section 5.5).
As for "very clean data," see section 5.7.4: "Omnilingual + OMSF ASR was intentionally curated to represent naturalistic (i.e., often noisy) audio conditions, diverse speaker identities, and spontaneous, expressive speech."
Only a few gb of weights will recognize speech in 1600+ languages.
Freely downloadable and usable by anyone for almost anything.
We truly live in the future.
Seeing the absurd number of languages made me think of the norm macdonald joke:
Music is the universal language, but one day soon it will be replaced by Chinese.
Does anyone else feel like they buried the lead?
> Omnilingual ASR was designed as a community-driven framework. People around the world can extend Omnilingual ASR to new languages by using just a few of their own samples.
The world just got smaller
Just killed my startup. https://6k.ai
Half joking - hopefully, we can still contribute something to this to this field. Looking forward to doing some tests with this.
what is the "Penguin" language?
Also, 1.6k < 6k, and I highly doubt this model is anywhere near as good as it is on EU languages for most of them.
That's a dumb joke. Yes, I hope to look in detail at their performance on a couple of low-resource languages. Without lots of speakers and data, I think good metrics are hard to come by. I've found that in Meta's massively-multilingual TTS - what looks impressive at first glance, you can see performance is quite bad on smaller languages.
Unfortunately I don't read anything in the paper about improvements to timing/timestamping. In particular unclean word boundaries are hard with wav2vev2.
And their use of LLMs as part of the transcription process makes it likely that they trained the model to correct mispronounciations by the speaker. This lowers CER because the human transcription often corrects for mispronounciations as well, but reduces the ability of the model to actually transcribe what was said.
What I really want to know is how well these could work for non-human languages. No, not aliens, but chimpanzees, dolphins, bonobos. We have hundreds or thousands of hours of recordings.
What would it take to start working on them?
Not tested on that particular model, but the idea has been flying around for some time: https://arxiv.org/abs/2509.04166v1
You can check whale sound recognition project https://arxiv.org/abs/2104.08614
I think linguistics, don't deem animals to have languages as you require human level intelligence to use and understand some of the features in human languages like communicating about things that are away from your current timespace location. Animals have communication systems
I'm not asserting that bonobos, for example, have as complex a language as humans, just that it would be interesting to understand what language that they do have.
"You haven't experienced Shakespeare until you've read him in the original Bonobo". :-)
It would be indeed. But it would not be a language. What animals have is called "communication system". A language can be seen as a type of communication system. Languages are complex by definition and require certain cognitive capabilities: ability to create new sentences based on a set of rules, spatial-temporal displacement, etc.
There is a dolphin language model project from Google and Georgia Tech: https://blog.google/technology/ai/dolphingemma/
That's exactly the kind of thing I was hoping people were working on!
How hard is it to make TTS out of this? A few independent journalists from Belarus asked for TTS in their language, but I am no expert, was thinking about re-using Mozilla's work. What's the easiest way to get working TTS for a language?
EDIT: My bad, please disregard; As akreal pointed out, the MMS TTS models aren’t using the SSL models.
Original post:
You can use the OmniASR SSL models instead of their older MMS models to create TTS models: https://github.com/ylacombe/finetune-hf-vits
As far as I understand, the MMS TTS models are trained from scratch (section 7.1 of [1]), they do not employ any SSL models. So the OmniASR SSL models are not useful here.
What might be interesting is the newly released OmniASR data, because the MMS data, which was used for the MMS TTS, was never released.
Also, the OmniASR can be used to transcribe some untranscribed speech to train a TTS on it.
[1] MMS paper: https://arxiv.org/pdf/2305.13516
You’re completely right, I misremembered. I edited my post.
Meta cheated with the mms models. That is they didn’t use a phonemeizsr step. This means they just won’t work or sound very strange. ASR data is usually not quite right for tts. But anyhow - not really answering your question but many of these languages already done in mms. Try them https://huggingface.co/spaces/willwade/sherpa-onnx-tts
From TFA, it says that it’s extremely easy to add new languages with just a few examples. I didn’t see specifics on how “few” it really is, though.
This is ASR not TTS though.
Swedish
Status: Endangered
"The child-bearing generation can use the language among themselves, but it is seldom being transmitted to children."
What!? A lot must have changed in one generation..
Yes, there seems to be lots of mistakes and no easy way to mark it. Highly endangered: Malayalam (=35 million speakers), Hungarian (14 million), Uighur (11 million), or Swedish as endangered... These are quite obvious mistakes even for a layperson.
> Bring Your Own Language
Few-shot new languages is going to be a game changer for linguists
I'm going to test this with Voice AI to see how it works compared to Whisper and Parakeet
https://voice-ai.knowii.net
looks like a paid and closed source fork of the free and open source project Handy: https://github.com/cjpais/Handy
can't say for sure, but a lot of the UI (and text) is quite familiar. the history page is a near rip off which is a giveaway.
i believe the mit license should be distributed since it's almost certainly a derivative work.
"The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software."
I can't confirm if the license is infact distributed, since I would have to pay $50, which quite frankly I'm not going to do.
a bit sad to see a ui reskin claimed as original work. the reskin is totally fine, but I believe the license must be distributed. i believe in the proliferation of this software so im happy to see this overall (it's good enough someone wants to charge for it! that's a big win!) but it's just a bit of a shame how this project has gone about it imo.
I thought it looked familiar! Looks like they only changed some of the UI/colors lol.
any insights on latency?
HF Demo: https://huggingface.co/spaces/facebook/omniasr-transcription...
GitHub: https://github.com/facebookresearch/omnilingual-asr
Thanks! I've added those links to the toptext as well.
the global language explorer is fascinating -great work guys
https://aidemos.atmeta.com/omnilingualasr/language-globe
- we are getting closer to BabelFish.. at least for the Earth!