MLCommons and Hugging Face Release Huge Public Speech Dataset for AI Research

Two kinds of public infrastructure

MLCommons approaches the problem from measurement: it is a nonprofit AI safety working group that convenes industry and academic researchers around shared evaluation. Hugging Face approaches it from distribution — its Hub is the standard place where the machine learning community shares models, and the company maintains tooling that much open audio and language work is built on.

Not Hugging Face's first large corpus

The company has done this before at scale. In April 2021 it launched the BigScience research workshop with a group of partner labs to build an open large language model; the effort concluded in 2022 with BLOOM, a multilingual model whose release established a template for community-built, openly licensed training data and weights.

Why speech data is the bottleneck

Text and image corpora have been assembled at enormous scale over the past decade. Speech is harder: recordings need consent, transcription and language coverage, and a great deal of the best material sits behind commercial licences. A large public speech collection matters because audio models — recognition, diarisation, translation — depend on exactly the kind of data that has been hardest to release openly, and because research groups without licensing budgets are usually shut out of it.