Multi-Lingual Datasets
We catalog 82 Multi-Lingual datasets for NLP and machine learning. Browse the list below or narrow down by task.
This page covers Multi-Lingual-language data. Our directory includes 82 datasets in Multi-Lingual.
- A Novel Approach to a Semantically-Aware Representation of Items (NASARI)Semantic Textual SimilarityMulti-Lingual
- AudioSetSpeech Recognition, VisualMulti-Lingual
- BianetMachine TranslationMulti-Lingual
- Bible CorpusMachine TranslationMulti-Lingual
- Books CorpusMachine TranslationMulti-Lingual
- BSNLP-2019Named Entity Recognition (NER), Entity LinkingMulti-Lingual
- CC NetText CorporaMulti-Lingual
- CCMatrixMachine TranslationMulti-Lingual
- CLIRMatrixInformation RetrievalMulti-Lingual
- Code-Mixed-DialogDialogueMulti-Lingual
- Common VoiceSpeech RecognitionMulti-Lingual
- CommonCrawlText CorporaMulti-Lingual
- ConceptNetCommonsense, Knowledge GraphMulti-Lingual
- COVID-19 Twitter Chatter DatasetText CorporaMulti-Lingual
- CoVoSTMachine Translation, Speech-To-TextMulti-Lingual
- Cross-lingual Choice of Plausible Alternatives (XCOPA)Commonsense ReasoningMulti-Lingual
- DbpediaKnowledge BaseMulti-Lingual
- Densely Annotated Wikipedia Texts (DAWT)Named Entity Recognition (NER)Multi-Lingual
- DSL Corpus Collection (DSLCC)Discriminating between similar languagesMulti-Lingual
- ECB CorpusText Corpora, Machine TranslationMulti-Lingual
- EMEAMachine TranslationMulti-Lingual
- EubookshopText Corpora, Machine TranslationMulti-Lingual
- Europarl-STSpeech TranslationMulti-Lingual
- European Parliament Proceedings (Europarl)Text Corpora, Machine TranslationMulti-Lingual
- Europeana NewspapersNamed Entity Recognition (NER)Multi-Lingual
- EXAMSQuestion AnsweringMulti-Lingual
- Global Voices Parallel CorpusMachine TranslationMulti-Lingual
- Google Books N-gramsClassification, ClusteringMulti-Lingual
- Gutenberg DialogueDialogueMulti-Lingual
- Guttenberg Book CorpusText CorporaMulti-Lingual
- IWSLT 15 English-VietnameseMachine TranslationMulti-Lingual
- IWSLT'15 English-Vietnamese Machine TranslationMulti-Lingual
- JW300Machine TranslationMulti-Lingual
- Leipzig Corpora CollectionText CorporaMulti-Lingual
- Mewsli-9 Entity LinkingMulti-Lingual
- Microsoft Speech Language Translation Corpus (MSLT)Speech Recognition, Machine TranslationMulti-Lingual
- MLSUMSummarizationMulti-Lingual
- MTOPSemantic ParsingMulti-Lingual
- MultiLing Pilot 2011 DatasetSummarizationMulti-Lingual
- Multilingual Corpus of Sentence-Aligned Spoken Utterances (MaSS)Speech CorporaMulti-Lingual
- Multilingual Knowledge Questions & Answers (MKQA)Question AnsweringMulti-Lingual
- MultiLingual Question Answering (MLQA)Question Answering, Reading ComprehensionMulti-Lingual
- MuST-CSpeech TranslationMulti-Lingual
- News Commentary Parallel CorpusMachine TranslationMulti-Lingual
- One Week of Global News FeedsText CorporaMulti-Lingual
- OneSeC SmallWord Sense Disambiguation Multi-Lingual
- OntoNotes 5.0Information Retrieval, Syntactic ParsingMulti-Lingual
- Open Super-Large Crawled Almanach Corpus (OSCAR)Text CorporaMulti-Lingual
- OpenSubtitlesDialogueMulti-Lingual
- ParaCrawl CorpusMachine TranslationMulti-Lingual
- Parallel Meaning BankText CorporaMulti-Lingual
- Paraphrase Adversaries from Word Scrambling (PAWS-X)Paraphrasing IdentificationMulti-Lingual
- RELX & RELX-DistantRelation ClassificationMulti-Lingual
- TanzilMachine TranslationMulti-Lingual
- TatoebaMachine TranslationMulti-Lingual
- The Cross-lingual Natural Language Inference corpus (XNLI)EntailmentMulti-Lingual
- The EUR-Lex DatasetClassificationMulti-Lingual
- The NewsReader MEANTIME CorpusNamed Entity Recognition (NER)Multi-Lingual
- The Winograd Schema ChallengeCoreference ResolutionMulti-Lingual
- Train-O-Matic LargeWord Sense Disambiguation Multi-Lingual
- Train-O-Matic SmallWord Sense Disambiguation Multi-Lingual
- TyDi QAQuestion Answering, Reading ComprehensionMulti-Lingual
- United Nations Parallel CorpusMachine TranslationMulti-Lingual
- VoxCelebSpeech Recognition, VisualMulti-Lingual
- VoxClamantisPhonetic TypologyMulti-Lingual
- VoxForgeSpeech CorporaMulti-Lingual
- Web Inventory of Transcribed and Translated Talks (WIT3)Machine TranslationMulti-Lingual
- Webis-CLS-10Classification, Sentiment AnalysisMulti-Lingual
- WikiAnnNamed Entity Recognition (NER)Multi-Lingual
- WikiMatrixMachine TranslationMulti-Lingual
- WikiNERNamed Entity Recognition (NER)Multi-Lingual
- WikiReadingKnowledge Base, NLUMulti-Lingual
- WMT 14 English-GermanMachine TranslationMulti-Lingual
- WMT 15 English-CzechMachine TranslationMulti-Lingual
- WMT 19 Multiple DatasetsText Corpora, Machine TranslationMulti-Lingual
- Worldwide News - Aggregate of 20K FeedsClustering, Events, Machine TranslationMulti-Lingual
- X-StanceStance DetectionMulti-Lingual
- XEDSentiment AnalysisMulti-Lingual
- XL-WiCWord Sense DisambiguationMulti-Lingual
- XOR-TyDi QAQuestion AnsweringMulti-Lingual
- XQuADQuestion Answering, Reading ComprehensionMulti-Lingual
- XQuaD-RQuestion AnsweringMulti-Lingual
What tasks do Multi-Lingual datasets cover?
Machine Translation (25)Text Corpora (12)Question Answering (7)Named Entity Recognition (NER) (6)Speech Recognition (4)Word Sense Disambiguation (4)Dialogue (3)Classification (3)Reading Comprehension (3)Visual (2)Entity Linking (2)Information Retrieval (2)Knowledge Base (2)Speech Translation (2)Clustering (2)Summarization (2)Speech Corpora (2)Sentiment Analysis (2)