Language Modeling Datasets
There are 6 language modeling datasets in our directory. Each links to its source, paper, and download — browse the full list below or filter by language.
Language Modeling is the task of predicting the next token in a sequence — the core pre-training objective behind every LLM. We catalog 6 datasets for it.
- 1 Billion Word Language Model Benchmark (lm1b)Language ModelingEnglish
- Language Modeling Broadened to Account for Discourse Aspects (LAMBADA)Natural Language Understanding, Language ModelingEnglish
- PG-19Text Corpora, Language ModelingEnglish
- The Benchmark of Linguistic Minimal Pairs (BLiMP)Language ModelingEnglish
- WikiText-103 & 2Language ModelingEnglish
- WikiText-TL-39Text Corpora, Language ModelingFilipino