Token Classification Datasets
There are 36 token classification datasets in our directory, 1 of which are benchmarks. Each links to its source, paper, and download — browse the full list below or filter by language.
Token Classification is the task of labelling individual tokens in a sequence, used for tasks like part-of-speech tagging. We catalog 36 datasets for it.
- ade-benchmark-corpus/ade_corpus_v2Text Classification, Token ClassificationEN
- ai4privacy/pii-masking-200kText Classification, Token Classification, Table Question Answering, Question Answering, Zero Shot Classification, Summarization, Feature Extraction, Text Generation, Translation, Fill Mask, Tabular Classification, Tabular To Text, Table To Text, Text Retrieval, OtherEN, FR, DE
- ai4privacy/pii-masking-300kText Classification, Token Classification, Table Question Answering, Question Answering, Zero Shot Classification, Summarization, Feature Extraction, Text Generation, Translation, Fill Mask, Tabular Classification, Tabular To Text, Table To Text, Text Retrieval, OtherEN, FR, DE
- ai4privacy/pii-masking-400kText Classification, Token Classification, Table Question Answering, Question Answering, Zero Shot Classification, Summarization, Feature Extraction, Text Generation, Translation, Fill Mask, Tabular Classification, Tabular To Text, Table To Text, Text Retrieval, OtherEN, FR, DE
- amirveyseh/acronym_identificationToken ClassificationEN
- aps/super_glueText Classification, Token Classification, Question AnsweringEN
- b3x0m/Chinese-H-NovelsText Classification, Summarization, Token Classification, Question Answering, Text Generation, Fill Mask, Sentence SimilarityZH
- Babelscape/wikineuralToken ClassificationDE, EN, ES
- bigcode/bigcode-pii-datasetToken ClassificationCODE
- eriktks/conll2003Token ClassificationEN
- flytech/python-codes-25kText Classification, Question Answering, Token Classification, Summarization, Text GenerationEnglish
- google/frames-benchmarkText Classification, Token Classification, Table Question Answering, Question AnsweringEN
- google/xtremeMultiple Choice, Question Answering, Token Classification, Text Classification, Text RetrievalAF, AR, BG
- gretelai/synthetic_pii_finance_multilingualText Classification, Fill Mask, Token ClassificationEN, FR, DE
- HUPD/hupdFill Mask, Summarization, Text Classification, Token ClassificationEN
- indonlp/indonluQuestion Answering, Text Classification, Token ClassificationID
- klue/klueFill Mask, Question Answering, Text Classification, Text Generation, Token ClassificationKO
- kyujinpy/OpenOrca-KOText Classification, Token Classification, Table Question Answering, Question Answering, Zero Shot Classification, Summarization, Feature Extraction, Text GenerationKO
- maveriq/bigbenchhardQuestion Answering, Token Classification, Text ClassificationENBenchmark
- ncbi/ncbi_diseaseToken ClassificationEN
- NTU-NLP-sg/xCodeEvalTranslation, Token Classification, Text Retrieval, Text Generation, Text Classification, Feature Extraction, Question AnsweringCODE, EN
- nvidia/Nemotron-PIIToken ClassificationEN
- ontonotes/conll2012_ontonotesv5Token ClassificationAR, EN, ZH
- Open-Orca/OpenOrcaText Classification, Token Classification, Table Question Answering, Question Answering, Zero Shot Classification, Summarization, Feature Extraction, Text GenerationEN
- Open-Orca/SlimOrcaText Classification, Token Classification, Table Question Answering, Question Answering, Zero Shot Classification, Summarization, Feature Extraction, Text GenerationEN
- peluz/lener_brToken ClassificationPT
- pfb30/multi_woz_v22Text Generation, Fill Mask, Token Classification, Text ClassificationEN
- proj-persona/PersonaHubText Generation, Text Classification, Token Classification, Fill Mask, Table Question AnsweringEN, ZH
- retkowski/ytsegToken Classification, Automatic Speech RecognitionEN
- rmyeid/polyglot_nerToken ClassificationAR, BG, CA
- squarelike/OpenOrca-gugugo-koText Classification, Token Classification, Table Question Answering, Question Answering, Zero Shot Classification, Summarization, Feature Extraction, Text GenerationKO
- Ujjwal-Tyagi/ai-ml-foundations-book-collectionText Generation, Text Classification, Question Answering, Summarization, Sentence Similarity, Feature Extraction, Zero Shot Classification, Text Retrieval, Token Classification, Multiple Choice, Fill MaskEN
- unimelb-nlp/wikiannToken ClassificationACE, AF, ALS
- universal-dependencies/universal_dependenciesToken ClassificationABQ, AB, AF
- WINGNUS/ACL-OCLToken ClassificationEN
- yys/OpenOrca-ChineseText Classification, Token Classification, Table Question Answering, Question Answering, Zero Shot Classification, Summarization, Feature Extraction, Text GenerationZH