Browse Datasets

CC-100

Common Crawl Corpus for 100+ Languages

tokenization pretraining embeddings Massive (varies by language) Common Crawl Terms
Access Dataset

Description

Constructed from Common Crawl snapshots, used to train XLM-R. Excellent for training monolingual tokenizers and embeddings due to its volume.

Quick Start

Python 
from tokenizers import ByteLevelBPETokenizer

tokenizer = ByteLevelBPETokenizer()
tokenizer.train(["amharic.txt"], vocab_size=52000)

Project Ideas

WordPiece for Amharic

Train a WordPiece tokenizer for Amharic and analyze Ge'ez script handling

Intermediate

Tokenizer Comparison

Compare BPE vs Unigram tokenization on Zulu (agglutinative language)

Advanced

Ethical Considerations

  • Web-crawled data contains noise and potential bias