Browse Datasets

OSCAR

Open Super-large Crawled Aggregated Corpus

language-modeling pretraining Millions of sentences CC0 1.0 (Metadata)
Access Dataset Read Paper

Description

A huge multilingual corpus obtained by language classification and filtering of Common Crawl. Provides data in both original and deduplicated forms, specifically aiming to improve data quality for low-resource languages.

Quick Start

Python 
from datasets import load_dataset

# Load deduplicated Amharic data
dataset = load_dataset("oscar-corpus/oscar", language="am", split="train", streaming=True)
print(next(iter(dataset)))

Project Ideas

Token Distribution Analysis

Analyze token distribution of Amharic script compared to Latin-script African languages

Intermediate

3-gram Language Model

Build a 3-gram language model for Malagasy

Beginner

Ethical Considerations

  • As a web crawl, it reflects biases of the internet
  • Privacy of individuals mentioned in crawled text is a concern