Build Your Own Small Language Model
Learn the fundamentals of language modeling by building a small language model from scratch. This introductory course demystifies the creation of language models, emphasizing that you don't need massive compute resources to understand the fundamental principles of Generative AI.
Key Topics
- Fundamentals of language modeling
- Evolution from n-gram models to Transformers
- The machine learning development pipeline
- Small Language Models (SLMs) as pedagogical tools
Learning Outcomes
- Gain a high-level understanding of how text generation works
- Understand the strengths and limitations of different modeling architectures
- Identify the trade-offs between model size and performance
- Build vocabularies and train tokenizers
- Conceptualize AI problems within a local community context
Practical Projects
- Write code to generate text and identify patterns using simple models
- Analyze real-world research engineering workflows
- Train a unidirectional LSTM or Transformer from scratch on African language text
Recommended Datasets
5 datasets are aligned with this course for hands-on exercises and projects.
WURA
Web-crawled Unsupervised Representation for Africa
mC4 (African Subset)
Multilingual Colossal Cleaned Crawled Corpus
MasakhaNEWS
News Topic Classification for 16 African Languages
NaijaVoices
Culturally Rich Nigerian Language Dataset
OSCAR
Open Super-large Crawled Aggregated Corpus
About This Course
In this foundational course, you’ll learn how language models work by building one from scratch. You’ll start with raw text data and progress through tokenization, vocabulary building, and model training.
Key Topics
- Tokenization: Breaking text into meaningful units (characters, subwords, words)
- Vocabulary Construction: Building efficient vocabularies for African languages
- Model Architecture: Understanding basic neural language model architectures
- Training: Implementing training loops and loss functions
- Evaluation: Using perplexity and other metrics to measure model quality
Prerequisites
- Basic Python programming
- Familiarity with NumPy
- Understanding of basic probability concepts