Hide navigation sidebar

Hide table of contents sidebar

Skip to content

Toggle site navigation sidebar

fairseq2 Documentation

Toggle table of contents sidebar

fairseq2 Documentation

Getting Started

Installation
Toggle navigation of Installation
- Installing from Source
- UV Setup
Quick Start

Basics

Overview
Design Philosophy
CLI
Assets
Data Pipeline
Checkpoint Management
Recipes
Runtime Extensions
Gang
Trainer

Tutorials

End-to-End Fine-Tuning
Preference Optimization
Monitor Your Experiments
Working with Presets
Debugging with PuDB
Add Your Own Model

Notebooks

✎ Data Pipeline
✎ Datasets

FAQ

Contributing to fairseq2

Reference

API Reference
Toggle navigation of API Reference
- fairseq2
- fairseq2.gang
- fairseq2.cli
  Toggle navigation of fairseq2.cli
  - Convert Checkpoints
- fairseq2.data
  Toggle navigation of fairseq2.data
  - fairseq2.data.text
    Toggle navigation of fairseq2.data.text
    - Text Tokenizers
- fairseq2.datasets
- Dataset Loaders
- fairseq2.nn
  Toggle navigation of fairseq2.nn
  - Position Encoders
- fairseq2.recipes
  Toggle navigation of fairseq2.recipes
- fairseq2.assets
Bibliography

Toggle table of contents sidebar

Text Tokenizers¶

fairseq2.datasets

fairseq2.data.text

Made with Sphinx and @pradyunsg's Furo