Convert Checkpoints

The checkpoint conversion handlers provides utilities to convert fairseq2 model checkpoints to different formats for interoperability with other frameworks.

Command Line Interface

fairseq2 llama convert_checkpoint --model <architecture> <fairseq2_checkpoint_dir> <output_dir>

Arguments

  • --model <architecture>: The model architecture name (e.g., llama3_2_1b) to generate correct params.json

  • <fairseq2_checkpoint_dir>: Directory containing the fairseq2 checkpoint (model.pt or model.{0,1,2…}.pt for sharded checkpoints)

  • <output_dir>: Output directory to store the converted checkpoint

Supported Architectures

The converter supports various LLaMA architectures including:

  • LLaMA 1: 7B, 13B, 33B, 65B

  • LLaMA 2: 7B, 13B, 70B

  • LLaMA 3: 8B, 70B

  • LLaMA 3.1: 8B, 70B

  • LLaMA 3.2: 1B, 3B

For the complete list of architectures and their configurations, see fairseq2.models.llama.archs.

Output Format

The converter produces:

  1. Model weights in the reference format: - Single checkpoint: consolidated.00.pth - Sharded checkpoints: consolidated.{00,01,02...}.pth

  2. params.json containing model configuration:

{
    "model": {
        "dim": 2048,                // Model dimension
        "n_layers": 16,             // Number of layers
        "n_heads": 32,              // Number of attention heads
        "n_kv_heads": 8,            // Number of key/value heads (if different from n_heads)
        "multiple_of": 256,         // FFN dimension multiple
        "ffn_dim_multiplier": 1.5,  // FFN dimension multiplier (if not 1.0)
        "rope_theta": 500000.0,     // RoPE theta value
        "norm_eps": 1e-5            // Layer norm epsilon
    }
}

Usage Example

  1. Convert a fairseq2 checkpoint to reference format:

fairseq2 llama convert_checkpoint --model llama3_2_1b \
    /path/to/fairseq2/checkpoint \
    /path/to/output/dir

  1. Convert to HuggingFace format:

fairseq2 llama write_hf_config --model <architecture> <fairseq2_checkpoint_dir>

  • <architecture>: Specify the architecture of the model – e.g., llama3 (see fairseq2.models.llama)

  • <fairseq2_checkpoint_dir>: Path to the directory containing your Fairseq2 checkpoint, where config.json will be added.

Note

Architecture --model must exist and be defined in e.g. fairseq2.models.llama._config.register_llama_configs().

API Details

final class fairseq2.cli.commands.llama.ConvertLLaMACheckpointHandler[source]

Bases: CliCommandHandler

init_parser(parser)[source]

Initialize parser with command-specific arguments.

run(context, parser, args)[source]

Run the command.

Return type:

int

final class fairseq2.cli.commands.llama.WriteHFLLaMAConfigHandler[source]

Bases: CliCommandHandler

init_parser(parser)[source]

Initialize parser with command-specific arguments.

run(context, parser, args)[source]

Run the command.

Return type:

int

See Also