Gang

class fairseq2.gang.Gang

Bases: ABC

Represents a set of processes that work collectively.

abstract all_gather(output_tensor, input_tensor)

Gather tensors from all processes and put them in output_tensor.

Parameters:

• output_tensor (Tensor) – The output tensor to accomodate tensors from all processes.

• input_tensor (Tensor) – The tensor to be gathered from this process.

abstract all_gather_to_list(output_tensors, input_tensor)

Gather tensors from all processes and put them in output_tensors.

Parameters:

• output_tensors (List[Tensor]) – The tensor list to accomodate tensors from all processes.

• input_tensor (Tensor) – The tensor to be gathered from this process.

abstract all_reduce(tensor, op)

Reduce tensor across all processes.

Parameters:

• tensor (Tensor) – The input and output tensor of the operation.

• op (ReduceOperation) – The element-wise reduce operation.

abstract as_process_group()

Return this gang as a process group.

Return type:

ProcessGroup

abstract barrier()

Synchronize all processes.

abstract broadcast(tensor, source_rank=0)

Broadcast tensor from source_rank to all processes.

Parameters:

• tensor (Tensor) – The tensor to be sent from source_rank.

• source_rank (int) – The rank of the process from which to broadcast tensor.

abstract broadcast_objects(objects, source_rank=0)

Broadcast picklable objects from source_rank to all processes.

Parameters:

• objects (List[Any]) – The list of picklable objects to broadcast. Each process must provide lists of equal sizes.

• source_rank (int) – The rank of the process from which to broadcast objects.

abstract close()

Close and destroy the gang.

abstract create_gang(ranks)

Create a new gang.

Parameters:

ranks (Sequence[int]) – The ranks of processes that will be part of the new gang.

Return type:

Gang | None

abstract property device: device

The associated device.

abstract property rank: int

The rank of this process in the gang.

abstract property size: int

The number of processes that are part of the gang.