Residual Connections

class fairseq2.nn.ResidualConnect(*args, **kwargs)[source]

Bases: Module, ABC

Represents a residual connection.

Initialize internal Module state, shared by both nn.Module and ScriptModule.

abstract forward(seqs: Tensor, residual: Tensor) → Tensor[source]
Parameters:

  • seqs – The sequences output by a module. Shape: \((N,S,M)\), where \(N\) is the batch size, \(S\) is the sequence length, and \(M\) is the dimensionality of the model.

  • residual – The input sequences to the module. Shape: Same as seqs.

Returns:

The output sequences with residuals applied. Shape: Same as seqs.

final class fairseq2.nn.AdditiveResidualConnect(*args, **kwargs)[source]

Bases: ResidualConnect

Sums inputs and outputs of a module.

Initialize internal Module state, shared by both nn.Module and ScriptModule.

forward(seqs: Tensor, residual: Tensor) → Tensor[source]
Parameters:

  • seqs – The sequences output by a module. Shape: \((N,S,M)\), where \(N\) is the batch size, \(S\) is the sequence length, and \(M\) is the dimensionality of the model.

  • residual – The input sequences to the module. Shape: Same as seqs.

Returns:

The output sequences with residuals applied. Shape: Same as seqs.

final class fairseq2.nn.ScaledResidualConnect(scale: float)[source]

Bases: ResidualConnect

Scales residuals by a constant factor before adding them to the output of a Transformer module.

Parameters:

scale – The scale factor.

forward(seqs: Tensor, residual: Tensor) → Tensor[source]
Parameters:

  • seqs – The sequences output by a module. Shape: \((N,S,M)\), where \(N\) is the batch size, \(S\) is the sequence length, and \(M\) is the dimensionality of the model.

  • residual – The input sequences to the module. Shape: Same as seqs.

Returns:

The output sequences with residuals applied. Shape: Same as seqs.