← Back to API reference

neuraltrain.models.transformer.TransformerEncoder

pydantic model neuraltrain.models.transformer.TransformerEncoder

Transformer encoder/decoder built on top of x_transformers.

Parameters:

• heads (int) – Number of attention heads.

• depth (int) – Number of Transformer layers.

• cross_attend (bool) – Enable cross-attention (decoder mode).

• causal (bool) – If True, build a causal Decoder instead of an Encoder.

• attn_flash (bool) – Use Flash Attention. Not compatible with ALiBi.

• attn_dropout (float) – Dropout probability inside the attention layers.

• ff_mult (int) – Feed-forward expansion factor (ff_dim = dim * ff_mult).

• ff_dropout (float) – Dropout probability in the feed-forward layers.

• use_scalenorm (bool) – Use ScaleNorm instead of LayerNorm.

• use_rmsnorm (bool) – Use RMSNorm instead of LayerNorm.

• rel_pos_bias (bool) – Use relative positional bias.

• alibi_pos_bias (bool) – Use ALiBi positional bias.

• rotary_pos_emb (bool) – Use rotary positional embeddings.

• rotary_xpos (bool) – Use xPos extension for rotary embeddings.

• residual_attn (bool) – Add residual connections around the attention output.

• scale_residual (bool) – Scale residual connections.

• layer_dropout (float) – Probability of dropping an entire Transformer layer during training.

Fields:

• alibi_pos_bias (bool)

• attn_dropout (float)

• attn_flash (bool)

• causal (bool)

• cross_attend (bool)

• depth (int)

• ff_dropout (float)

• ff_mult (int)

• heads (int)

• layer_dropout (float)

• rel_pos_bias (bool)

• residual_attn (bool)

• rotary_pos_emb (bool)

• rotary_xpos (bool)

• scale_residual (bool)

• use_rmsnorm (bool)

• use_scalenorm (bool)

field heads: int = 8

field depth: int = 12

field cross_attend: bool = False

field causal: bool = False

field attn_flash: bool = False

field attn_dropout: float = 0.1

field ff_mult: int = 4

field ff_dropout: float = 0.0

field use_scalenorm: bool = True

field use_rmsnorm: bool = False

field rel_pos_bias: bool = False

field alibi_pos_bias: bool = False

field rotary_pos_emb: bool = True

field rotary_xpos: bool = False

field residual_attn: bool = False

field scale_residual: bool = True

field layer_dropout: float = 0.0

build(dim: int) → Module

← Back to API reference