Welcome to xFormers’s documentation!

xFormers is a PyTorch based library which hosts flexible Transformers parts. They are interoperable and optimized building blocks, which can optionally be combined to create some state of the art models.

Components Documentation

• API Reference
  • xFormers optimized operators
  • Attention mechanisms
  • Feedforward mechanisms
  • Position Embeddings
  • Reversible layer
  • Multi Head Attention

Tutorials and examples

• Tutorials
  • Replace all attentions from an existing ViT model with a sparse equivalent?
  • Using BlockSparseAttention
  • Extend the xFormers parts zoo
  • I’m only interested in testing out the attention mechanisms that are hosted here
  • Using the Reversible block
  • Using Triton-based layers

Some custom parts

• Custom parts reference
  • Sparse CUDA kernels
  • Triton parts