fairseq2.data.text¶
This module contains text tokenizers and text specific data pipeline operators.
classDiagram Enum <|-- LineEnding
This module contains text tokenizers and text specific data pipeline operators.
classDiagram Enum <|-- LineEnding