fairseq2 Reference
DataPipeline
DataPipelineBuilder
list_files()
read_sequence()
read_zipped_records()
read_text()
FileMapper
Collater
CollateOptionsOverride
Misc
Bases: TextTokenDecoder
TextTokenDecoder
token_indices (Tensor) – The token indices to decode from.
str | CString
tokens (Sequence[str | CString]) – The tokens to decode from.