IO Module

The spdl.io module is a stand-alone module that implements efficient loading of data into array formats.

It is mainly designed for AI training/inference running in the cloud. It supports converting various data formats in byte strings (downloaded from remote storage) into array format, and transferring it to GPU without interrupting the model computation on GPU.

The module can decode images and videos into various color formats (RGB, YUV420p, NV12, etc.) to support different use cases and optimize memory usage for specific applications. See Video Color Space for the supported pixel formats and how to select them.

The following diagram illustrates this workflow:

flowchart subgraph C[Compute Node] direction LR b(Binary String&#10&#40video, audio, image, NumPy etc...&#41) c(CPU array) subgraph G[GPU] g(GPU array) end b e1@<==> |<b>Decoding</b>| c c e2@<==> |<b>Transfer</b>| g e1@{ animation: slow } e2@{ animation: slow } end

The IO module contains efficient implementations for commonly used formats. This section provides detailed documentation on how to use these implementations.

• High-Level Loading Functions
  • Overview
  • Basic Usage
  • Buffer Objects
  • Default Output Formats
  • Customizing Output Format
  • Transferring to GPU
• Decoding Process Overview
  • The Decoding Process
  • Stage 1: Demuxing
  • Stage 2: Decoding
  • Stage 3: Filtering
  • Stage 4: Buffer Conversion
  • Optional: Bitstream Filtering
  • Complete Example
  • Decoding to Different Color Formats
  • Performance Considerations
• Packets and Frames
  • Packets
  • Frames
• Filter Graphs
  • FFmpeg Filters Overview
  • Filter Descriptions
  • Helper Functions
  • Custom Filter Descriptions
  • Common Filter Examples
  • Data Augmentation with Filters
• Video Color Space
  • The filter_desc argument
  • Supported pixel formats
  • Examples
  • Skipping the conversion entirely
  • See also
• Advanced Filter Graphs
  • When to Use FilterGraph Directly
  • FilterGraph Basics
  • Simple FilterGraph Example
  • Multiple Input Graphs
  • Multiple Output Graphs
  • Multimedia Filters
  • Complex Graph Examples
  • Debugging Filter Graphs
• Streaming Decoding
  • Overview
  • Basic Streaming Pattern
  • Streaming Video
  • Streaming Audio
  • Multi-Stream Decoding
  • Streaming with Filtering
• Hardware-Accelerated Video Decoding
  • Overview
  • Basic Usage
  • Hardware-Accelerated Resize and Crop
  • Pixel Format Conversion
  • Memory Management
  • Streaming Decoding
  • Complete Example
  • Performance Considerations
  • Troubleshooting
  • See Also
• Loading NumPy Arrays
  • Overview
  • Loading NPY Files
  • Loading NPZ Files
  • Use Cases
  • Performance Considerations
  • See Also
• Media Encoding
  • The Encoding Process
  • Creating Reference Frames
  • Using Encoders
  • Using the Muxer
  • Customizing Encoders
  • Applying Filters to Reference Frames
  • See Also
• FFmpeg CLI Cheat Sheet
  • DemuxConfig — demux_config(...)
  • DecodeConfig — decode_config(...)
  • VideoEncodeConfig — video_encode_config(...)
  • AudioEncodeConfig — audio_encode_config(...)
  • Encoder selection — Muxer.add_encode_stream(...)
  • Output container — Muxer(...) / Muxer.open(...)
  • Video filters — get_video_filter_desc(...)
  • Audio filters — get_audio_filter_desc(...)
  • Complex filter graphs — FilterGraph
  • CUDAConfig — cuda_config(...)
  • NVDEC / nvJPEG — not a 1:1 of *_cuvid
  • Deliberately not mapped