Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode Skip to content
SPDL 0.1.4 documentation
SPDL 0.1.4 documentation
  • Home

Contents

  • Overview
  • Installation
  • Getting Started
    • Building and Running Pipeline
    • Pipeline Stages
    • Concurrency
    • Pipeline Parallelism
    • Collecting Runtime Statistics
  • Introduction to Async I/O
    • Implementing Orchestration Mechanism
    • The Event Loop
    • The async and await keywords
    • Running Synchronous Functions
  • Optimization Guide
    • Headspace Analysis
    • Noisy Neighbour
    • Composition of Pipeline
    • Analyzing the Performance
    • Metrics across Ranks
    • The Straggler
    • Resolving the Bottleneck
  • Case Studies
    • Parallelism and Performance
    • Data Format and Performance
  • Migration Guide
    • Why migrate to SPDL?
    • Paradigm Shift
    • Migrating from PyTorch DataLoader
  • Best Practices
  • Examples
    • Image dataloading
    • Video dataloading
    • Imagenet classification
    • Multi thread preprocessing
    • Streaming nvdec decoding
    • Streaming video processing
    • Performance analysis
    • Data formats
    • Hydra integration
  • Frequently Asked Questions

API References

  • API Reference
    • spdl.pipeline
    • spdl.pipeline.defs
    • spdl.dataloader
    • spdl.source
    • spdl.io
    • spdl.io.utils
  • API Reference (C++)
    • Class List
    • File List
    • API
      • Namespace spdl
      • Namespace spdl::core
      • Namespace spdl::core::detail
      • Namespace std
      • Struct AVRational
      • Struct BytesAdaptor
      • Struct CPUBuffer
      • Struct DataInterface
      • Struct DecodeConfig
      • Struct DemuxConfig
      • Template Struct EncodeConfigBase
      • Template Struct Generator
      • Struct Generator::promise_type
      • Template Struct Packets
      • Struct RawPacketData
      • Struct SourceAdaptor
      • Struct Storage
      • Template Class BSF
      • Template Class Codec
      • Class CPUStorage
      • Template Class Decoder
      • Class Demuxer
      • Template Class DecoderImpl
      • Template Class EncoderImpl
      • Template Class Encoder
      • Class FilterGraph
      • Template Class Frames
      • Class InternalError
      • Class Muxer
      • Class PacketSeries
      • Class StreamingDemuxer
      • Class TracingSession
      • Enum CodecID
      • Enum ElemClass
      • Enum MediaType
      • Template Function spdl::core::clone
      • Template Function spdl::core::convert_frames(const std::vector<const Frames<media> *>&, std::shared_ptr<CPUStorage>)
      • Template Function spdl::core::convert_frames(const Frames<media> *, std::shared_ptr<CPUStorage>)
      • Function spdl::core::cpu_buffer
      • Function spdl::core::create_reference_audio_frame
      • Function spdl::core::create_reference_video_frame
      • Function spdl::core::extract_packets_at_indices
      • Function spdl::core::get_ffmpeg_filters
      • Function spdl::core::get_ffmpeg_log_level
      • Function spdl::core::get_ffmpeg_versions
      • Template Function spdl::core::get_timestamps
      • Function spdl::core::init_glog
      • Function spdl::core::init_tracing
      • Function spdl::core::make_demuxer(const std::string&, const SourceAdaptorPtr&, const std::optional<DemuxConfig>&)
      • Function spdl::core::make_demuxer(const std::string_view, const std::optional<DemuxConfig>&)
      • Function spdl::core::make_filter_graph
      • Function spdl::core::register_avdevices
      • Function spdl::core::set_ffmpeg_log_level
      • Template Function spdl::core::trace_counter
      • Function spdl::core::trace_event_begin
      • Function spdl::core::trace_event_end
      • Define _IS_AUDIO
      • Define _IS_IMAGE
      • Define _IS_VIDEO
      • Define SPDL_DEFAULT_BUFFER_SIZE
      • Typedef AVRational
      • Typedef spdl::core::AnyFrames
      • Typedef spdl::core::AnyPackets
      • Typedef spdl::core::AudioCodec
      • Typedef spdl::core::AudioEncodeConfig
      • Typedef spdl::core::AudioEncoder
      • Typedef spdl::core::AudioEncoderPtr
      • Typedef spdl::core::AudioFrames
      • Typedef spdl::core::AudioFramesPtr
      • Typedef spdl::core::AudioPackets
      • Typedef spdl::core::AudioPacketsPtr
      • Typedef spdl::core::CPUBufferPtr
      • Typedef spdl::core::DataInterfacePtr
      • Typedef spdl::core::DecoderPtr
      • Typedef spdl::core::DemuxerPtr
      • Typedef spdl::core::EncoderPtr
      • Typedef spdl::core::FilterGraphPtr
      • Typedef spdl::core::FramesPtr
      • Typedef spdl::core::ImageCodec
      • Typedef spdl::core::ImageFrames
      • Typedef spdl::core::ImageFramesPtr
      • Typedef spdl::core::ImagePackets
      • Typedef spdl::core::ImagePacketsPtr
      • Typedef spdl::core::MuxerPtr
      • Typedef spdl::core::OptionDict
      • Typedef spdl::core::PacketSeriesPtr
      • Typedef spdl::core::PacketsPtr
      • Typedef spdl::core::Rational
      • Typedef spdl::core::SourceAdaptorPtr
      • Typedef spdl::core::StreamingDemuxerPtr
      • Typedef spdl::core::VideoCodec
      • Typedef spdl::core::VideoEncodeConfig
      • Typedef spdl::core::VideoEncoder
      • Typedef spdl::core::VideoEncoderPtr
      • Typedef spdl::core::VideoFrames
      • Typedef spdl::core::VideoFramesPtr
      • Typedef spdl::core::VideoPackets
      • Typedef spdl::core::VideoPacketsPtr
  • API Index

Development Notes

  • Development Notes
    • Implementation detail of Pipeline
  • Versions
    • dev
    • latest-release
    • 0.1.4
    • 0.1.3
    • 0.1.2
    • 0.1.1
    • 0.1.0
    • 0.0.14
    • 0.0.13
    • 0.0.12
    • 0.0.11
    • 0.0.10
    • 0.0.9
    • 0.0.8
Back to top
View this page

API ReferenceΒΆ

spdl.pipeline

Implements Pipeline, a generic task execution engine.

spdl.pipeline.defs

This module provides building block definitions for Pipeline.

spdl.dataloader

Task specific data loading solutions based on Pipeline.

spdl.source

Iterables for traversing datasets and utilities for transforming them.

spdl.io

Implements the core I/O functionalities.

spdl.io.utils

Utility functions.

Next
spdl.pipeline
Previous
Frequently Asked Questions
Copyright © 2025, Meta Platforms, Inc.
Made with Sphinx and @pradyunsg's Furo