Contents Menu Expand Light mode Dark mode Auto light/dark, in light mode Auto light/dark, in dark mode Skip to content
SPDL 0.1.1 documentation
SPDL 0.1.1 documentation
  • Home

Contents

  • Overview
  • Installation
  • Getting Started
    • Building and Running Pipeline
    • Pipeline Stages
    • Concurrency
    • Pipeline Parallelism
    • Collecting Runtime Statistics
  • Optimization Guide
    • Headspace Analysis
    • Noisy Neighbour
    • Composition of Pipeline
    • Analyzing the Performance
    • Metrics across Ranks
    • Resolving the Bottleneck
  • Case Studies
    • Parallelism and Performance
    • Data Format and Performance
  • Migration Guide
    • Why migrate to SPDL?
    • Paradigm Shift
    • Migrating from PyTorch DataLoader
  • Best Practices
  • Examples
    • Image dataloading
    • Video dataloading
    • Imagenet classification
    • Multi thread preprocessing
    • Streaming nvdec decoding
    • Streaming video processing
    • Performance analysis
    • Data formats
  • Frequently Asked Questions

API References

  • API Reference
    • spdl.pipeline
    • spdl.dataloader
    • spdl.source
    • spdl.io
    • spdl.io.utils
  • API Reference (C++)
    • Class List
    • File List
    • API
      • Namespace spdl
      • Namespace spdl::core
      • Namespace spdl::core::detail
      • Namespace std
      • Struct AVRational
      • Struct BytesAdaptor
      • Struct CPUBuffer
      • Struct DataInterface
      • Struct DecodeConfig
      • Struct DemuxConfig
      • Template Struct EncodeConfigBase
      • Template Struct Generator
      • Struct Generator::promise_type
      • Template Struct Packets
      • Struct RawPacketData
      • Struct SourceAdaptor
      • Struct Storage
      • Template Class BSF
      • Template Class Codec
      • Class CPUStorage
      • Template Class Decoder
      • Class Demuxer
      • Template Class DecoderImpl
      • Template Class EncoderImpl
      • Template Class Encoder
      • Class FilterGraph
      • Template Class Frames
      • Class InternalError
      • Class Muxer
      • Class PacketSeries
      • Class StreamingDemuxer
      • Class TracingSession
      • Enum CodecID
      • Enum ElemClass
      • Enum MediaType
      • Template Function spdl::core::clone
      • Template Function spdl::core::convert_frames(const std::vector<const Frames<media> *>&, std::shared_ptr<CPUStorage>)
      • Template Function spdl::core::convert_frames(const Frames<media> *, std::shared_ptr<CPUStorage>)
      • Function spdl::core::cpu_buffer
      • Function spdl::core::create_reference_audio_frame
      • Function spdl::core::create_reference_video_frame
      • Function spdl::core::extract_packets_at_indices
      • Function spdl::core::get_ffmpeg_filters
      • Function spdl::core::get_ffmpeg_log_level
      • Function spdl::core::get_ffmpeg_versions
      • Template Function spdl::core::get_pts
      • Function spdl::core::init_glog
      • Function spdl::core::init_tracing
      • Function spdl::core::make_demuxer(const std::string&, const SourceAdaptorPtr&, const std::optional<DemuxConfig>&)
      • Function spdl::core::make_demuxer(const std::string_view, const std::optional<DemuxConfig>&)
      • Function spdl::core::make_filter_graph
      • Function spdl::core::register_avdevices
      • Function spdl::core::set_ffmpeg_log_level
      • Template Function spdl::core::trace_counter
      • Function spdl::core::trace_event_begin
      • Function spdl::core::trace_event_end
      • Define _IS_AUDIO
      • Define _IS_IMAGE
      • Define _IS_VIDEO
      • Define SPDL_DEFAULT_BUFFER_SIZE
      • Typedef AVRational
      • Typedef spdl::core::AnyFrames
      • Typedef spdl::core::AnyPackets
      • Typedef spdl::core::AudioCodec
      • Typedef spdl::core::AudioEncodeConfig
      • Typedef spdl::core::AudioEncoder
      • Typedef spdl::core::AudioEncoderPtr
      • Typedef spdl::core::AudioFrames
      • Typedef spdl::core::AudioFramesPtr
      • Typedef spdl::core::AudioPackets
      • Typedef spdl::core::AudioPacketsPtr
      • Typedef spdl::core::CPUBufferPtr
      • Typedef spdl::core::DataInterfacePtr
      • Typedef spdl::core::DecoderPtr
      • Typedef spdl::core::DemuxerPtr
      • Typedef spdl::core::EncoderPtr
      • Typedef spdl::core::FilterGraphPtr
      • Typedef spdl::core::FramesPtr
      • Typedef spdl::core::ImageCodec
      • Typedef spdl::core::ImageFrames
      • Typedef spdl::core::ImageFramesPtr
      • Typedef spdl::core::ImagePackets
      • Typedef spdl::core::ImagePacketsPtr
      • Typedef spdl::core::MuxerPtr
      • Typedef spdl::core::OptionDict
      • Typedef spdl::core::PacketSeriesPtr
      • Typedef spdl::core::PacketsPtr
      • Typedef spdl::core::Rational
      • Typedef spdl::core::SourceAdaptorPtr
      • Typedef spdl::core::StreamingDemuxerPtr
      • Typedef spdl::core::VideoCodec
      • Typedef spdl::core::VideoEncodeConfig
      • Typedef spdl::core::VideoEncoder
      • Typedef spdl::core::VideoEncoderPtr
      • Typedef spdl::core::VideoFrames
      • Typedef spdl::core::VideoFramesPtr
      • Typedef spdl::core::VideoPackets
      • Typedef spdl::core::VideoPacketsPtr
  • API Index

Development Notes

  • Development Notes
    • Implementation detail of Pipeline
  • Versions
    • dev
    • latest-release
    • 0.1.1
    • 0.1.0
    • 0.0.14
    • 0.0.13
    • 0.0.12
    • 0.0.11
    • 0.0.10
    • 0.0.9
    • 0.0.8
Back to top
View this page

Getting Started¶

  • Building and Running Pipeline
    • Building a Pipeline
    • Running a Pipeline
    • ⚠ Caveats ⚠
  • Pipeline Stages
    • Source
    • Processing
    • Sink
  • Concurrency
    • Stage Concurrency
    • Thread Pool Size
  • Pipeline Parallelism
    • Specifying an executor
    • Multi-threading (default)
    • Multi-threading (custom)
    • Multi-processing (stage)
    • Multi-processing (combined)
    • Multi-threading in subprocess
  • Collecting Runtime Statistics
Next
Building and Running Pipeline
Previous
Installation
Copyright © 2025, Meta Platforms, Inc.
Made with Sphinx and @pradyunsg's Furo