Development Notes

• Implementation detail of Pipeline
  • Architecture Overview
  • Async IO
  • Queue vs Async Queue as Buffer
  • Thread, loop and task
  • Building Pipeline from Configuration
    • Overview of the Build Process
    • Detailed Build Steps
      • Step 1: Configuration to Node Graph
      • Step 2: Recursive Coroutine Creation
      • Step 3: Main Coroutine Assembly
      • Step 4: Task Creation and Monitoring
      • Step 5: Error Handling and Cleanup
      • Step 6: Pipeline Object Creation
  • API reference
    • _Node
      • _Node.cfg
      • _Node.coro
      • _Node.create_task()
      • _Node.name
      • _Node.queue
      • _Node.task
      • _Node.upstream
    • _convert_config()
    • _build_node_recursive()
    • _build_node()
    • _source()
    • _merge()
    • _pipe()
    • _ordered_pipe()
    • _sink()
    • _run_pipeline_coroutines()
• Remote Iterable Protocol
  • Worker State
  • Helper functions and data structures
    • _Cmd
      • _Cmd.ABORT
      • _Cmd.START_ITERATION
      • _Cmd.STOP_ITERATION
    • _Status
      • _Status.INITIALIZATION_FAILED
      • _Status.INITIALIZATION_SUCCEEDED
      • _Status.ITERATION_FINISHED
      • _Status.ITERATION_STARTED
      • _Status.ITERATOR_FAILED
      • _Status.ITERATOR_SUCCESS
      • _Status.UNEXPECTED_CMD_RECEIVED
    • _enter_iteration_mode()
    • _execute_iterable()
    • _drain()
    • _iterate_results()
    • _SubprocessIterable
      • _SubprocessIterable.__iter__()
• PyTorch CUDA Race Condition in Multi-threading
  • Background
  • The Problem
  • The Solution: Caching the reference in transfer_tensor
  • References