The paper’s key thesis is that DMAs are known to be good for bulk transfers, but bad for small ones due to non-copy-related overheads. So, it tries batching of DMA commands to lower that cost.

GPU DMA engines (DMAs; AMD SDMA, NVIDIA CE)

  • Hardware unit only for communication and data transfers
  • DMAs are off-CTA so doesn’t interfere on CUs, but still interferes on memory subsystem resources.
  • Traditionally, only triggered by host-side API.
    • MSCCL++: Enables GPU kernel to indirectly request DMA operation via a GPU-CPU FIFO channel polled by host proxy that actually triggers the DMAs.
  • Recently, device-side API was added (“shader-initiated bypass”): ROCm XIO, NCCL 2.28

DMAs execution phases (for host-side API, which is DMA-Latte’s target)

  • “Control”: hipMemcpyAsync -> CPU enqueues command (cmd) to request queue of a specific DMA.
  • “Schedule”: CPU rings doorbell, updating the DMA’s doorbell pointer to the cmd position in the queue, followed by CPU to DMA handover, awaking the DMA to fetch the cmd.
  • “Copy”: The DMA decodes the cmd, translates addrs, issues ld/st instructions to perform the copy.
  • “Sync”: On copy completion, the DMA atomic-writes 64b to specific location to notify the CPU.

Prior work having been using the DMAs mostly for peer-to-peer copies or GPU-CPU copies (e.g., Strata (OSDI’26): KV cache offloading)

  • The DMAs have been known for low performance for small-size (< MB) transfers due to non-“Copy” phase overheads, and hence wasn’t used for small-sized, latency-bound collectives.
  • DMA-Latte reveals the untapped DMA cmds besides the copy, constructs AG/A2A using those cmds in a few ways, and demonstrates these options performance across various sizes (1KB – 4GB).
    • Beside copy (single src->single dst), the untapped cmds are: bcast (single src->two dst), swap (peer<->peer), back2back (multiple independent copies before single sync), prelaunch (pre-launch DMA command to DMA command queue for hiding DMA overheads)

Comments

  • Fig. 13/14 and §5.2.3-5.2.8, demonstrating the performance of DMA cmd options for collectives across a wide communication size spectrum and explaining the reasons, is very well analyzed.