Direct Memory Access (DMA)

  • Method to transfer packets between NIC and CPU, reading/writing packets via a DMA ring buffer.
  • Optimized with Direct Cache Access (DCA), provisioning specific LLC ways for it (aka DCA ways).
    • The problem is DCA write miss, e.g., when DCA ways are full, it evicts un-processed packet to DRAM (“DMA leak”). The DMA leak incurs even worse performance than non-DCA because the NIC waits the eviction, then writes the new packet, and the CPU later reloads the evicted packet from DRAM.

Sub-NUMA Clustering (SNC)

  • Method to expose each chiplet as NUMA node to the OS and preserve locality.
    • e.g., A core accesses only the LLC slices and DRAM partitions within the NUMA node (unless explicitly instructed otherwise).
  • Trades off memory access latency and memory capacity.

The paper has strong empirical motivation: SNC benefits networking normally; but when a long burst of packets triggers DMA leak, performance can worsen than non-SNC. Specifically, it shows that SNC underperforms when the DMA buffer footprint exceeds the single chiplet’s DCA way capacity. On the evaluated platform, with 2MB DCA capacity per chiplet and a 100Gb/s arrival rate, the crossover occurred at 400µs of burst duration.

TiNA presents a tiered network buffer system for chiplet-based CPUs. While remaining in SNC mode, it extends the networking stack to allocate DMA buffers not only locally but also in remote chiplets, forming local and remote tiers backed by their respective DCA ways. TiNA selectively places packets in the remote tier when local placement would cause DMA leaks (NIC’s RSS queue-steering logic allows to direct packets to desired DMA ring using a programmable hash function). Thus, TiNA systematically combines the benefits of SNC and non-SNC across varying workload characteristics, such as burst length and network load, tailored for multi-chiplet CPUs.