FabricPerf measures scale-up networks (e.g., GPU rack connected entirely through NVLink). Such a network has low observability problem as NIC and host aren’t involved in the data path. So, it chooses the GPU communication kernel as the profiling interface.

  • Because it’s the scale-up’s transport layer; exposes network semantics (send/recv) to application while accesses fabric via memory semantics (ld/st/fence); provides buffers/protocol/algorithm; lowest SW programmable layer.
  • FabricPerf profiles this layer in three complementary ways.

Key techniques

Communication kernel instrumentation for fine-grained timestamping

  • Prior work, Neutrino, provides timestamp code injection in asm (e.g., smem_realtime).
  • FabricPerf uses it to timestamp transport events, e.g., first st, fence, and polling loop bra, corresponding to different stages of send/recv processing.
  • These timestamps approximate packet-level lifecycle events, despite lack of NIC HW timestamps.

Cross-GPU timestamp alignment using GPTP

  • Timestamps collected on different GPUs need to be synchronized.
  • FabricPerf develops PTP-like mechanism to derive offset ($d$; follower GPU clock - leader GPU clock).
  • Assuming fabric latency as $f$, (1) leader sends msg at $T_1$, follower receives at $T_1^{\ast}$; (2) leader sends $T_1$ value, so follower knows $T_1^{\ast} = T_1 + d + f$; (3) follower sends at $T_2$, leader receives at $T_2^{\ast}$; (4) leader sends $T_2^{\ast}$ value, so follower knows $T_2^{\ast} = T_2 - d + f$; $d$ can be computed.

Modeling scale-up communication as GPU memory traffic

  • As scale-up fabric operates through memory semantics, FabricPerf models each communication operation as a set of memory flows through src buffers, network buffers, LLC, and dst buffers.
  • For a measured fabric bandwidth, it models expected read/write memory BW and LLC hit rate.
  • It then compares with actual CUPTI measurement: e.g., if model predicts high LLC hit rate for network buffer but CUPTI reports low, it could hint inefficiency at GPU memory/cache hierarchy.

The paper performs two case studies and optimizations using FabricPerf:

  • Finds per-channel latency imbalance and long tail; Optimizes with cross-channel work stealing.
  • Finds near 0% LLC hit rate for communication network buffer; Optimizes with eviction hints.

Comments

  • FabricPerf’s main value is turning the opaque scale-up network into a kernel profiling problem.
  • Despite endpoint timing and memory-side effects, the fabric is still a black-box.
  • Microbenchmarks may better for exposing behaviors of links, switches, routing, congestion, etc.