Core Stateless Fair Queuing (CSFQ; Ion Stoica et al., SIGCOMM’98)

  • Edge routers maintain per-flow state; estimate incoming rate $r_i$ and encode it to packet header.
  • Core routers maintain no per-flow state; try to set a fair share $\alpha$ s.t., $ \sum_{i}^{}min(r_i,\alpha) \approx \text{capacity}$.
    • Drops packets probabilistically, if $r_i \gt \alpha$.

Paper’s thesis

  • Switched CXL memory pooling incurs contention between concurrent load/store streams across entire datapath, specifically at host uncore/adapter, CXL switch, and remote adapter.
  • It should be treated as end-to-end transport problem, and by contextualizing the CSFQ, a sender-driven bandwidth admission can improve fairness and utilization.

Key techniques

mChannel represents end-to-end $\text{Core}_i \leftrightarrow \text{remote-DIMM}_j$ memory stream.

For each 100µs scheduling window:

  1. Estimate mChannel’s current rate and unconstrained demand (using per-core CXL request counters).
  2. Obtain mChannel’s fair share: (1) Aggregate traffic at each shared component (e.g., host adapter, CXL switch, and remote adapter) along the path; (2) compute CSFQ fair share for each component; (3) assign mChannel the minimal fair share (limiting its rate by the most congested component).
  3. Enforce through source throttling, specifically by converting into CPU duty cycle (execution/suspension time for the application thread).

Comments

  • MemChannel’s key contributions are: (a) Characterization of various choke points of switched CXL memory system (§2); and (b) Modeling it as an end-to-end transport problem.
  • Its novelty is contextualizing CSFQ into CXL transport, despite it was originally developed for Internet congestion control. For example, given the CXL’s lossless fabric, it replaces the packet drop at core router into a thread duty cycle. Its implementation is a interesting future design reference; the estimated fair share is enforced at the end host in a very simple form: a SW-based, compute-side thread throttling.
  • A minor preference. Although the paper proposes mechanisms to make the throttle injection at host transparent to the user/application using dynamic link library, I’d prefer a clean-state implementation at the adapter’s arbitration policy. But probably there’s no programmability support, so the authors would have fall back to current implementation.
  • The proposed transport mechanism may have constraints in directly applying to networks that need sub-µs control but limited component-level observability (e.g., lack of counters).