Add CCL (Collective Communication Library) API model

[xiongxuecheng]

otg_ccl_API_design_V1.md

Summary

Add CCL (Collective Communication Library) API model to OTG for AI fabric collective operation testing. Enables simulation and measurement of NCCL-style AllReduce / AllToAll operations over RoCEv2, with Job Completion Time (JCT) and bandwidth metrics.

---

Feature Overview

• Related Issue: Add CCL (Collective Communication Library) support #489

• Brief Description:

  AI training clusters rely on collective communication primitives (AllReduce, AllToAll, PointToPoint) to synchronize gradients across GPUs. This PR adds a first-class Ccl model to OTG, allowing test tools to:

  • Configure collective operations with NCCL-compatible algorithms (Ring AllReduce, Halving-Doubling, AllToAll, etc.)
  • Support single-stage and multi-stage (LLM training pipeline) scenarios via an ordered stages[] model
  • Control CCL traffic lifecycle (start / stop) via State.Traffic.ccl
  • Measure JCT, bus bandwidth, tail latency, and per-iteration drop count via GetMetrics

---

Feature Details

OTG Location	New Object	Description
Config.ccl	Ccl.Config	Top-level CCL configuration (single CCL job)
Ccl.Config.stages[]	Ccl.Stage	Ordered stage list; communicators within a stage run in parallel
Ccl.Stage.algorithm	Ccl.Algorithm	Stage-level algorithm default (choice + phase)
Ccl.Stage.communicators[]	Ccl.Communicator	Rank group definitions; may override stage-level algorithm
Ccl.Stage.duration	Ccl.Stage.Duration	Run control per stage: fixed_iterations or fixed_seconds
Ccl.Communicator.generation	Ccl.Generation	Data size, compute gap, QP scheduling
Ccl.Communicator.rocev2	Ccl.Rocev2Policy	RoCEv2 transport: encapsulation, connection type, IB MTU, QPs per peer
State.Traffic.ccl	State.Traffic.Ccl	CCL traffic control (start / stop)
MetricsRequest.ccl	Ccl.Metrics.Request	CCL metrics request with communicator / stage filters
MetricsResponse.ccl_metrics[]	Ccl.Metric	Per-communicator JCT / BW results

---

Code Snippet

// Configure a 2-rank Ring AllReduce over RoCEv2 (back-to-back smoke test)
api := gosnappi.NewApi()
cfg := api.NewConfig()

for _, p := range []string{"p0", "p1"} {
    cfg.Ports().Add().SetName(p).SetLocation("...")
}

devices := []string{"dev0", "dev1"}
for i, d := range devices {
    dev := cfg.Devices().Add().SetName(d)
    eth := dev.Ethernets().Add().SetName(d+".eth").SetPortName([]string{"p0", "p1"}[i])
    eth.Ipv4Addresses().Add().SetName(d+".ip").
        SetAddress(fmt.Sprintf("192.168.1.%d", i+1)).
        SetGateway("192.168.1.254").SetPrefix(24)
}

ccl := cfg.Ccl().
    SetName("ring_allreduce_test").
    SetTransport("rocev2").
    SetSendMode("async")

stage := ccl.Stages().Add().SetName("stage0")
stage.Algorithm().SetChoice("ring_allreduce")
stage.Duration().FixedIterations().SetCount(10).SetIntervalMs(1000)

comm := stage.Communicators().Add().
    SetName("comm0").
    SetDeviceNames(devices)

comm.Generation().
    SetDataSize(160).
    SetDataSizeUnit("gigabyte")

comm.Rocev2().
    SetEncapsulation("ipv4").
    SetIbMtu(4096).
    SetQpsPerPeer(64)

api.SetConfig(cfg)

cs := gosnappi.NewControlState()
cs.Traffic().Ccl().SetState("start")
api.SetControlState(cs)

mr := gosnappi.NewMetricsRequest()
mr.Ccl().SetCommunicatorNames([]string{"comm0"})
resp, _ := api.GetMetrics(mr)
for _, m := range resp.CclMetrics().Items() {
    fmt.Printf("JCT=%.3fs  BusBW=%.2fGbps  JCT%%=%.1f  Drops=%d\n",
        m.Jct(), m.BusBw(), m.JctPercent(), m.DropCount())
}

---

Test Specification

**Example test scenario: 2-port B2B Ring AllReduce **

Topology: Two ports connected back-to-back, simulating a 2-rank AllReduce collective.

Configuration:

• Ccl.Config.transport = rocev2, send_mode = async
• Ccl.Stage.algorithm.choice = ring_allreduce
• Ccl.Stage.duration = fixed_iterations, count = 10, interval_ms = 1000
• Ccl.Communicator.generation.data_size = 160, data_size_unit = gigabyte
• Ccl.Communicator.rocev2.encapsulation = ipv4, ib_mtu = 4096, qps_per_peer = 64
• One Ccl.Communicator with device_names = ["dev0", "dev1"]

Control:

• State.Traffic.ccl.state = start → triggers ARP resolution + RC QP setup + 10 iterations
• State.Traffic.ccl.state = stop → interrupts execution

Verification via GetMetrics:

Metric	Expected
jct_percent	>= 90% (near ideal)
bus_bw	Close to line rate x algorithm coefficient 2(N-1)/N
drop_count	0 (lossless fabric)
iteration_count	10
max_tail_latency_us	Within configured threshold

Open Questions

1. RoCEv2 UD Transport — The proposal currently reserves Ccl.Rocev2Policy.connection_type = unreliable_datagram pending Device.Rocev2Peer UD support. Can CCL-level UD parameters be defined independently of single-flow UD support?

2. Ultra Ethernet Transport (UET) — We reserve Config.ccl.transport = uet pending UEC v1.0 ratification. Can UET be supported exclusively at the CCL level without requiring a single-flow UET API as a prerequisite?