Observability
the layer that measures the other seven

00 One number, and the axis it hides

Walk back up everything you just read. Training pins the cluster at 100% while a third of it is pipeline bubble and all-reduce wait. Serving reads 100% through a memory-bound decode that touches a few percent of the tensor cores.RL holds a flat 100% while it oscillates between a starved generation phase and a saturated update. MoE reads 100% on an expert that fires on 3% of tokens. Every layer, the same tell: the dashboard is green, and green means almost nothing.

The reason is a single missing axis. nvidia-smi reports utilization — the fraction of recent time the GPU had at least one kernel resident. It does not report achieved fraction of peak — the useful FLOP/s divided by what the silicon can do. A kernel stalled on HBM the entire time it runs is 100% utilized and ~2% useful. Pick any workload from across this site and watch the two numbers come apart:

That right-hand number is what every lab on this site was quietly about. It is not exotic to compute — you need the achieved FLOP/s of the kernel and the peak of the card it ran on — but it is nowhere on the dashboards the industry watches. Recovering it, per kernel, at fleet scale, is the wedge.

01 The vertical trace — the one idea the product is

Here is the shape of the problem the gauge above doesn’t show: the answer to “why was this busy moment useless?” lives in a different layer than the moment itself. The device counter says “100% util.” The reason is a NCCL all-reduce (a cluster-layer fact), triggered by an optimizer step (a model-layer fact), that stalled the tensor cores (a kernel-layer fact). No single tool holds all four. The signature idea of Plasmient is to put them on one clock and one timeline — avertical trace — so the cause and the symptom finally sit in the same column. Watch a training step fall through the layers:

Read one column top to bottom and you get a sentence the dashboard can never say: “at t = 6.2 ms the device is 99% utilized and 4% of peak, because the kernel is idle, because the op is an all-reduce, because the cluster is mid-collective.” That vertical join — kernel ↔ op ↔ model ↔ cluster, correlated on a shared clock — is the whole invention. Everything else is plumbing to make it cheap and always-on.

02 Where the numbers actually come from

None of the four signals are secret; they are just scattered. Each layer already emits something — the hard, unsolved part is the join: giving every event a shared timestamp and a correlation ID so a device stall can be traced up to the op and collective that caused it. That is the same trick distributed tracing (OpenTelemetry) plays for microservices, applied downward to silicon.

  layer            signal it emits              source                the join key
  ──────────────   ──────────────────────────   ───────────────────   ──────────────
  cluster          collective start/stop, size  NCCL / RCCL hooks     step + rank
  model / op       op name, shape, launch site  framework profiler    launch timestamp
  kernel           duration, achieved FLOP/s    CUPTI / CUDA events    correlation id
  device           SM active, DRAM active, %    NVML / DCGM           device clock
  ─────────────────────────────────────────────────────────────────────────────────
  the product      one timeline, one clock  ◀── correlate all four ──▶  vertical trace
                   + roofline ceiling per op → achieved % of peak per kernel

Nsight Systems already lines up CUDA and NCCL on one timeline for a single run — proof the join is possible. What does not exist is that join running continuously, in production, at fleet scale, with the roofline ceiling attached so every kernel carries its achieved-fraction-of-peak next to the utilization everyone already trusts. That is the gap the landscape of ~98 tools leaves open, and the one thing this company builds.

03 What it says out loud — the verdict

The output of all of this is not another dashboard of green bars. It is a verdict: for a given run, the fraction of peak actually achieved, the layer that stole the rest, and what to change. The number the industry celebrates and the number that pays the bill, side by side, with the cause named.

  ┌─ plasmient · run 4a1c · training step ─────────────────────────────┐
  │  nvidia-smi utilization ........ 99%   ← what the dashboard shows   │
  │  achieved fraction of peak ..... 41%   ← what you actually paid for │
  │                                                                     │
  │  where the other 59% went:                                         │
  │    all-reduce wait (NCCL) ...... 31%   overlap comm with backward   │
  │    pipeline bubble ............. 14%   raise microbatch count       │
  │    memory-bound kernels ........ 14%   fuse; raise arithmetic int.  │
  │                                                                     │
  │  VERDICT: busy, not useful. 2.4× headroom before new hardware.     │
  └─────────────────────────────────────────────────────────────────────┘

That last line is the sales pitch and the engineering result at once. A cluster reading 100% looks maxed-out — the only apparent fix is buying more of it. The vertical trace says otherwise: most of the “busy” was tax, and the tax is addressable in software. The gap is the headroom, and naming it is worth more than the next rack of H100s.

04 Why the whole map ended here