GPU utilization is a lie —
and nobody owns the fix.
nvidia-smi's "utilization" only means a kernel was resident on the device. A memory-bound job can pin it at 100% while the tensor cores — the expensive silicon you rent — sit at 0%. The number everyone trusts to bill, autoscale, and plan capacity is measuring the wrong thing.
Every layer of the stack has a tool. None of them join. The causal chain from a stalled kernel to the dollars it burns is nobody's product. That column is Plasmient.
- Model step
loss, tokens/s, step timeowned today by W&B / TensorBoard - Framework op
aten::mm, fused kernelsowned today by PyTorch profiler / Kineto - CUDA kernel
launch, occupancy, durationowned today by Nsight Systems - GPU counters
TENSOR_ACTIVE, DRAM, SMowned today by DCGM / CUPTI - Interconnect
NCCL all-reduce, NVLinkowned today by NCCL logs - Cluster
node health, stragglersowned today by Prometheus / Grafana - Cost & energy
$ / GPU-hour, wattsowned today by the cloud bill
Every tool owns one row. None of them join. Plasmient owns the column — the causal chain from a stalled kernel to the dollars it burns.
The market gap
Each row is well served. The column — the join across rows — is empty.
| Layer | Owned by | Sees | Blind to |
|---|---|---|---|
| Model / training | W&B, TensorBoard | loss, throughput | why a step is slow |
| Framework | PyTorch profiler, Kineto | op timeline (on demand) | continuous, in prod |
| Kernel | Nsight Systems/Compute | one trace, opened by hand | always-on correlation |
| Device counters | DCGM, CUPTI | SM/DRAM/tensor active | tied back to your code |
| Cluster | Prometheus, Grafana | node health | per-kernel causality |
| Cost | the cloud bill | GPU-hours billed | GPU-hours wasted |
The signal that tells the truth
The gap is measurable today. DCGM field DCGM_FI_PROF_GR_ENGINE_ACTIVE(reported "busy") diverges hard from DCGM_FI_PROF_PIPE_TENSOR_ACTIVE(actual tensor-core work). Join that to CUPTI/Kineto op traces and you can point at the exact kernel wasting the GPU. Our open utilization-truth kit does the first half of this on a single device — run it and watch the two numbers split.
The bet
Prove the gap on real clusters, then build the always-on correlation layer that closes it. The two highest-conviction entry points — Utilization Truth and a kernel profilerfor torch.compile/Triton — are on the product page.