GPU Observability

Your GPUs are
lying to you.

nvidia-smi says 100% utilized. You are getting a fraction of the compute you pay for — stalled on memory, comms, and idle kernels. Plasmient shows you the truth, from a single kernel to a trillion-parameter job.

nvidia-smi reports0%GPU "utilization"
actually achieved0%of peak FLOP/s
you are paying for0%idle silicon
pip install plasmient-utilization-truth
Read the thesis →

Open measurement kit. Run it on your own GPU and see the gap yourself.

01 · The gap nobody owns

Every tool watches one layer.
None of them join.

That empty space — from a stalled kernel up to the dollars it burns — is what we measure. One causal chain, not seven disconnected dashboards.

Every tool owns one row. None of them join. Plasmient owns the column — the causal chain from a stalled kernel to the dollars it burns.

02 · Reported ≠ real

Utilization only means a kernel was resident.

It can read 100% while the tensor cores sit idle. We measure achieved work against the hardware's peak — the number that actually tells you what you're paying for.

  • Reported — was a kernel scheduled? (what nvidia-smi shows)
  • Achieved — what fraction of peak FLOP/s you actually got
  • Wasted — GPU-hours you rented and never used
python measure.py --workload memory
utilization-truth · measure.py
python measure.py --workload memory --gpu 0
device   : NVIDIA A100-SXM4-80GB
workload : elementwise (memory-bound)   dtype: fp16
sampling nvml @ 100 Hz over 5.0 s …
 
nvidia-smi utilization ....... 92 %
achieved  (% of peak FLOP/s) .. 3.1 %
DRAM bandwidth ............... 1.72 TB/s  (86% of peak)
tensor-core active ........... 0.0 %
 
VERDICT: nvidia-smi says 92% busy — you are doing 3.1% of peak compute.
         memory-bound. the tensor cores never woke up.
03 · What we build on it

One correlation layer. Five ways in.

The vertical trace is the platform. These are the first products it powers — led by the two highest-conviction bets.

Ahigh conviction

Utilization Truth ★

Prove the gap between reported utilization and achieved compute — continuously, on real clusters.

for ML platform & infra teams

Dhigh conviction

Kernel Profiler ★

Per-kernel achieved-vs-peak for torch.compile / Triton, in your dev loop. The developer beachhead.

for kernel & model engineers

Bpromising

Inference Tail-Latency

Attribute p99 spikes to the kernel, batch, or memory stall that actually caused them.

for inference / serving teams

Cpromising

Straggler Attribution

In multi-node training, find the one rank, link, or kernel dragging the whole all-reduce.

for large-scale training teams

Eexploring

GPU FinOps

Turn wasted GPU-hours into dollars — join achieved work to the cloud bill.

for eng leadership & finance

Conviction reflects where the evidence points today, not a roadmap promise. See each wedge →

04 · Built in the open, learned from zero

To measure every bit that moves,
you have to understand every bit that moves.

So we're learning the whole stack in public — linear algebra → CUDA kernels → the observability build — and rendering it as the Study Lab: interactive chapters you can play with, not walls of text. The lessons are ours, the code runs, the results are measured.

Enter the Study Lab →
Curriculum drawn from open courseware —

Names shown for factual reference. Independent project — not affiliated with or endorsed by these institutions. Attribution →

Watching the gap on your cluster?

We're early and building in the open. Run the kit, read the thesis, or follow the build.

Explore the labsRead the thesis