AI Agent Workflow Collection

Investigate a GPU or CPU workload through the zymtrace MCP. The MCP does most of the analysis. This skill enforces the cross-view discipline -- always pull the matching opposite-side flamegraph (CPU for GPU workloads, GPU for CPU workloads) with the same filter. Most bottlenecks hide on the side the customer didn't ask about. Triggers on "analyze my GPU workload", "where's the bottleneck in vllm", "investigate my training job", "find the hot kernel", "GPU isn't saturated", "investigate using flamegraph", "use zymtrace mcp to analyze", or any combination of "analyze / investigate / bottleneck / hot kernel" with "GPU / CPU / vllm / training / flamegraph / zymtrace".

Scaffold a new evidence bundle directory ready for reproducibility-grade evidence capture: SOURCE.md (operator + cluster + git SHA + UTC timestamp), summary.md (verdict skeleton), commands/ (for the four-file .cmd/.stdout/.stderr/.exit tuple capture per shell command). Workload-agnostic. Works for any experiment family. Operator names family + run-id (or accepts default). Triggers on "new evidence bundle", "init evidence", "scaffold bundle", "start a new artifact bundle", "new run-id", "evidence-bundle-init", "set up a bundle", or any combination of "new / init / scaffold / create / start" with "evidence / bundle / artifact / run-id / experiment".

Reproduce the Artificial Analysis (AA) language-model performance workload shapes against an OpenAI-compatible chat endpoint using NVIDIA AIPerf. Drives the three AA text shapes (1k input / >=1k answer, 10k / >=1.5k, 100k / >=2k) with temperature 0, topp 1, and the vLLM-style mintokens + ignoreeos "at least N answer tokens" guarantee. Two modes: synthetic (AIPerf generates the prompt at the token mean) and dataset-replay (a generated o200kbase-counted JSONL replayed identically). Ships a self-contained script and an aa perftunereport cellrun backend. Use when comparing a hosted inference endpoint to AA leaderboard numbers or reproducing AA's methodology. Triggers on "artificial analysis workload", "AA benchmark", "AA 1k/10k/100k shapes", "reproduce artificialanalysis.ai", "AA methodology", "aa-10k", "compare to AA leaderboard", or any combination of "artificial analysis / AA" with "workload / shape / benchmark / dataset".

SLA-first GPU capacity sizing for a serving deployment: given a tokens-per-minute (TPM) target AND the interactivity SLA (output tokens/s/user), compute the pods and GPUs needed from a model's measured tok/s/user-vs-concurrency curve. Sizing MUST start from the tok/s/user SLA, not the TPM number alone: the SLA picks the concurrency, which sets per-pod throughput, which sets the GPU count. Also translates validated optimizations into delta-GPUs at the SLA-pinned concurrency. Backed by the standalone capacitysizing.py tool. Use to answer "how many GPUs for N TPM", "size this for the customer", "pods for the SLA", or to sanity-check a sizing. Triggers on "capacity sizing", "how many GPUs", "how many pods", "tokens per minute sizing", "TPM target", "size for the customer", "interactivity SLA", "tokens per second per user", "GPU count for throughput", or any combination of "size / capacity / how-many" with "GPU / pod / TPM / SLA / tok-per-user / concurrency".

Correlate DCGM Prometheus byte-traffic counters with an inference-perf-bench sweep window to compute byte-grounded workload-level Speed-of-Light. The third tier of the SoL rigor hierarchy (after zymtrace sample-share and ncu per-kernel arithmetic intensity). Reads the sweep window from inferenceperfbenchv1.json.bench, queries Prometheus via the Prometheus MCP for the DCGM PROF group (DRAMACTIVE, NVLINKTX/RXBYTES, PIPETENSORACTIVE, PIPEFP16ACTIVE), falls back to DCGMFIDEV counter-tier metrics when PROF is not exported, and writes /dcgmcorrelation.json with per-resource %SoL = real byte traffic / (peak duration ngpus). Triggers on "dcgm correlate", "dcgm sol", "byte-grounded sol", "workload %sol", "dram active over sweep", "nvlink bytes", "tensor pipe active", "real-vs-peak workload bandwidth", or combinations of "dcgm / prometheus" with "sweep / window / sol / workload / byte-traffic".

Measure the low-concurrency (c=1..c=8) decode hot-path of a live vLLM pod FAST and CORRECTLY: where each token's time actually goes (GPU-busy vs host-idle gap vs comm), whether the workload is kernel-bound / host-bound / comm-bound, and what the addressable lever is. Uses vLLM's native /startprofile + /stopprofile HTTP endpoints (enable once, then restart-free shell-in captures in seconds). Encodes the correctness gates that prevent contaminated or under-counted budgets (clean single-stream driver, GPU-busy must include CUDA-graph execution, reconcile against driver-measured TPOT, reject load-time/lull/truncated captures). Triggers on "decode step budget", "c=1 hot path", "where does decode time go", "TPOT breakdown", "is decode kernel-bound or host-bound", "latency-tier profile", "GPU idle during decode", "startprofile", "torch profiler vllm", or any combination of "decode / c=1 / latency / TPOT / ITL" with "profile / budget / breakdown / hot-path / where" on a vllm deploy.

Render cross-model fleet leaderboards from the local perf-report campaigns in ONE command (perftunereport fleetleaderboard): a latency tier (aa-1k/10k/100k tok/s/user + TTFT + cost at c=1/c=10), a throughput tier (peak tok/s/GPU per (model,quant,TP) with latency + $/1M-tok), and a decision capstone (the perf Pareto frontier across decode-latency x first-token x throughput, with the perf-dominated set and a hard PERF != quality caveat). Auto-discovers every model's AA + roofline cells in campaigns//atlas.jsonl, so re-running refreshes as new campaigns publish. Use to answer "which model do I pick" / "rank the fleet on latency/throughput/cost" / "is model X perf-dominated". Triggers on "fleet leaderboard", "which model should I use", "cross-model comparison", "rank the fleet", "model selection guide", "pareto frontier of models", "perftunereport fleetleaderboard", or any combination of "fleet / cross-model / which-model / leaderboard / pareto" with "latency / throughput / cost / pick / rank / compare".

Diff the compiled FX / Inductor graphs across two vLLM versions or two helm configs to see exactly which fused kernels / passes / partitions changed. Uses torch.dynamo.explain + torch.compile's graph-dump hooks. Useful when you anticipate a model-graph-level change (e.g. swapping cudagraphmode, enabling fuseallreducerms, toggling passconfig.fuseattnquant). Triggers on "graph diff", "dynamo explain", "fx graph", "compile graph", "torch.compile diff", "compilation pass change", or any combination of "graph / fx / inductor / dynamo" with "diff / compare / explain / dump".

Capture per-kernel CUDA hardware-counter data (occupancy, achieved warps active, regs/thread, smem/block, DRAM throughput, arithmetic intensity, branch divergence %, warp-stall reason %) from a live vLLM inference pod via an ncu debug sidecar (no production image rebuild). Outputs .ncu-rep + per-kernel CSV. Scoped via --kernel-name to avoid the 10-100x kernel-replay slowdown that an unscoped ncu attach would impose on serving. Pairs with inference-kernel-profile (nsys: absolute kernel duration + cuda-graph timeline) and zymtrace-anchored-query (sample- share + kernel-name resolution). Triggers on "ncu", "nsight compute", "kernel internals", "occupancy", "register pressure", "shared memory", "DRAM bandwidth", "roofline", "arithmetic intensity", "warp stalls", "branch divergence", or any combination of "ncu / nsight-compute / occupancy / regs / smem / dram / roofline" with "vllm / kimi / glm / deepseek / inference".

Capture per-kernel CUDA profile data from a live vLLM inference pod via an nsys debug sidecar (no production image rebuild). Outputs .nsys-rep + summary CSV + top-kernels table that joins with the zymtrace per-kernel breakdown and the inference-perf-bench bundle's inferenceperfbenchv1.json. Triggers on "kernel profile", "nsys profile", "what's in the native bucket", "per-kernel breakdown", "ncu", "Nsight Systems", "kernel-level analysis", or any combination of "profile / capture / nsys / ncu" with "vllm / kimi / glm / deepseek".

White-box debug a custom CUDA/CUTLASS kernel producing a WRONG numeric result (over-amplification, NaN, coherence) after black-box bisection is EXHAUSTED: every external lever varied (grid, pipeline depth, layout, dtype), the result deterministic + lever-independent, yet the structure reads correct. Two tracks: (A) an in-kernel operand + accumulator trace (if constexpr-gated to survive megakernel pipelining), (B) a standalone reproducer .cu instantiating the EXACT kernel template + TMA descriptors with controlled inputs (all-ones then real-dump) vs a host GEMM. Localizes the defect to operand-load vs MMA vs epilogue vs descriptor. Escalation tier above inference-kernel-ncu-profile when H/P are fine but the output is numerically wrong. Triggers on "white-box kernel", "kernel over-amplifies / NaN", "standalone reproducer", "in-kernel trace", "operand dump", "cutlass UMMA bug", or any combination of "white-box / reproducer / in-kernel / operand-trace" with "kernel / cutlass / UMMA / megakernel / vllm".

Capture + enforce per-model KNOWN-GOOD serving configs: the REQUIRED serve flags (boot-blockers / crash-at-high-c / deploy-correctness workarounds) plus the champion + the bug each flag avoids, in one queryable registry (perf-tune-report/configs/known-good-configs.yaml) so a hard-won workaround (e.g. Qwen3-Next's gdnprefillbackend=triton on vLLM 0.22) is NEVER re-discovered the hard way. These are NOT well-known upstream defaults -- several are field-discovered workarounds where the upstream auto path is actively broken on the target hardware. record after a champion is found, check (fail-closed) before any deploy/ship. Keep the registry private -- never post an entry publicly. Triggers on "known-good config", "required flags for ", "did we capture the flag combo", "register the config", "check the deploy config", "config drift", "what flags does need", or any combination of "known-good / required / champion config" with "record / check / register / capture / drift / flags / serve".

Drive lm-eval-harness quality evals (GPQA, MMLU-Pro) inside model pods plus optional ExternalEval (externally operated). Use to validate model quality before promoting to staging or prod, after vLLM / quantization / KV-cache changes, or to compare against published HuggingFace / paper baselines. Pair with inference-perf-bench (the perf-side counterpart) for full pre-promotion validation. Triggers on "lm-eval-harness", "GPQA", "MMLU-Pro", "ExternalEval", "model quality eval", "/run-model-eval", "run evals", "run gpqa", "run mmlu", "run external-eval", "run evals on the model", or any combination of "eval / quality / accuracy" with "inference / model / vllm".

End-to-end orchestrator that takes a NEW model from a bare HuggingFace id to a perf-lake-published, validated CROSS-ENGINE (vLLM + SGLang) champion on B200/GB300. Scaffolds a per-model harness + evidence bundle (run-id == experiment-id), then drives gate-driven phases: deploy baseline -> 4-layer profile (zymtrace L1 / DCGM L3 / ncu L4) -> tune CROSS-ENGINE via the variant A/B (vLLM AND SGLang arms) -> quantize -> validate -> spec-decode train+validate -> bench the multi-workload suite -> championselect (baseline vs top-X, the obvious production pick) -> publishtolake. Pauses at every red gate. Every number defaults to DRAFT (a champion VERDICT needs the multi-workload + accuracy gates + L3 byte-grounding). Triggers on "optimize a new model", "bring up a model end-to-end", "model bring-up pipeline", "find the best perf for ", "run the full optimization pipeline", or any combination of "optimize / bring-up / end-to-end / pipeline / champion" with "new model / inference / vllm / sglang".

Bridge between the inference-perf-bench / ai-bench output bundle and the profile-and-optimize perf-baseline registry. Knows the canonical inference perf metric shape (TTFT p50/p95/p99, ITL p50, throughput, tok/s/user, request latency p50, prefix cache hit rate delta, GPU cache utilization peak), records it under inferenceperfbenchv1 schema, and on diff applies per-metric tolerances tuned for inference workloads (TTFT regressions <5 percent, throughput <3 percent, cache hit rate <2 absolute points). Wraps the existing perf-baseline-record and perf-baseline-diff MCP verbs, no new MCP verbs introduced. Triggers on "register inference baseline", "inference perf-baseline", "diff inference perf", "is this inference regression real", "perf-bench baseline", "ai-bench baseline", or any combination of "register / record / diff / compare" with "inference / ai-bench / perf-bench / vllm / kimi / glm / deepseek / minimax".

Canonical inference perf-bench skill (formal name. The colloquial alias is ai-bench - identical behaviour). Drives NVIDIA AIPerf + the replay-playback dataset against an in-cluster vLLM endpoint to measure TTFT, ITL, throughput, tok/s/user, request latency, and prefix cache hit rate. Iterative 9-phase workflow. Use when promoting a model to staging/prod, after vllmArgs / vLLM / KV-cache-dtype changes, or for A/B comparison across configs. Triggers on "perf-bench", "AIPerf", "Replay Playback", "throughput sweep", "TTFT P95", "concurrency sweep on inference", "/run-perf-bench", "benchmark throughput", "benchmark latency", "run aiperf", "performance test", or any combination of "perf / latency / throughput / tps / TTFT / ITL" with "inference / vllm / serverless / bench".

Fuse the four profilers -- nsys (timeline + kernel durations), ncu (roofline / occupancy / arithmetic intensity), zymtrace (CPU+GPU flamegraph), and DCGM (byte-grounded %SoL) -- into ONE ranked, data-backed recommendation ledger for a vLLM deployment. Every recommendation (including concrete vLLM source/config changes) cites its backing artifact + solrigor tier, and each is staged as an A/B experiment so it is proven, not asserted. Emits a findings doc + perftunereport view + publishes to the perf-lake. The holistic-analysis phase of inference-model-optimize (Phase 2.5 + 8.5), usable standalone. Triggers on "synthesize the profiles", "what should I change to make this faster", "holistic perf analysis", "fuse nsys ncu zymtrace dcgm", "ranked perf recommendations", "suggest vllm changes from profiles", "explain the bottleneck with data", or any combination of "synthesize / fuse / recommend / holistic / explain" with "profile / nsys / ncu / zymtrace / dcgm / roofline / bottleneck / vllm".

Build a multi-page benchmark report PDF (scatter panels + per-concurrency heatmap tables) from vllm bench sweep serveworkload and/or AIPerf outputs, in the GLM-5.1 reference layout (5x2 scatter facet by maxnumbatchedtokens + 3x2 heatmap tables for C={8,16,32} x {tok/s/GPU, TTFT}. First-class failed/partial cells). Also renders the cross-engine (vLLM + SGLang) CHAMPION page 8: baseline vs top-X variants + the overlaid roofline + a RECOMMENDED-FOR-PRODUCTION pick via championselect. Backed by the perftunereport CLI library. Triggers on "build a perf report", "render the atlas PDF", "perf-report campaign", "glm5p1-style report", "pareto report", "perftunereport smoke", "champion selection", "which variant to ship", "production champion", "baseline vs top variants", or any combination of "perf / atlas / report / champion" with "inference / vllm / sglang / aiperf / b200 / gb300 / nvfp4 / fp8".

Produce quantized inference weights from a BF16/FP8 base checkpoint via a post-training-quantization (PTQ) pipeline -- instead of only ever pulling NVFP4 weights pre-quantized. A calibration prep Job then a quantize Job: NVIDIA ModelOpt -> NVFP4 (default), or llm-compressor -> FP8 (alt). Writes weights + per-layer scales to an experiment-prefixed PVC, gated on inference-model-eval accuracy + a perf A/B. When a vendor nvidia/-NVFP4 exists, runs a 3-way comparison (base vs ours vs NVIDIA) on accuracy + perf + ncu. Ack-gated. The calibrate+quantize phase of inference-model-optimize. Triggers on "quantize a model", "calibrate NVFP4", "run modelopt PTQ", "produce NVFP4 weights", "llm-compressor FP8", "PTQ pipeline", "compare our PTQ to NVIDIA NVFP4", "our nvfp4 vs nvidia nvfp4", "calibration dataset for quantization", "quantize to NVFP4/FP8", or any combination of "quantize / calibrate / PTQ / modelopt / llm-compressor" with "nvfp4 / fp8 / weights / model / inference".

Closed-loop speculative-decoding-as-a-service orchestrator -- a self-hosted analog of Fireworks FireOptimizer's adaptive speculative execution. Composes the per-phase skills into one profile-matched loop: profile live traffic (inference-workload-profile) -> hit-rate-matched corpus -> train a draft (inference-spec-decode-train) -> in-engine acceptance + same-node TPOT A/B vs the standing config -> promote ONLY on a measured win (standing config stays otherwise) -> publish. It is Phase 6 of inference-model-optimize run as a profile-matched closed loop instead of one-shot. Defines a service-state.json contract + a --mode oneshot|controller flag so it can later graduate to a deployed in-cluster controller without a rewrite. Triggers on "spec-dec as a service", "adaptive speculative decoding service", "fireoptimizer equivalent", "auto-train a draft", "close the spec-decode loop", "stand up the spec-dec service", or any combination of "service / loop / adaptive" with "spec-decode / speculative / draft".

Train + validate a speculative-decoding draft head (EAGLE3 or DFlash) for an ARBITRARY target LLM, generalizing the GLM-5.1-only SpecForge recipe so any new model gets one. Parameterizes what that offline recipe hard-codes: target path, aux-hidden-states layers (derived [1, L/2-1, L-4] from config.json numlayers), chat template, draft-head config, and method. Wires the SpecForge preparehiddenstates -> traineagle3 / traindflash -> convert-to-vLLM flow on Slurm, then gates on a measured acceptance-length A/B (dflashvllmeval.py + vLLM specdecode counters) vs the standing config. This is the train-spec-decode phase of inference-model-optimize, usable standalone. Triggers on "train an EAGLE3 draft", "train a DFlash head", "build a speculative decoder for ", "spec-decode draft training", "draft model acceptance", "SpecForge for ", "generalize eagle3-train", or any combination of "train / build / validate" with "eagle3 / dflash / draft / speculative / spec-decode".

Tune a speculative-decoding DRAFT head's TRAINING hyperparameters (global batch, learning rate, accumulation, warmup) for a target LLM, optimizing the TRUE serving in-engine acceptance length (vLLM specdecode counters) with a cheap training-acc proxy for triage. Reuses the search ALGORITHMS (hyperband/grid/random natively. Bayesian-TPE via optuna if installed) wired to a GB300/managed K8s pod launcher + an offline EAGLE3/DFlash trainer (distinct from the MLPerf aituning contract, which does not map to draft training). The draft-training analog of inference-tune-sweep (which tunes vLLM SERVING config). Triggers on "tune the draft head", "tune eagle3 / dflash training", "bayesian/hyperband tune the draft", "search global batch and LR for the speculator", "spec-decode hyperparameter sweep", "draft-training tuner", or any combination of "tune / sweep / optimize / search / bayesian / hyperband" with "eagle3 / dflash / draft / speculator / spec-decode".

Search a vLLM serving model's config space for its best inference performance: a perftunereport campaignrun matrix sweep over concurrency x maxnumbatchedtokens x kv-cache-dtype x cudagraphmode, plus a same-node controlled A/B (run-controlled-ab.sh) for any decode-latency claim, gated by perf-baseline-diff. Picks a champion config that beats the baseline on the run's focus metric (throughput tok/s OR median TPOT/ITL), honoring warm/cold + eager/cudagraph methodology rules. This is the tuning phase of inference-model-optimize, usable standalone. Not a Bayesian optimizer -- it sweeps an operator-named grid (an --optimizer hook is stubbed). Triggers on "tune the vllm config", "sweep maxnumbatchedtokens", "find the best serving config", "vllmArgs sweep", "cudagraphmode A/B", "kv-cache-dtype sweep", "tune concurrency", "config search for ", or any combination of "tune / sweep / optimize / search / A-B" with "vllm / config / vllmArgs / concurrency / batched-tokens / cudagraph".

Render the leadership value-prop ledger for the inference effort: the deployable wins vs the FlashInfer-TRTLLM + best-tuned-vLLM 0.21/0.22 baseline, grouped DONE / IN-PROGRESS / NOT-DONE / CLOSED-NEGATIVE, each row data-backed by a perf-lake campaign (live solrigor + verdict tier), plus the ranked GRIND FRONTIER of next levers (the always-be-grinding performance ratchet). Joins the curated perf-tune-report/configs/value-findings.yaml registry with the live campaigns via perftunereport valueview. Flags any finding whose backing campaign is missing or ungrounded. Use when you need to show value / report to leadership / answer "what have we uncovered that we can deploy, revalidate, or pursue". Triggers on "value ledger", "value prop", "show value", "show the value prop", "leadership view", "what wins do we have", "vs flashinfer", "value-findings", "what can we deploy", or any combination of "value / wins / findings / leadership / report" with "inference / vllm / flashinfer / perf".