GPU RMA Diagnostic Toolkit — User Guide

Version 1.2 · August 2026 · Primitive

1. What it is and why it exists

The RMA toolkit turns a suspected GPU problem into a defensible decision. Given a machine on the Primitive platform, one dispatched job runs a tiered diagnostic cascade, computes a deterministic verdict (from "healthy" to "submit an RMA"), and produces an evidence package strong enough to hand to NVIDIA or a hardware vendor — with every claim carrying provenance for how it was obtained and how much it proves.

It exists because GPU fleet triage is usually ad-hoc: an engineer SSHes in, runs nvidia-smi and some dcgmi commands, forms an opinion, and the evidence evaporates. The toolkit makes that process repeatable, honest, and platform-native — the same fault produces the same verdict and the same package, every run.

Design principles

• Evidence before power: no power action ever happens before volatile evidence (kernel logs, Xids) is sealed. RMA boxes carry evidence that a reboot can destroy.
• Verify outcomes, not return codes: power cycles are confirmed by an observed boot identity change.
• Results that state their own confidence: the toolkit never overstates what it found. Every result says how strong its evidence is, what was and wasn't tested, and exactly which follow-up run would strengthen it. A conclusion you can't fully trust is labeled that way, rather than presented as certain. (Section 6 covers how to read these labels.)
• Clean-run contract: on customer hardware, the toolkit collects all evidence locally, then removes its workspace and run outputs when finished. Returning a cloud box to its provider image, or restoring a lab machine to a known-good configuration, is done with re-flashing & re-provisioning via the Primitive platform.

2. Field-proven (both fault classes, real hardware)

3. How it works

A diagnostic run walks an escalating cascade — easy and passive first, invasive last:

Verdict bands

4. Getting a machine ready (provisioning)

Prerequisite: the machine is registered on Primitive as a Hardware node in your organization.

Then dispatch the RMA Toolkit Provisioning diagnostics job from the UI or Primitive MCP server.

The job is idempotent and reboot-durable: it installs/aligns the NVIDIA driver stack, fabricmanager, DCGM, deploys the toolkit workspace, deploys the correct nvFieldDiag bundle, and validates the box is diagnostic-ready.

5. Running a diagnostic

Primary path: from the UI

Select a Hardware's action dropdown → Run Diagnosis (or manually via Start Job, select RMA Diagnostic in the dropdown). Inputs:

Secondary path: from Claude Code or Codex

Install the primitive-mcp server (https://mcp.primitive.tech/mcp) and ask the agent to run diagnosis: "Run diagnosis on X server".

What to expect: a run progresses through the passive and stress tiers for the first ~30–45 minutes. Then a power cycle as the next diagnostic step. The run picks up where it left off automatically. With a deep field diagnostic selected, the full run can take several hours; progress streams live on the run's Logs page throughout, and it's safe to walk away.

Choosing a cycle method

• auto — recommended default. Uses the box's proven method (cached on its power card) or discovers one.
• ipmi-inband — true DC cycle via the box's own BMC interface; the strongest in-band evidence.
• warm-reboot — OS reboot. For machines with no BMC/OOB path (typical cloud/CSP boxes). The findings will carry an explicit degraded-cycle caveat and recommend an AC-cycle re-run — the run still completes and the package is still produced.
• oob-redfish — out-of-band via BMC Redfish, when the platform has BMC credentials/host configured.

6. Reading the results

Where results land

Hardware page → Reports: the platform report generated from the run.

The artifacts

Trust labels — read these first

• package_status: COMPLETE = real per-GPU fielddiag evidence exists, or an evidence-bearing impossibility (a GPU that cannot even attach — that IS the evidence).
• package_status: PROVISIONAL = a gap with a reason code: no_diag_results (enumeration-only mode), bundle_missing, operator_skip, power_cycle_unavailable. The label tells you the exact re-run that would upgrade it.
• Power provenance: power_cycle_source (method used), power_cycle_verified (was a reboot actually observed), power_cycle_power_class (ac / dc / warm). Warm and DC cycles cannot prove everything an AC drain can — the findings say so explicitly rather than overclaiming.
• kernel_log_source: which log source, at which privilege, produced the kernel evidence.

Platform reports

Every completed diagnostic run generates a platform Report citing the toolkit's tier-4 verdict and package label. Reports can also be generated from telemetry alone (time-window mode, no run) and will themselves tell you what evidence they lack; a toolkit run is how you supply it. The toolkit's findings.json is authoritative.

7. From verdict to RMA

• RMA_REQUIRED → the tarball is the submission package: board serial, per-GPU identifiers (BDF/UUID), fielddiag results or no-attach evidence, journals across boots, OOB dumps, sha256 manifest.
• CHECK_MECHANICALS → service action (reseat / cold plate / TIM) with the thermal evidence chain; re-run after service to confirm.
• PROVISIONAL packages are submittable when the evidence is overwhelming — but the label and reason go with it. Prefer upgrading via the indicated re-run.

8. Current limitations (honest list)

• Boxes without a BMC path get warm-cycle evidence with an explicit caveat — AC-drain conclusions require an AC-capable path (smart PDU / provider API when available).
• level1 fielddiag is mainly investigative (~1–2 hours per run) — pick level2/gpufielddiag when a full run is needed (10–11 hours).