Open-source · Local-first

From failed run to verified fix

WatcherML is the reliability and recovery layer for ML training. It captures evidence, diagnoses failures, proposes bounded interventions, reruns controlled trials, and proves which change actually worked.

See the recovery loop View source

Deterministic-firstDiagnosis works without an LLM

LLM optionalLocal or cloud via any-llm

Plays well with othersStandalone or export to MLflow / W&B

AUTOPILOT CAMPAIGN

Recover & optimize

objective 0.928

Best trial#05

Peak VRAM13.9 GB

GPU budget2h 46m

StatusRecovered

Recovery reasoning LIVE

Observed CUDA OOM at step 417.

Compared with nearest successful run.

Testing bf16 + gradient checkpointing.

Trial objective↑ 4.1%

#03batch 4 → 2OOM resolvedkeep

#04bf16 enabled+18% throughputaccept

#05lr 3e-4 → 1.8e-4F1 0.928best

↳

Verified fixOOM signature resolved

✦

Ollama localNo run data leaves Buffy

Official PyPI install

01 / 04

Core recorder

Install the full local foundation.

Get the SDK, CLI, failure capsules, and recovery —without the notebook extension or web UI.

local-firstSDK + CLIno web UI

terminal

pip install watcherml

Choose your surface

Add notebooks, the local UI, or both.

Keep the core lean, then install only the integrations that match how you work.

%load_ext watchermlwatcher uioptional extras

extras

# Jupyter / IPython
pip install watcherml[notebook]

# Local web UI
pip install watcherml[ui]

# Everything
pip install watcherml[notebook,ui]

Python SDK

Wrap the training loop. Preserve the evidence.

Record configuration and run context with a small API, then let WatcherML freeze forensic evidence when execution fails.

context managerrun metadatafailure capsule

train.py

import watcherml as watcher

with watcher.run(
    project="mistral-lora",
    config=config,
) as run:
    train(model, dataset, run=run)

Forensics from the terminal

Inspect, compare, explain, and export.

Move from a failed run to an auditable diagnosis without leaving your local development workflow.

initrunsinspectfailurescompareadviseexportrecoveriesrecoveryui

watcher CLI

watcher init
watcher runs --project mistral-lora
watcher inspect RUN_ID --advise
watcher compare RUN_A RUN_B --advise
watcher export RUN_ID --format capsule
watcher ui --port 7331

Designed to work with the stack you already use

PyTorchJupyterOllamaNVIDIAMLflowWeights & BiasesOptuna

THE MISSING LAYER

Trackers show you what happened.
WatcherML investigates why.

Metrics dashboards are useful after a clean run. WatcherML is built for the messy reality: CUDA OOMs, NaNs, stalled data loaders, poor GPU utilization, regressions, and experiments that need more than another chart.

Traditional tracker

“Run failed at step 417.”

statusfailed

exceptionCUDA out of memory

peak VRAM15.88 GB

loss—

WatcherML forensic report

“Here is the cause, evidence, and verified recovery.”

Cause ranked
Sequence length increased 33%; activation memory exceeded available VRAM.

Baseline selected
Nearest successful run matched model, dataset, hardware, and Git ancestry.

Fix verified
Trial completed with 1.9 GB headroom and improved throughput.

FIRST RECOVERY VERTICAL

A scientific loop for every CUDA OOM.

OOM is the first proof point, but not the entire identity of WatcherML. It is common, reproducible, measurable, and constrained enough to build a recovery engine that users can trust.

Freeze the state around failure.

Capture stack frames, tensor summaries, allocator state, recent metrics, Git changes, environment drift, and the last successful step.

CUDA statetensor shapesGit diffstdout

capsule = watcher.capture(run)
capsule.failure = "cuda_oom"
capsule.peak_vram = 15.88
capsule.last_step = 417
capsule.freeze()

CAPABILITIES

Everything needed to go from failure to evidence-backed progress.

A focused layer for PyTorch forensics and controlled experimentation—not another general-purpose dashboard.

⌁

FORENSIC CAPSULES

Capture the moment a run goes wrong.

Preserve the execution context around crashes, numerical instability, stalls, and regressions without collecting raw training data by default.

Failure classNaN loss

First invalid layerencoder.9.mlp

Gradient norm∞

⌕

NEAREST-SUCCESS DIFF

Compare against the right baseline automatically.

Find the closest successful run by model, data fingerprint, hardware, configuration, and Git ancestry—then rank the differences most likely to matter.

✦

OPTIONAL LLM INTEGRATION

Reason over private evidence locally.

Use structured any-llm tool calls to produce hypotheses and experiment plans while keeping stack traces, metrics, and source changes on your machine.

↻

BOUNDED RECOVERY CAMPAIGNS

Run controlled variants—not infinite guesses.

Enforce trial, time, GPU, file-change, and parameter budgets. Require approval before code, dependency, or dataset changes.

✓

DETERMINISTIC EVALUATION

Let code decide whether a trial worked.

Judge every intervention against explicit metrics and constraints. The LLM proposes; the evaluator verifies.

⎇

RESOLUTION MEMORY

Remember fixes that survived reality.

Build a graph of failure signatures, evidence, interventions, and measured outcomes so future campaigns start from verified experience.

DEFINE SUCCESS BEFORE RECOVERY LOOP

Every campaign runs against a strict contract.

Set a primary metric, operational constraints, permissions, and a hard budget. WatcherML stops when the goal is met, the budget is exhausted, or human approval is required.

Explicit metric and acceptance threshold
Maximum trials, GPU-hours, and wall time
Allowlisted configuration and code changes
Automatic rollback and complete audit trail

campaign.yaml● valid

objective:
  metric: validation_f1
  direction: maximize

acceptance:
  validation_f1: 0.92
  peak_vram_gb: 14.5

budget:
  max_trials: 8
  max_gpu_hours: 5

permissions:
  config: automatic
  code: approval_required
  dataset: forbidden

FOCUSED BY DESIGN

A recovery layer—not another platform rewrite.

Use WatcherML by itself for local experiments, or connect it to the tracking system your team already trusts.

PyTorch trainingJupyter notebooksMLflow / W&B

runtime events ↓

WatcherMLlocal reliability runtime

Recorderruns · metrics · environment

Forensicscapsules · evidence · diffs

Policy engineactions · budgets · approvals

Trial runnerfresh isolated processes

Verifierruntime · metrics · constraints

Resolution memory verified outcomes only

any-llmoptional planning and explanationMLflow sink optional campaign exportW&B sinkoptional campaign export

DESIGNED FOR THE TRAINING LOOP

Start with a small API. Unlock the full forensic loop.

WatcherML is your recovery layer powered even better with any LLM of your choice.

Available through pipInstall the lean core, then add notebook and local UI extras only when needed.

import watcherml as watcher

with watcher.run(
    project="mistral-lora",
    config=config,
    export_to="mlflow",
) as run:
    train(model, dataset, run=run)

# On failure: evidence is frozen automatically.
# Start a bounded recovery campaign.
watcher.autopilot(
    run_id=run.id,
    contract="campaign.yaml",
    agent="ollama/qwen3",
)

# Initialize local metadata and artifacts
watcher init

# Find and investigate a failed run
watcher runs --project mistral-lora
watcher inspect run_7f21 --advise

# Compare evidence and export a reproduction capsule
watcher compare run_7f21 run_8a03 --advise
watcher export run_7f21 --format capsule --out recovery.zip

# Open the local interface
watcher ui --port 7331

SAFE BY CONSTRUCTION

WatcherML never gets a blank check.

Autonomy without guardrails is just expensive uncertainty. WatcherML constrains what can change, where code can run, and when a human must approve the next step.

Read the design principles →

01No open shellStructured, allowlisted actions instead of arbitrary commands.

02Hard budgetsLimits for GPU-hours, wall time, trials, disk, and memory.

03Human gatesApproval before code, dependency, or dataset changes.

04Private defaultsLocal inference and redacted forensic evidence.

05Deterministic verdictsMetrics and tests—not the LLM—decide success.

BUILT IN PUBLIC

A real MLOps system, tested on Nvidia GPUs.

OpenPython package

LocalDeterministic Recovery

AuditableEvery intervention recorded

VerifiedFixes proven through controlled reruns

Open-source · installable with pip

Help build the experiment engineer your GPU deserves.

Have new ideas and want to contribute? Open a new pull request on Github!

Follow on GitHub Meet the builder