Open-source · Local-first

From failed runs to verified fix

WatcherML is the reliability and recovery layer for ML training. It captures evidence, diagnoses failures, proposes bounded interventions, reruns controlled trials, and proves which change actually worked.

See the recovery loop View source

Deterministic capsuleVersioned failure evidence, no LLM required

Bounded executionHard trial, timeout, and GPU budgets

Verifier-owned truthFull trials plus independent confirmations

CAMPAIGN

Recover & verify

validation loss 0.421

SourceCUDA OOM

Trials4 / 6

Confirmations2 required

StatusVerified

Campaign evidence RECORDED

Sealed source capsule and contract.

Probed batch 16 in a fresh process.

Confirmed the candidate twice.

Validation losswithin +0.030

probebatch 32 → 1630 stepssurvived

fullbatch 16 + accum 2completedprovisional

confirmsame sealed config2 / 2 passedverified

↳

Verified recovery2 confirmations passed

Official PyPI install

01 / 04

Core SDK

Install deterministic OOM forensics.

Get the recorder, CLI, failure capsules, bounded recovery policy, and isolated trial runner—without installing the notebook extension or web UI.

local-first SDK + CLI Python 3.10+

terminal

pip install watcherml

Choose your surface

Add notebooks, the local UI, or both.

Keep the core package lean. Add notebook support for Jupyter and Colab, or install the optional local web interface for desktop workflows.

Jupyter + Colab optional local UI CLI always available

optional extras

# Jupyter, IPython, or Colab
pip install "watcherml[notebook]"

# Local web UI
pip install "watcherml[ui]"

# Install both
pip install "watcherml[notebook,ui]"

Python SDK

Wrap the run. Preserve the evidence.

Record configuration, metrics, environment, dataset identity, and runtime telemetry. If training fails, WatcherML freezes the evidence into a deterministic failure capsule.

small API runtime evidence OOM capsule

train.py

import watcherml as watcher

with watcher.init(
    project="mistral-lora",
    config=config,
) as run:
    train(
        model,
        dataset,
        run=run,
    )

Controlled OOM recovery

Rerun, intervene, and verify.

Reproduce the original OOM as a control, evaluate bounded interventions in fresh processes, and require confirmation before any recovery is marked verified.

init runs inspect failures compare export recover recoveries recovery ui

watcher CLI

# Inspect the captured OOM
watcher inspect RUN_ID

# Seal constraints before compute
watcher prepare-recovery RUN_ID \
  --entrypoint train:train \
  --metric validation_loss:minimize:0.42:0.03 \
  --minimum-progress-steps 1000 \
  --confirmation-runs 2 \
  --out recovery-plan.json

# Review, then run bounded trials
watcher recover --plan recovery-plan.json

# Inspect the complete campaign
watcher recovery CAMPAIGN_ID

WHY WATCHERML EXISTS

The crash is one line.
The cost is everything before it.

CUDA Out of Memory (OOM) errors are arguably the most common bottleneck in AI dev. They can erase hours, or even days, of GPU training.

CUDA OOM HORROR STORIES

01 / 03

train_llm.py FAILED AFTER 71H 48M

step 48,996 / 50,000 running final validation... RuntimeError: CUDA out of memory. Tried to allocate 2.00 GiB. GPU 0 has 312 MiB free.

THE 99% EPOCH CRASH

Many days of training. One validation batch ends it.

The training loop fits, but validation uses a larger batch or longer sequence. VRAM crosses its limit near the finish and the entire run terminates.

Failure appears only in a later training phase
The last successful state may not explain the spike
Rerunning the full job to test one change is expensive

finetune.py OOM WITH BATCH SIZE 1

model = "llm-7b" batch_size = 1 max_length = 4096 gradient_checkpointing = true torch.cuda.OutOfMemoryError

THE LONG-CONTEXT TRAP

Even the smaller model still runs out of memory.

Batch size is already one. Mixed precision and gradient checkpointing are enabled. But sequence length, activations, optimizer state, and temporary allocations still exceed VRAM.

Several memory variables interact at once
The obvious intervention may already be exhausted
A successful launch does not guarantee stable headroom

recovery_notes.txt ATTEMPT 07

✓ lowered batch size ✓ enabled mixed precision ✓ shortened sequences ✓ emptied CUDA cache Result: CUDA OOM still reproduced

CONFIGURATION ROULETTE

Risky YOLO ML runs waste your money.

Without a preserved failure record and a controlled recovery contract, engineers test loosely related fixes without knowing whether the original failure was reproduced or truly resolved.

Trials are difficult to compare reliably
Multiple changes obscure which intervention mattered
An incomplete test run can be mistaken for a verified recovery

WatcherML turns the crash into a bounded recovery protocol. It captures a versioned failure capsule, seals constraints before compute, evaluates typed interventions in fresh processes, and lets independent confirmation—not a promising trial—decide recovery.

See the recovery loop →

MASSIVELY IMPROVE YOUR ML WORKFLOWS

Stop paying full price.
Save your GPU-hours.

WatcherML assists ML engineers when training is expensive, failures recur, multiple engineers are involved, and successful retries must be trusted later. Trackers show you what failed. WatcherML investigates why.

01 PASSING IS NOT PROVING

“Batch size 16 worked” leaves the important questions unanswered.

Manual recovery can make one run complete. It rarely leaves a sealed record of what failed, what changed, and whether repeated runs satisfied constraints declared before execution.

training evaluation fragmentation tensor growth optimizer state GPU contention

MANUAL RETRY “Batch size 16 worked.”

WATCHERML RECORD

Original failure and training phase
Captured configuration and environment
Allowlisted intervention
Probe and full-trial results
Quality-regression check
Independent confirmation status

02 CHANGE THE GPU ECONOMICS

Reject obvious failures before paying for another complete job.

If a full attempt takes eight GPU-hours, three manual retries can consume up to 24 GPU-hours. WatcherML evaluates candidates with bounded probes before promoting a survivor.

A probe cannot guarantee that a late-stage OOM is gone—but it can cheaply eliminate candidates that fail immediately.

MANUAL RETRIES

24 potential GPU-hours

3 complete attempts × 8 hours

→

BOUNDED CAMPAIGN

A OOM at step 6 eliminated

B OOM at step 18 eliminated

C survived 30 steps promote to full trial

03 PRESERVE WORKLOAD MEANING

Errors can disappear because you accidentally changed the experiment.

Shortening sequences, dropping examples, disabling validation, reducing training steps, or changing the dataset may make a job pass while invalidating the intended workload.

recovery-contract.yaml INVARIANTS ACTIVE

Dataset fingerprint must match preserved

Effective batch 32 preserved

Validation required executed

Maximum loss regression +0.030 +0.006

Sequence truncation not allowed blocked

04 TRUST THE COMPARISON

Every trial starts in a fresh process, not a haunted notebook.

Notebook reruns can inherit cached CUDA allocations, old model objects, modified globals, optimizer state, imported modules, and partially executed cells.

WatcherML rebuilds each candidate in an isolated worker and releases the child CUDA context when the worker exits.

NOTEBOOK RETRY

! Cached allocations

! Old optimizer state

! Modified globals

! Partial cell history

WATCHERML WORKER

✓ Fresh interpreter

✓ Rebuilt model

✓ Declared configuration

✓ Independent run record

05 COMPOUND ACROSS THE TEAM

The twentieth engineer should not rediscover the first engineer’s fix.

Integration is paid once. Every incident then contributes reusable evidence, contracts, entrypoints, policies, and comparable intervention outcomes.

Capture → Propose → Probe → Eliminate → Full trial → Report

Failure capsules Standard incident evidence

Recovery contracts Shared safety policy

Training entrypoints Reusable workload contract

Verified recoveries Reusable institutional knowledge

The campaign handles bounded work unattended. The engineer returns to a structured report instead of another crashed run.

06 USE ONLY WHAT YOU NEED

Start with evidence. Add GPU recovery only when the incident warrants it.

Capturing a failure should require almost no setup. Entrypoints, contracts, probes, full trials, and confirmation are needed only when the user chooses deeper recovery.

When manual recovery is better

For a five-minute, one-off experiment with an obvious cause, changing the batch size manually is probably faster.

CaptureDeterministic OOM capsule

Almost zero cost

PlanEvidence-backed interventions

No additional GPU run

ProbeEliminate early failures

Short GPU runs

Full trialTest complete recovery

Provisional success

ConfirmRepeat independently

Verified recovery

Successful trial — not independently confirmed

THE BOTTOM LINE
Manual retries can make one ML run pass. WatcherML finds what failed, proposes recovery steps, and executes verified trials.

Designed to work with the stack you already use

PyTorchJupyterNVIDIAMLflowWeights & Biases

V1 PROOF VERTICAL · CUDA OOM RECOVERY

Turn one CUDA OOM into a controlled recovery experiment

WatcherML is a local-first reliability layer for ML training. V1 starts with tackling OOM errors that can be captured, changed, rerun, and verified without guesswork.

Capture the evidence that actually existed at failure.

Persist a versioned capsule with the traceback, configuration, last logged step, recent metrics, sampled resources, environment fingerprint, Git state, dataset fingerprint, and optional CUDA allocator bytes.

schema 1.0tracebackallocator statecapture score

capsule.schema.version = "1.0"
capsule.failure.class = "cuda_out_of_memory"
capsule.evidence.training_state.last_logged_step = 417
capsule.capture = { score: 8, maximum: 10 }

LOCAL-FIRST SDK + CLI

Capture locally. Recover through Python or the CLI.

The recorder works in scripts, Jupyter, and Google Colab. Recovery uses an importable training entrypoint so every probe, full trial, and confirmation can start in a fresh supervised process.

Available through pipNo hosted service, web UI, LLM, or container runtime is required.

import watcherml as watcher

with watcher.init(
    project="mistral-lora",
    config=config,
) as run:
    for step, batch in enumerate(loader):
        loss = train_step(model, batch)
        run.log_metric("loss", float(loss), step=step)

# On failure, the original exception still propagates.
# WatcherML has already persisted the deterministic capsule.

pip install watcherml

# Local terminal—or prefix commands with ! in Colab
watcher init
watcher runs --project mistral-lora
watcher failures --project mistral-lora
watcher inspect RUN_ID
watcher compare FAILED_RUN SUCCESSFUL_RUN
watcher export RUN_ID --out failure-capsule.zip

# Optional local interface; not required in Colab
pip install 'watcherml[ui]'
watcher ui --port 7331

CONTROLLED BY CONSTRUCTION

A recovery trial is evidence, not a guess.

WatcherML v1 cannot rewrite training code, install dependencies, alter datasets, or silently keep searching. It materializes a typed, contract-approved intervention in a fresh process and records exactly what happened.

See the recovery contract →

01Fresh processesEach trial receives a clean interpreter and an independent evidence record.

02Source run preservedThe failed run and its capsule are immutable campaign inputs.

03Allowlisted patchesV1 changes declared configuration keys—not code, packages, or data.

04Hard stop conditionsTrial count, timeout, and wall-clock limits end the campaign predictably.

05Deterministic verdictsProgress, resource, metric, and confirmation checks decide recovery.

BUILT IN PUBLIC

A real MLOps system, tested on Nvidia GPUs.

CaptureVersioned deterministic evidence

BoundTrial, timeout, and GPU budgets

IsolateFresh subprocess per attempt

VerifyIndependent confirmation owns the verdict

Open-source · installable with pip

Help build the experiment engineer your GPU deserves.

Have new ideas and want to contribute? Open a new pull request on Github!

Install WatcherML View source