Local model training
Reviewing corrections and comparing local model updates.
promote_or_rollback = gated
These notes describe the implementation without client-specific details.
Problem
Corrections need review before they become training material.
A correction can be useful, incomplete, or wrong. This tool collects corrections as candidate lessons, screens them before training, and compares the resulting model with the current version. Promotion includes a review step, with versioning to support rollback.
How it works
Capture corrections
Corrections are stored as structured candidate lessons for review before use.
Screen candidates
An immunity gate checks candidate lessons before they enter training.
Compare versions
Faithfulness and competence scorecards compare a candidate with the current model.
Review promotion
Evaluation results and human approval determine whether a candidate is promoted.
Tradeoffs
Iteration & evaluation
Screening and evaluation add time between a correction and a model update.
Automation & review
The workflow automates training and comparison steps, with a person reviewing promotion.
Updates & rollback
Keeping previous versions available adds versioning work but supports reverting a change.
Checks
- Immunity gate
- Screen candidate lessons before training
- Head-to-head evaluation
- Compare candidate and current model results
- Promotion gate
- Review results before using an update
- Versioning and rollback
- Keep a path back to a previous model
My contribution
My work covers the correction and evaluation workflow.
- candidate capture
- screening + scoring
- evaluation harness
- promotion gates
- versioning + rollback
This includes the promotion criteria and the tooling used to compare and manage model versions.
The workflow produces candidate lessons, model comparisons, and a review step before promotion. The scorecards describe performance on the evaluated tasks, and previous model versions remain available for rollback.
- local models
- immunity gate
- faithfulness scorecards
- competence scorecards
- promotion checks
- versioning + rollback