chore(semantics): add v6 release gate pipeline
- Add reproducible v6 boundary, blessing, and consensus-adjudication corpora, plus the tiny-transformer trainer and v6 release-gate evaluator that gate every candidate on the deployed baselines. - Wire consensus-label merging, product-policy anchor evaluation, and sealed blessing benchmark review with their pytest coverage. - Refresh open-training corpus generation, iterative retraining runner, and random-holdout evaluation so v6 candidates can be benchmarked end-to-end.
This commit is contained in:
@@ -0,0 +1,67 @@
|
||||
# Blessing benchmark review
|
||||
|
||||
This directory contains a blind, evaluation-only review queue derived from the
|
||||
frozen comprehensive holdout. It must never be merged into training.
|
||||
|
||||
## Review process
|
||||
|
||||
1. Give `review-queue.jsonl` and one annotation template to each of two
|
||||
independent human annotators.
|
||||
2. Do not give annotators `sealed-provenance.jsonl`, the other annotator's
|
||||
answers, model predictions, or previous weak labels.
|
||||
3. Each annotator fills every `label`, `boundaryCategory`, and `confidence`
|
||||
field in their own JSONL file.
|
||||
4. Run `finalize_blessing_benchmark.py` with distinct annotator IDs.
|
||||
5. If annotations disagree, give only `adjudication-needed.jsonl` to a third
|
||||
reviewer and rerun the finalizer with the adjudication file.
|
||||
|
||||
## Label
|
||||
|
||||
Set `label` to `true` only when the author directly expresses a good wish,
|
||||
congratulation, prayer, or hope for a recipient. Third-person and self-directed
|
||||
wishes count. Requests for a blessing, quoted examples, received thanks,
|
||||
celebration descriptions, ordinary greetings, reports of someone else's wish,
|
||||
and sarcasm do not count.
|
||||
|
||||
## Boundary categories
|
||||
|
||||
Use one of these stable values:
|
||||
|
||||
### Positive
|
||||
|
||||
- `festival_or_birthday`
|
||||
- `congratulation`
|
||||
- `health_or_recovery`
|
||||
- `travel_or_safety`
|
||||
- `study_or_career`
|
||||
- `general_good_wish`
|
||||
- `third_person_or_group`
|
||||
- `spiritual_or_prayer`
|
||||
|
||||
### Negative
|
||||
|
||||
- `meta_request_or_template`
|
||||
- `received_thanks`
|
||||
- `quoted_or_documented`
|
||||
- `celebration_mention`
|
||||
- `ordinary_greeting`
|
||||
- `positive_language_only`
|
||||
- `reported_wish`
|
||||
- `sarcasm_or_anti_blessing`
|
||||
- `unrelated`
|
||||
|
||||
Use `confidence` values `high`, `medium`, or `low`. Explain genuinely
|
||||
ambiguous context in `notes`.
|
||||
|
||||
## Finalization
|
||||
|
||||
```bash
|
||||
python3 Scripts/clipboard_semantics/finalize_blessing_benchmark.py \
|
||||
--annotator-a-id reviewer-a \
|
||||
--annotator-b-id reviewer-b
|
||||
```
|
||||
|
||||
The command refuses incomplete annotation, duplicate IDs, non-boolean labels,
|
||||
invalid confidence, missing adjudication, or use of the same person as both
|
||||
annotators. The finalized benchmark is split deterministically into calibration
|
||||
and test records.
|
||||
Reference in New Issue
Block a user