Rocky
aa37067f79
- Add reproducible v6 boundary, blessing, and consensus-adjudication corpora, plus the tiny-transformer trainer and v6 release-gate evaluator that gate every candidate on the deployed baselines. - Wire consensus-label merging, product-policy anchor evaluation, and sealed blessing benchmark review with their pytest coverage. - Refresh open-training corpus generation, iterative retraining runner, and random-holdout evaluation so v6 candidates can be benchmarked end-to-end.
68 lines
2.0 KiB
Markdown
68 lines
2.0 KiB
Markdown
# Blessing benchmark review
|
|
|
|
This directory contains a blind, evaluation-only review queue derived from the
|
|
frozen comprehensive holdout. It must never be merged into training.
|
|
|
|
## Review process
|
|
|
|
1. Give `review-queue.jsonl` and one annotation template to each of two
|
|
independent human annotators.
|
|
2. Do not give annotators `sealed-provenance.jsonl`, the other annotator's
|
|
answers, model predictions, or previous weak labels.
|
|
3. Each annotator fills every `label`, `boundaryCategory`, and `confidence`
|
|
field in their own JSONL file.
|
|
4. Run `finalize_blessing_benchmark.py` with distinct annotator IDs.
|
|
5. If annotations disagree, give only `adjudication-needed.jsonl` to a third
|
|
reviewer and rerun the finalizer with the adjudication file.
|
|
|
|
## Label
|
|
|
|
Set `label` to `true` only when the author directly expresses a good wish,
|
|
congratulation, prayer, or hope for a recipient. Third-person and self-directed
|
|
wishes count. Requests for a blessing, quoted examples, received thanks,
|
|
celebration descriptions, ordinary greetings, reports of someone else's wish,
|
|
and sarcasm do not count.
|
|
|
|
## Boundary categories
|
|
|
|
Use one of these stable values:
|
|
|
|
### Positive
|
|
|
|
- `festival_or_birthday`
|
|
- `congratulation`
|
|
- `health_or_recovery`
|
|
- `travel_or_safety`
|
|
- `study_or_career`
|
|
- `general_good_wish`
|
|
- `third_person_or_group`
|
|
- `spiritual_or_prayer`
|
|
|
|
### Negative
|
|
|
|
- `meta_request_or_template`
|
|
- `received_thanks`
|
|
- `quoted_or_documented`
|
|
- `celebration_mention`
|
|
- `ordinary_greeting`
|
|
- `positive_language_only`
|
|
- `reported_wish`
|
|
- `sarcasm_or_anti_blessing`
|
|
- `unrelated`
|
|
|
|
Use `confidence` values `high`, `medium`, or `low`. Explain genuinely
|
|
ambiguous context in `notes`.
|
|
|
|
## Finalization
|
|
|
|
```bash
|
|
python3 Scripts/clipboard_semantics/finalize_blessing_benchmark.py \
|
|
--annotator-a-id reviewer-a \
|
|
--annotator-b-id reviewer-b
|
|
```
|
|
|
|
The command refuses incomplete annotation, duplicate IDs, non-boolean labels,
|
|
invalid confidence, missing adjudication, or use of the same person as both
|
|
annotators. The finalized benchmark is split deterministically into calibration
|
|
and test records.
|