Files

T

Rocky aa37067f79 chore(semantics): add v6 release gate pipeline

- Add reproducible v6 boundary, blessing, and consensus-adjudication
  corpora, plus the tiny-transformer trainer and v6 release-gate
  evaluator that gate every candidate on the deployed baselines.
- Wire consensus-label merging, product-policy anchor evaluation, and
  sealed blessing benchmark review with their pytest coverage.
- Refresh open-training corpus generation, iterative retraining runner,
  and random-holdout evaluation so v6 candidates can be benchmarked
  end-to-end.

2026-08-29 11:51:42 +08:00

2.0 KiB

Raw Permalink Blame History

Blessing benchmark review

This directory contains a blind, evaluation-only review queue derived from the frozen comprehensive holdout. It must never be merged into training.

Review process

Give review-queue.jsonl and one annotation template to each of two independent human annotators.
Do not give annotators sealed-provenance.jsonl, the other annotator's answers, model predictions, or previous weak labels.
Each annotator fills every label, boundaryCategory, and confidence field in their own JSONL file.
Run finalize_blessing_benchmark.py with distinct annotator IDs.
If annotations disagree, give only adjudication-needed.jsonl to a third reviewer and rerun the finalizer with the adjudication file.

Label

Set label to true only when the author directly expresses a good wish, congratulation, prayer, or hope for a recipient. Third-person and self-directed wishes count. Requests for a blessing, quoted examples, received thanks, celebration descriptions, ordinary greetings, reports of someone else's wish, and sarcasm do not count.

Boundary categories

Use one of these stable values:

Positive

festival_or_birthday
congratulation
health_or_recovery
travel_or_safety
study_or_career
general_good_wish
third_person_or_group
spiritual_or_prayer

Negative

meta_request_or_template
received_thanks
quoted_or_documented
celebration_mention
ordinary_greeting
positive_language_only
reported_wish
sarcasm_or_anti_blessing
unrelated

Use confidence values high, medium, or low. Explain genuinely ambiguous context in notes.

Finalization

python3 Scripts/clipboard_semantics/finalize_blessing_benchmark.py \
  --annotator-a-id reviewer-a \
  --annotator-b-id reviewer-b

The command refuses incomplete annotation, duplicate IDs, non-boolean labels, invalid confidence, missing adjudication, or use of the same person as both annotators. The finalized benchmark is split deterministically into calibration and test records.

2.0 KiB Raw Permalink Blame History