Rocky
aa37067f79
- Add reproducible v6 boundary, blessing, and consensus-adjudication corpora, plus the tiny-transformer trainer and v6 release-gate evaluator that gate every candidate on the deployed baselines. - Wire consensus-label merging, product-policy anchor evaluation, and sealed blessing benchmark review with their pytest coverage. - Refresh open-training corpus generation, iterative retraining runner, and random-holdout evaluation so v6 candidates can be benchmarked end-to-end.
3.0 KiB
3.0 KiB
Blessing intent labeling guidelines
Definition
Label blessing = true when the author directly expresses a good wish,
congratulation, prayer, or hope for a recipient. The recipient may be the
reader, a named third party, a group, or the author.
The label is intentionally broad. It includes:
- Festival, birthday, wedding, anniversary, graduation, promotion, new-job, housewarming, newborn, and retirement wishes.
- Short congratulations such as
恭喜,恭喜发财,Congratulations, andCongrats. - Health, recovery, travel, safety, exam, competition, career, and general good-luck wishes.
- Good-night and good-day wishes when they express a desired outcome, such as
祝你好梦orHope you have a wonderful day. - Religious or spiritual prayers directed toward a recipient.
- Wishes expressed for a third person, such as
我衷心祝愿她早日康复.
Negative boundaries
Label blessing = false when the text only:
- Requests, searches for, or discusses how to write a blessing.
- Thanks someone for a blessing already received.
- Mentions that other people sent blessings.
- Describes a celebration without wishing anyone well.
- Quotes or documents a blessing as an example.
- Greets someone without expressing a desired outcome.
- Gives positive feedback or praise without a wish or congratulation.
- Contains a lexical collision such as a title, name, product, or historical reference that happens to include a blessing keyword.
Context-dependent cases
Annotators must use surrounding context when available:
- Sarcastic congratulations are negative unless the product intentionally treats the surface utterance as reply-worthy congratulations.
祝我好运and equivalent self-directed wishes are positive.- Conditional or unrealized intent such as
等会儿再祝他生日快乐is negative until the text actually expresses the wish. - A message may be both
blessingand another intent. For example, a wedding invitation containing祝你们幸福is both an invitation and a blessing.
Annotation process
- Normalize only invisible whitespace; preserve wording, punctuation, and emoji for labeling.
- Two annotators label every golden-set record independently.
- Disagreements are adjudicated by a third reviewer using this document.
- Record the boundary category and adjudication reason, not only the binary label.
- Keep all evaluation examples isolated from generation prompts, training sources, and active-learning exports.
Release evaluation
The dedicated blessing benchmark must contain:
- At least 3,000 Chinese and 1,500 English human-reviewed records.
- Equal positive and hard-negative strata for diagnostic metrics.
- A separate natural-prevalence calibration set for threshold selection.
- At least 100 records for each major positive and negative boundary category.
The release gate is precision at least 0.95, recall at least 0.85, and F1 at least 0.90 on the adjudicated benchmark, with no major boundary category below 0.80 F1.