rocky/OSGKeyboard
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore(semantics): add v6 release gate pipeline

Browse Source 
- Add reproducible v6 boundary, blessing, and consensus-adjudication
  corpora, plus the tiny-transformer trainer and v6 release-gate
  evaluator that gate every candidate on the deployed baselines.
- Wire consensus-label merging, product-policy anchor evaluation, and
  sealed blessing benchmark review with their pytest coverage.
- Refresh open-training corpus generation, iterative retraining runner,
  and random-holdout evaluation so v6 candidates can be benchmarked
  end-to-end.
This commit is contained in:
Rocky
2026-08-29 11:51:42 +08:00
parent b275b6b0d9
commit aa37067f79
50 changed files with 12107 additions and 197 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ModelTraining/ClipboardSemantics/BlessingBenchmark/README.md
+67
View File
@@ -0,0 +1,67 @@
# Blessing benchmark review
This directory contains a blind, evaluation-only review queue derived from the
frozen comprehensive holdout. It must never be merged into training.
## Review process
1. Give `review-queue.jsonl` and one annotation template to each of two
independent human annotators.
2. Do not give annotators `sealed-provenance.jsonl`, the other annotator's
answers, model predictions, or previous weak labels.
3. Each annotator fills every `label`, `boundaryCategory`, and `confidence`
field in their own JSONL file.
4. Run `finalize_blessing_benchmark.py` with distinct annotator IDs.
5. If annotations disagree, give only `adjudication-needed.jsonl` to a third
reviewer and rerun the finalizer with the adjudication file.
## Label
Set `label` to `true` only when the author directly expresses a good wish,
congratulation, prayer, or hope for a recipient. Third-person and self-directed
wishes count. Requests for a blessing, quoted examples, received thanks,
celebration descriptions, ordinary greetings, reports of someone else's wish,
and sarcasm do not count.
## Boundary categories
Use one of these stable values:
### Positive
- `festival_or_birthday`
- `congratulation`
- `health_or_recovery`
- `travel_or_safety`
- `study_or_career`
- `general_good_wish`
- `third_person_or_group`
- `spiritual_or_prayer`
### Negative
- `meta_request_or_template`
- `received_thanks`
- `quoted_or_documented`
- `celebration_mention`
- `ordinary_greeting`
- `positive_language_only`
- `reported_wish`
- `sarcasm_or_anti_blessing`
- `unrelated`
Use `confidence` values `high`, `medium`, or `low`. Explain genuinely
ambiguous context in `notes`.
## Finalization
```bash
python3 Scripts/clipboard_semantics/finalize_blessing_benchmark.py \
--annotator-a-id reviewer-a \
--annotator-b-id reviewer-b
```
The command refuses incomplete annotation, duplicate IDs, non-boolean labels,
invalid confidence, missing adjudication, or use of the same person as both
annotators. The finalized benchmark is split deterministically into calibration
and test records.
ModelTraining/ClipboardSemantics/Consensus/adjudication-instructions-v1.md
+40
View File
@@ -0,0 +1,40 @@
# Clipboard semantic evidence adjudication v1
Review only the fields listed in `unresolvedFields`. Judge from the text itself;
do not inspect previous model votes, source labels, or another adjudicator.
Return one JSON object per input record:
```json
{
"id": "same id",
"resolutions": {
"task": "true",
"ambiguous": "false"
},
"confidence": {
"task": 0.97,
"ambiguous": 0.94
},
"evidence": {
"task": "send the report",
"ambiguous": "by Friday"
}
}
```
Requirements:
- `resolutions`, `confidence`, and `evidence` must contain exactly the fields in
`unresolvedFields`.
- Intent and flag values are `true`, `false`, or `unknown`.
- Sentiment values are `positive`, `neutral`, `negative`, or `unknown`.
- Evidence must be a short exact quote copied from the input text.
- Use `unknown` when the text alone does not justify a decision.
- Confidence is per field and must be between 0 and 1.
- Do not output explanations, markdown, or additional fields.
Use the boundaries from `labeling-instructions-v2.md`. In particular, distinguish
requests from personal plans, genuine information questions from request-shaped
commands, direct messages from terminal notices, and expressed wishes from
quoted, future, sarcastic, or received blessings.
ModelTraining/ClipboardSemantics/Consensus/adjudication-instructions-v2.md
+56
View File
@@ -0,0 +1,56 @@
# Clipboard semantic evidence adjudication v2
Review only the fields listed in `unresolvedFields`. Judge from the text itself;
do not inspect previous model votes, source labels, or another adjudicator.
Apply the product-approved boundaries from `labeling-instructions-v3.md`.
Return one JSON object per input record:
```json
{
"id": "same id",
"recordDisposition": "keep",
"dispositionConfidence": 0.98,
"dispositionEvidence": "send the report",
"resolutions": {
"task": "true",
"ambiguous": "false"
},
"confidence": {
"task": 0.97,
"ambiguous": 0.94
},
"evidence": {
"task": "send the report",
"ambiguous": "by Friday"
}
}
```
Requirements:
- `recordDisposition` must be `keep` or `exclude-device-command`.
- Use `exclude-device-command` only when the text is clearly addressed to a
device, app, search engine, or virtual assistant. Do not use it for an
ordinary request to another person.
- `dispositionConfidence` must be between 0 and 1.
- `dispositionEvidence` must be a short exact quote copied from the text.
- `resolutions`, `confidence`, and `evidence` must contain exactly the fields in
`unresolvedFields`, even when the record is marked for exclusion.
- Intent and flag values are `true`, `false`, or `unknown`.
- Sentiment values are `positive`, `neutral`, `negative`, or `unknown`.
- Field evidence must be a short exact quote copied from the input text.
- Use `unknown` when the text alone does not justify a decision.
- Confidence is per field and must be between 0 and 1.
- Do not output explanations, markdown, or additional fields.
Critical product decisions:
- An invitation question is `invitation=true`, `question=false`, and normally
`replyableMessage=true`.
- An explicit self-reminder is `followUpReminder=true`, `task=false`.
- A first-person need implying action is `task=true`.
- A problem is not a complaint unless dissatisfaction, criticism, or objection
is explicitly expressed.
- Generic encouragement or happiness is not a blessing; require an explicit
wish, prayer, congratulation, or conventional blessing.
ModelTraining/ClipboardSemantics/Consensus/adjudication-instructions-v3.md
+23
View File
@@ -0,0 +1,23 @@
# Clipboard semantic evidence adjudication v3
Review only the fields listed in `unresolvedFields`. Judge from the text itself;
do not inspect previous model votes, source labels, or another adjudicator.
Apply `labeling-instructions-v4.md`.
Return one JSON object per input record using the exact schema defined in
`adjudication-instructions-v2.md`.
For compatibility, `recordDisposition` remains `keep` or
`exclude-device-command`. In v3, `exclude-device-command` also covers generic
search, system/account queries, and other clearly virtual-assistant-only text.
It does not cover real-world service requests, named-recipient communication
with content, or private/shared-context interpersonal questions.
Requirements:
- Evidence must be a short exact quote copied from the input text.
- `resolutions`, `confidence`, and `evidence` must contain exactly the fields in
`unresolvedFields`.
- Use `unknown` rather than inventing context for low-information fragments.
- Confidence is per field and must be between 0 and 1.
- Do not output explanations, markdown, or additional fields.
ModelTraining/ClipboardSemantics/Consensus/adjudication-instructions-v4.md
+17
View File
@@ -0,0 +1,17 @@
# Clipboard semantic evidence adjudication v4
Review only the fields listed in `unresolvedFields`. Judge from the text itself;
do not inspect human anchor labels, previous model votes, source labels, or
another adjudicator.
Apply `labeling-instructions-v5.md` and every inherited rule from
`labeling-instructions-v4.md`. Return one JSON object per input record using the
exact schema from `adjudication-instructions-v2.md`.
For compatibility, `recordDisposition` remains `keep` or
`exclude-device-command`; the excluded state also covers generic search,
system/account queries, and clearly virtual-assistant-only commands.
Evidence must be an exact text quote. Use `unknown` for every affected intent
when a short fragment lacks enough context. Do not output explanations,
markdown, or additional fields.
ModelTraining/ClipboardSemantics/Consensus/adjudication-instructions-v5.md
+79
View File
@@ -0,0 +1,79 @@
# Clipboard semantic evidence adjudication v5
Review only the fields listed in `unresolvedFields`. Judge from the text itself;
do not inspect source labels, provenance, previous model votes, human anchors,
or another adjudicator.
Apply `labeling-instructions-v6.md`. Return one JSON object per input record
using the evidence schema from `adjudication-instructions-v2.md`:
```json
{
"id": "same id",
"recordDisposition": "keep",
"dispositionConfidence": 0.98,
"dispositionEvidence": "play my workout playlist",
"resolutions": {
"assistantCommand": "true",
"domain": "media"
},
"confidence": {
"assistantCommand": 0.97,
"domain": 0.95
},
"evidence": {
"assistantCommand": "play",
"domain": "workout playlist"
}
}
```
Requirements:
- `recordDisposition` remains `keep` or `exclude-device-command` for file-format
compatibility. Under v6, assistant commands, information queries, and system
notifications are taxonomy records and must be `keep`.
- Use `exclude-device-command` only when replaying a queue explicitly frozen
under v4/v5 exclusion policy. Do not use it in a new v6 queue.
- `dispositionConfidence` and every per-field confidence must be between 0 and
1.
- `dispositionEvidence` and field evidence must be short exact quotes copied
from the input text.
- `resolutions`, `confidence`, and `evidence` must contain exactly the fields in
`unresolvedFields`, even if a legacy replay record is excluded.
- Intent and flag values are `true`, `false`, or `unknown`.
- `domain` is `finance`, `travel`, `calendar`, `communication`, `media`,
`smartHome`, `shopping`, `dining`, `health`, `weather`, `accountService`,
`generalKnowledge`, or `unknown`.
- `sentiment` is `positive`, `neutral`, `negative`, or `unknown`.
- Use `unknown` when the text alone does not justify a stable decision. A
low-information fragment must not be forced to `false` or into
`generalKnowledge`.
- Do not output explanations, markdown, comments, or additional fields.
Critical boundary checks:
- Resolve clearly device-, app-, or assistant-directed direct digital
operations as `assistantCommand=true`; resolve generic lookups as
`informationQuery=true`; resolve machine-authored alerts and status messages
as `systemNotification=true`.
- A request for a real-world service that naturally needs confirmation, such as
booking a taxi, restaurant, hotel, or ticket, remains `task=true` rather than
`assistantCommand=true`.
- Do not infer `task`, `question`, or `replyableMessage` from those records
unless the text independently contains an interpersonal act.
- A named-recipient communication request with actual content is an
interpersonal replyable task. A bare “call Mark” remains unknown for affected
fields when the addressee is unclear.
- Invitation questions are `invitation=true`, `question=false`; explicit
self-reminders are `followUpReminder=true`, `task=false`; first-person needs
implying personal action remain tasks.
- Complaints require explicit dissatisfaction, and blessings require an
explicit wish, prayer, congratulation, or conventional blessing.
- Choose one domain from the operation target. If no primary target can be
established, resolve `domain` as `unknown`.
`knownLabels` is corpus metadata and must not appear in adjudicator output.
Downstream merging may add an adjudicated field to `knownLabels` only when both
the field value and its evidence pass the configured acceptance gate and the
value is not `unknown`.
ModelTraining/ClipboardSemantics/Consensus/labeling-instructions-v2.md
+70
View File
@@ -0,0 +1,70 @@
# Clipboard semantic consensus labeling v2
Prompt version: `clipboard-consensus-v2`
Label each record independently using only its text. Do not infer missing
conversation history, and do not inspect source labels or other model outputs.
Return exactly one JSON object per input record:
```json
{
"id": "same id",
"labels": {
"task": "false",
"question": "false",
"invitation": "false",
"complaint": "false",
"scheduleNegotiation": "false",
"confirmationDecision": "false",
"followUpReminder": "false",
"blessing": "false",
"replyableMessage": "true"
},
"sentiment": "neutral",
"ambiguous": false,
"quotedOrMeta": false,
"confidence": 0.96
}
```
Every intent value must be `true`, `false`, or `unknown`. Use `unknown` when
the text alone does not contain enough evidence. Absence of evidence is not
automatically evidence of a negative label.
## Intent boundaries
- `task`: another person is explicitly requested or assigned to perform an
action. A personal plan is not a task.
- `question`: a genuine request for information. Rhetorical, quoted, search,
and documentation examples are not questions.
- `invitation`: an invitation to join an event, meeting, visit, meal, or social
activity.
- `complaint`: present dissatisfaction, malfunction, bad service, or an
unresolved problem. Negative sentiment alone is insufficient.
- `scheduleNegotiation`: proposing, changing, comparing, or choosing between
times. A fixed appointment or deadline alone is insufficient.
- `confirmationDecision`: explicit approval, rejection, commitment, or
selection of an option. Acknowledgment alone is insufficient.
- `followUpReminder`: a request to remind, check back, or follow up later or
after a trigger. An ordinary task with a deadline is insufficient.
- `blessing`: the author directly expresses a good wish, congratulation,
prayer, or hope for any recipient, including self or third parties.
- `replyableMessage`: a direct conversational message that naturally invites
a response. Terminal acknowledgments, personal notes, quoted examples, and
factual notices are negative.
Multi-label combinations are valid. For example, “Could you send the report?”
is `task + question + replyableMessage`.
## Special cases
- Set `quotedOrMeta = true` when intent-bearing language is quoted, reported,
searched, documented, requested as a writing example, or discussed rather
than performed.
- Set `ambiguous = true` when material context is missing or multiple
interpretations remain equally plausible.
- Sarcasm, negation, hypothetical future intent, and received thanks must be
interpreted semantically rather than by keyword matching.
- `sentiment` must be `positive`, `neutral`, `negative`, or `unknown`.
- Do not output reasoning, markdown, comments, or additional fields.
ModelTraining/ClipboardSemantics/Consensus/labeling-instructions-v3.md
+46
View File
@@ -0,0 +1,46 @@
# Clipboard semantic consensus labeling v3
Prompt version: `clipboard-consensus-v3`
Label each record independently using only its text. Do not infer missing
conversation history, and do not inspect source labels or other model outputs.
Return exactly one JSON object per input record using the schema from
`labeling-instructions-v2.md`.
## Product-approved boundaries
These rules override the corresponding v2 boundaries:
- Exclude commands that are clearly addressed to a device, app, search engine,
or virtual assistant rather than another person. Examples include opening an
inbox, playing music, changing device volume, or showing an account value.
Ordinary requests sent to another person remain in scope.
- `task`: a first-person need that implies an action is a task even when the
recipient is not explicit. A self-reminder is not a task. A device or virtual
assistant command is excluded before intent labeling.
- `question`: an invitation phrased as a question is not a `question`.
Request-shaped commands are also not information questions.
- `invitation`: an invitation phrased as a question is
`invitation=true`, `question=false`, and normally
`replyableMessage=true`.
- `complaint`: require an explicit expression of dissatisfaction, criticism, or
objection. A loss, theft, malfunction, or unresolved problem without
expressed dissatisfaction is not a complaint.
- `followUpReminder`: an explicit self-reminder is
`followUpReminder=true` and `task=false`.
- `blessing`: require an explicit wish, prayer, congratulation, or conventional
blessing. Generic encouragement, happiness for someone, optimism, or “good
luck”-free motivational language is not sufficient. Conventional expressions
such as “生日快乐”, “一路顺风”, “恭喜晋升”, “happy birthday”, and
“congratulations on the promotion” are explicit.
## Unchanged requirements
- Every intent value is `true`, `false`, or `unknown`.
- `sentiment` is `positive`, `neutral`, `negative`, or `unknown`.
- Set `quotedOrMeta=true` for quoted, reported, searched, documented, or
example-only intent language.
- Set `ambiguous=true` only when missing context materially prevents a stable
product label.
- Do not output reasoning, markdown, comments, or additional fields.
ModelTraining/ClipboardSemantics/Consensus/labeling-instructions-v4.md
+64
View File
@@ -0,0 +1,64 @@
# Clipboard semantic consensus labeling v4
Prompt version: `clipboard-consensus-v4`
Label from the text alone. Do not inspect source labels, model votes, or hidden
conversation history. The product owner decisions below override all earlier
versions.
## Record scope
Exclude text that is clearly a generic search, device control, app/account
query, alarm/calendar operation, or other virtual-assistant-only command.
Keep these in scope:
- a request to perform a real-world service that naturally needs confirmation,
such as booking a taxi;
- a request to communicate with a named recipient when the content to convey is
present;
- a private or shared-context information question that could naturally be sent
to another person, such as asking for a relative's email address.
If a short fragment does not contain enough evidence to distinguish a human
message from a query or command, keep it unresolved with `ambiguous=true` and
the affected intents set to `unknown`. Do not force it into the excluded or
negative class.
## Product intent boundaries
- `replyableMessage=true` when an in-scope interpersonal message naturally
supports a response. Questions, assignments, ongoing decisions, emotional
updates, and outcome sharing can be replyable.
- Terminal acknowledgments and thanks such as “知道了，谢谢” are not
replyable. A passive factual notice that creates no conversational next step
is also not replyable.
- `task=true` for an assigned action, an explicit first-person commitment, or a
first-person need that implies a personal action. A pure status question is
not a task.
- A request to email, text, or otherwise contact a named recipient is a
replyable task when the message content or purpose is included. A bare
command such as “call Mark” is ambiguous without more context.
- `question=true` for any genuine request for information, including an
imperative such as “tell me her email address”.
- A polite interrogative action request such as “Can you send the report?” is
both `task=true` and `question=true`. A question about when an existing task
will happen is `question=true`, `task=false`.
- A request for a recommendation is excluded when it is clearly a generic
assistant/search query rather than an interpersonal request.
- An invitation phrased as a question remains `invitation=true`,
`question=false`, and normally `replyableMessage=true`.
- An explicit self-reminder remains `followUpReminder=true`, `task=false`.
- A complaint still requires explicit dissatisfaction, criticism, or objection.
- A blessing still requires an explicit wish, prayer, congratulation, or
conventional blessing.
## Output states
- Intent values are `true`, `false`, or `unknown`.
- `sentiment` is `positive`, `neutral`, `negative`, or `unknown`.
- Use `ambiguous=true` only when missing context materially prevents a stable
product label.
- Use `quotedOrMeta=true` for quoted, reported, searched, documented, or
example-only intent language.
- Do not output explanations, markdown, comments, or additional fields.
ModelTraining/ClipboardSemantics/Consensus/labeling-instructions-v5.md
+39
View File
@@ -0,0 +1,39 @@
# Clipboard semantic consensus labeling v5
Prompt version: `clipboard-consensus-v5`
Apply every rule in `labeling-instructions-v4.md`, with the following
product-owner clarifications taking precedence.
## Replyable message clarifications
- “take your time” is an interpersonal supportive message:
`replyableMessage=true`, `task=false`.
- “知道了，谢谢” is terminal and not replyable.
- “活动规则按当前方案通过” is a passive decision notice and not replyable.
- Sharing a personal outcome such as “事情总算处理完了，结果居然成了” is
replyable even without a direct question.
- A private first-person need such as “I need to set up a new PIN” is a task
but not replyable unless it is addressed to another person.
## Task clarifications
- A first-person decision followed by an impersonal consequence is not
automatically an assignment. “我拍板先发布基础版，其他候选停止评估” is
replyable but not a task because it does not directly assign the recipient.
- A decision that explicitly hands off a next action is a task. “我批准退款流程
的最终版本，可以签字” is replyable and a task.
- Named-recipient communication with actual content is a replyable task:
“text Sarah that I'll be late” and “send an email to Julie that I can meet
Saturday” are both `replyableMessage=true`, `task=true`.
## Scope and ambiguity clarifications
- “tell me what's new” and “my claim status” are generic assistant/system
queries and must be excluded.
- A bare fragment such as “call Mark” does not reveal whether it is an
interpersonal assignment or an assistant command. Keep it unresolved:
`ambiguous=true`, with `replyableMessage`, `task`, and `question` all
`unknown`.
- Apply the same unknown treatment to other low-information fragments rather
than converting unspecified fields to `false`.
ModelTraining/ClipboardSemantics/Consensus/labeling-instructions-v6.md
+201
View File
@@ -0,0 +1,201 @@
# Clipboard semantic consensus labeling v6
Prompt version: `clipboard-consensus-v6`
Label each record independently from its text. Do not inspect source labels,
model votes, provenance, or hidden conversation history. This version keeps the
nine product intents from v5 and adds three routing intents plus one domain
field. Its definitions override earlier instructions when they conflict.
## Output schema
Return exactly one JSON object per input record:
```json
{
"id": "same id",
"labels": {
"task": "false",
"question": "false",
"invitation": "false",
"complaint": "false",
"scheduleNegotiation": "false",
"confirmationDecision": "false",
"followUpReminder": "false",
"blessing": "false",
"replyableMessage": "true",
"assistantCommand": "false",
"informationQuery": "false",
"systemNotification": "false"
},
"domain": "communication",
"sentiment": "neutral",
"ambiguous": false,
"quotedOrMeta": false,
"confidence": 0.96
}
```
Every intent value is `true`, `false`, or `unknown`. `domain` is one of the
twelve values below or `unknown`. `sentiment` is `positive`, `neutral`,
`negative`, or `unknown`. Do not output reasoning, comments, markdown, or
additional fields.
## Nine product intents
- `task`: a person is assigned or asked to perform an action, or the author
states an explicit personal commitment or need that implies action. A
request for a real-world service that naturally needs confirmation, such as
booking a taxi, restaurant, hotel, or ticket, is also a task. A status
question, self-reminder, passive decision notice, and pure device/app
operation are not tasks.
- `question`: a genuine interpersonal request for information. It includes
imperative requests such as “tell me her email address” when they could
naturally be sent to a person. Invitation questions and clearly generic
assistant/search/account queries are not questions.
- `invitation`: an invitation to join an event, meeting, visit, meal, or social
activity. A question-shaped invitation is normally also
`replyableMessage=true`, but `question=false`.
- `complaint`: explicit present dissatisfaction, criticism, objection, bad
service, or an unresolved problem framed as a complaint. A loss,
malfunction, negative fact, or negative sentiment without expressed
dissatisfaction is insufficient.
- `scheduleNegotiation`: proposing, changing, comparing, or choosing between
times. A fixed appointment, reminder time, or deadline alone is insufficient.
- `confirmationDecision`: explicit approval, rejection, commitment, or
selection of an option. Acknowledgment, receipt confirmation, and passive
status notice alone are insufficient.
- `followUpReminder`: an explicit request to remind, check back, or follow up
later or after a trigger, including a self-reminder. An ordinary task with a
deadline is insufficient; a self-reminder is not a `task`.
- `blessing`: the author directly expresses an explicit wish, prayer,
congratulation, or conventional blessing. Generic encouragement, optimism,
happiness for someone, quoted wishes, and requests to write a blessing are
insufficient.
- `replyableMessage`: an interpersonal message that naturally supports a
response. Questions, assignments, invitations, ongoing decisions, emotional
updates, and outcome sharing may qualify. Terminal acknowledgments or thanks,
private notes, passive factual notices, generic assistant interactions, and
machine notifications do not.
The v5 product-owner examples remain authoritative: “take your time” is
replyable but not a task; “知道了，谢谢” is terminal; personal outcome sharing
may be replyable; a private first-person need may be a task without being
replyable; and named-recipient communication with actual content is a replyable
task.
## Three routing intents
- `assistantCommand`: an instruction to a device, app, service, search engine,
or virtual assistant to perform a direct digital or device operation. This
includes opening or changing app state, alarms and calendar operations, media
playback, smart-home control, and immediate account/app settings. A
real-world service request that naturally needs confirmation remains a
`task`, even when submitted through an assistant.
- `informationQuery`: a generic assistant, search, reference, weather, account,
or service-status lookup that asks for information rather than asking a
person. “tell me what's new”, “my claim status”, and generic recommendation
searches qualify.
- `systemNotification`: machine- or service-generated status, alert, receipt,
security warning, delivery update, or other notification presented to the
user rather than authored as an interpersonal message.
These three labels replace the old blanket exclusion of assistant-only text.
Keep such records and label them explicitly. They are normally mutually
exclusive, and their clearly assistant/system-scoped records must not become
`task`, `question`, or `replyableMessage` merely because similar words could
occur in human conversation. Real-world bookings remain tasks. A request to
contact a named person with message content is interpersonal, not an
`assistantCommand`; a bare fragment such as “call Mark” remains ambiguous when
addressee and interaction mode cannot be determined.
## Domains
Choose the single primary subject or operation target:
- `finance`: banking, payments, cards, transfers, investments, insurance, or
claims.
- `travel`: transport, routes, tickets, hotels, trips, or reservations other
than restaurant bookings.
- `calendar`: dates, events, meetings, availability, alarms, reminders, or
scheduling.
- `communication`: calls, contacts, messages, email, social communication, or
interpersonal conversation.
- `media`: music, podcasts, radio, video, photos, news playback, or media
discovery.
- `smartHome`: lights, appliances, climate, locks, cameras, or other connected
home devices.
- `shopping`: products, orders, retail delivery, returns, refunds, or
marketplace activity.
- `dining`: restaurants, food, menus, takeaway, restaurant reservations, or
dining service.
- `health`: symptoms, care, medicine, fitness, wellbeing, or medical
appointments.
- `weather`: current conditions, forecasts, temperature, or weather alerts.
- `accountService`: login, identity, profile, PIN/password, subscription,
membership, entitlement, or general service support not better covered above.
- `generalKnowledge`: general facts, definitions, recommendations, and
non-specialized content that does not fit another domain.
Use the action target to resolve a cross-domain record: “text Sam about the
flight” is `communication`, while “is my flight delayed?” is `travel`. Use
`unknown`, not `generalKnowledge`, when missing context prevents a stable
choice.
## Unknown, ambiguity, and metadata
- Use `unknown` only when the text lacks enough evidence for that field. Do not
turn missing annotation or missing context into `false`.
- Use `false` when the field is in scope and the text provides enough evidence
that the intent is absent.
- Set `ambiguous=true` when missing context materially prevents a stable product
label. Set each affected intent and `domain` to `unknown`; unaffected fields
may still be resolved.
- Set `quotedOrMeta=true` when intent-bearing language is quoted, reported,
searched, documented, requested as a writing example, or discussed rather
than performed.
- Multi-label product combinations remain valid, such as
`task + question + replyableMessage` for an interpersonal “Could you send the
report?”
## `knownLabels` contract for corpus records
`knownLabels` is ingestion metadata, not part of labeler output. It lists only
the fields a source genuinely annotates after an audited deterministic mapping.
Allowed names are the twelve intent names, `domain`, and `sentiment`.
- A field in `knownLabels` may train from its resolved value, including an
explicit `false`.
- A field absent from `knownLabels` is `unknown` for training and contributes no
positive or negative loss.
- Source intent names, topic names, or missing columns must never be expanded
into negative labels for the rest of the taxonomy.
- A mapped source label may make only its audited target fields known.
Synthetic data must not claim all labels known merely because the generator
omitted them.
- Consensus or human review may add a field to `knownLabels` only after that
field receives a non-`unknown` decision under this taxonomy.
## Training-data boundary
- External data may enter candidate generation only when its commercial-use
rights and required notices are recorded, its immutable revision is pinned,
and it comes from the upstream official `train` split. Upstream validation,
development, test, challenge, and hidden-evaluation records never train.
- When an upstream source publishes only one split explicitly named `train`, it
may supply training candidates but may not supply OSGKeyboard calibration or
evaluation truth. If no official train designation exists, the source waits
in audit and is not locally re-split into eligibility.
- Exact and normalized near-duplicate overlap with any frozen local holdout is a
fatal exclusion. Privacy, credentials, direct contact data, unsafe content,
and unsupported language variants are filtered before labeling.
- Synthetic records are training-only, carry explicit synthetic provenance,
use sample weight at most `0.35`, and may know only the fields guaranteed by
their generation contract. They cannot enter calibration, evaluation, human
gold, or policy-anchor sets; cannot override a conflicting human or licensed
non-synthetic example; and cannot by themselves authorize a new boundary or
deployment threshold.
- Dataset admission means eligibility for the audited candidate queue, not
automatic inclusion in commercial training. Every generated artifact still
requires pinned license evidence, attribution, mapping review, deduplication,
and acceptance gates.
ModelTraining/ClipboardSemantics/Consensus/product-policy-anchors-v1.json
+182
View File
@@ -0,0 +1,182 @@
[
{
"id": "human-anchor-001",
"text": "下一班离开纽约市的火车几点出发",
"language": "zh-Hans",
"expected": {"recordDisposition": "exclude-device-command"}
},
{
"id": "human-anchor-002",
"text": "告诉我露丝姑妈的电子邮件地址",
"language": "zh-Hans",
"expected": {"recordDisposition": "keep", "replyableMessage": "true", "task": "false", "question": "true", "ambiguous": "false"}
},
{
"id": "human-anchor-003",
"text": "顺便说一下，这件事我会亲自处理：预约会议室，计划周五之前开始。",
"language": "zh-Hans",
"expected": {"recordDisposition": "keep", "replyableMessage": "true", "task": "true", "question": "false", "ambiguous": "false"}
},
{
"id": "human-anchor-004",
"text": "在我上班的路上有什么事故吗",
"language": "zh-Hans",
"expected": {"recordDisposition": "exclude-device-command"}
},
{
"id": "human-anchor-005",
"text": "take your time",
"language": "en",
"expected": {"recordDisposition": "keep", "replyableMessage": "true", "task": "false", "question": "false", "ambiguous": "false"}
},
{
"id": "human-anchor-006",
"text": "tell me what's new",
"language": "en",
"expected": {"recordDisposition": "exclude-device-command"}
},
{
"id": "human-anchor-007",
"text": "知道了，这边谢谢。",
"language": "zh-Hans",
"expected": {"recordDisposition": "keep", "replyableMessage": "false", "task": "false", "question": "false", "ambiguous": "false"}
},
{
"id": "human-anchor-008",
"text": "另外，活动规则按当前方案通过。",
"language": "zh-Hans",
"expected": {"recordDisposition": "keep", "replyableMessage": "false", "task": "false", "question": "false", "ambiguous": "false"}
},
{
"id": "human-anchor-009",
"text": "我拍板用先发布基础版，其他候选停止评估。",
"language": "zh-Hans",
"expected": {"recordDisposition": "keep", "replyableMessage": "true", "task": "false", "question": "false", "ambiguous": "false"}
},
{
"id": "human-anchor-010",
"text": "还有一件事，我批准退款流程的最终版本，可以签字。",
"language": "zh-Hans",
"expected": {"recordDisposition": "keep", "replyableMessage": "true", "task": "true", "question": "false", "ambiguous": "false"}
},
{
"id": "human-anchor-011",
"text": "事情总算处理完了，结果居然成了。",
"language": "zh-Hans",
"expected": {"recordDisposition": "keep", "replyableMessage": "true", "task": "false", "question": "false", "ambiguous": "false"}
},
{
"id": "human-anchor-012",
"text": "好的，谢谢您的回答。",
"language": "zh-Hans",
"expected": {"recordDisposition": "keep", "replyableMessage": "false", "task": "false", "question": "false", "ambiguous": "false"}
},
{
"id": "human-anchor-013",
"text": "你能把报告发给我吗？",
"language": "zh-Hans",
"expected": {"recordDisposition": "keep", "replyableMessage": "true", "task": "true", "question": "true", "ambiguous": "false"}
},
{
"id": "human-anchor-014",
"text": "报告什么时候发给我？",
"language": "zh-Hans",
"expected": {"recordDisposition": "keep", "replyableMessage": "true", "task": "false", "question": "true", "ambiguous": "false"}
},
{
"id": "human-anchor-015",
"text": "请推荐一家附近评分 4.5 以上的餐馆。",
"language": "zh-Hans",
"expected": {"recordDisposition": "exclude-device-command"}
},
{
"id": "human-anchor-016",
"text": "I need to set up a new PIN.",
"language": "en",
"expected": {"recordDisposition": "keep", "replyableMessage": "false", "task": "true", "question": "false", "ambiguous": "false"}
},
{
"id": "human-anchor-017",
"text": "my claim status",
"language": "en",
"expected": {"recordDisposition": "exclude-device-command"}
},
{
"id": "human-anchor-018",
"text": "send an email to Julie that I can meet her Saturday at 1 p.m.",
"language": "en",
"expected": {"recordDisposition": "keep", "replyableMessage": "true", "task": "true", "question": "false", "ambiguous": "false"}
},
{
"id": "human-anchor-019",
"text": "call Mark",
"language": "en",
"expected": {"replyableMessage": "unknown", "task": "unknown", "question": "unknown", "ambiguous": "true"}
},
{
"id": "human-anchor-020",
"text": "text Sarah that I'll be late",
"language": "en",
"expected": {"recordDisposition": "keep", "replyableMessage": "true", "task": "true", "question": "false", "ambiguous": "false"}
},
{
"id": "human-anchor-021",
"text": "open my inbox",
"language": "en",
"expected": {"recordDisposition": "exclude-device-command"}
},
{
"id": "human-anchor-022",
"text": "show me my account balance",
"language": "en",
"expected": {"recordDisposition": "exclude-device-command"}
},
{
"id": "human-anchor-023",
"text": "book me a taxi to the airport",
"language": "en",
"expected": {"recordDisposition": "keep", "replyableMessage": "true", "task": "true", "question": "false", "ambiguous": "false"}
},
{
"id": "human-anchor-024",
"text": "set an alarm for 4 a.m.",
"language": "en",
"expected": {"recordDisposition": "exclude-device-command"}
},
{
"id": "human-anchor-025",
"text": "internet",
"language": "en",
"expected": {"replyableMessage": "unknown", "task": "unknown", "question": "unknown", "ambiguous": "true"}
},
{
"id": "human-anchor-026",
"text": "chennai 45879",
"language": "en",
"expected": {"replyableMessage": "unknown", "task": "unknown", "question": "unknown", "ambiguous": "true"}
},
{
"id": "human-anchor-027",
"text": "seat assignment",
"language": "en",
"expected": {"replyableMessage": "unknown", "task": "unknown", "question": "unknown", "ambiguous": "true"}
},
{
"id": "human-anchor-028",
"text": "maximum kilometer to delivery",
"language": "en",
"expected": {"replyableMessage": "unknown", "task": "unknown", "question": "unknown", "ambiguous": "true"}
},
{
"id": "human-anchor-029",
"text": "事件重复的整个月",
"language": "zh-Hans",
"expected": {"replyableMessage": "unknown", "task": "unknown", "question": "unknown", "ambiguous": "true"}
},
{
"id": "human-anchor-030",
"text": "意大利面的食谱",
"language": "zh-Hans",
"expected": {"recordDisposition": "exclude-device-command"}
}
]
ModelTraining/ClipboardSemantics/blessing-labeling-guidelines.md
+70
View File
@@ -0,0 +1,70 @@
# Blessing intent labeling guidelines
## Definition
Label `blessing = true` when the author directly expresses a good wish,
congratulation, prayer, or hope for a recipient. The recipient may be the
reader, a named third party, a group, or the author.
The label is intentionally broad. It includes:
- Festival, birthday, wedding, anniversary, graduation, promotion, new-job,
housewarming, newborn, and retirement wishes.
- Short congratulations such as `恭喜`, `恭喜发财`, `Congratulations`, and
`Congrats`.
- Health, recovery, travel, safety, exam, competition, career, and general
good-luck wishes.
- Good-night and good-day wishes when they express a desired outcome, such as
`祝你好梦` or `Hope you have a wonderful day`.
- Religious or spiritual prayers directed toward a recipient.
- Wishes expressed for a third person, such as `我衷心祝愿她早日康复`.
## Negative boundaries
Label `blessing = false` when the text only:
- Requests, searches for, or discusses how to write a blessing.
- Thanks someone for a blessing already received.
- Mentions that other people sent blessings.
- Describes a celebration without wishing anyone well.
- Quotes or documents a blessing as an example.
- Greets someone without expressing a desired outcome.
- Gives positive feedback or praise without a wish or congratulation.
- Contains a lexical collision such as a title, name, product, or historical
reference that happens to include a blessing keyword.
## Context-dependent cases
Annotators must use surrounding context when available:
- Sarcastic congratulations are negative unless the product intentionally
treats the surface utterance as reply-worthy congratulations.
- `祝我好运` and equivalent self-directed wishes are positive.
- Conditional or unrealized intent such as `等会儿再祝他生日快乐` is negative
until the text actually expresses the wish.
- A message may be both `blessing` and another intent. For example, a wedding
invitation containing `祝你们幸福` is both an invitation and a blessing.
## Annotation process
1. Normalize only invisible whitespace; preserve wording, punctuation, and
emoji for labeling.
2. Two annotators label every golden-set record independently.
3. Disagreements are adjudicated by a third reviewer using this document.
4. Record the boundary category and adjudication reason, not only the binary
label.
5. Keep all evaluation examples isolated from generation prompts, training
sources, and active-learning exports.
## Release evaluation
The dedicated blessing benchmark must contain:
- At least 3,000 Chinese and 1,500 English human-reviewed records.
- Equal positive and hard-negative strata for diagnostic metrics.
- A separate natural-prevalence calibration set for threshold selection.
- At least 100 records for each major positive and negative boundary category.
The release gate is precision at least 0.95, recall at least 0.85, and F1 at
least 0.90 on the adjudicated benchmark, with no major boundary category below
0.80 F1.
ModelTraining/ClipboardSemantics/chinese-corpus-candidate-audit-v1.json
+235
View File
@@ -0,0 +1,235 @@
{
"schemaVersion": 1,
"auditVersion": "chinese-corpus-candidate-audit-v1",
"reviewedAt": "2026-08-28",
"taxonomyVersion": "clipboard-consensus-v6",
"scope": "面向简体中文、繁体中文及配套英文边界样本的 taxonomy 语料候选准入审计。",
"policy": {
"admittedMeaning": "准入仅允许来源进入候选抽取、映射和清洗队列，不代表已经进入商业训练集。",
"splitBoundary": "只允许上游官方明确标识的 train 数据产生训练候选；dev、validation、test、challenge、hidden test 和本地冻结 holdout 一律不得训练。",
"commercialBoundary": "导入前必须固定上游 revision、保存许可证及 attribution、确认数据本体而非仅代码受该许可证覆盖，并完成隐私、内容安全和重复检查。",
"knownLabelsBoundary": "每条记录只声明来源实际标注且经过确定性映射审计的 knownLabels；未声明字段保持 unknown，不得作为负例。",
"syntheticBoundary": "合成语料仅用于 train，必须记录生成与上游 provenance，sampleWeight 不得高于 0.35，不得进入校准、评估、人类 gold 或 policy anchors，也不得覆盖冲突的真人或许可非合成样本。",
"holdoutBoundary": "与任一冻结 holdout 精确或归一化近重复的文本全局禁止训练。"
},
"admittedSources": [
{
"id": "MASSIVE",
"status": "admitted",
"languages": [
"zh-CN",
"en-US"
],
"license": "CC-BY-4.0",
"commercialUse": "允许，须署名并保留许可证与修改说明。",
"url": "https://huggingface.co/datasets/AmazonScience/massive",
"licenseUrl": "https://huggingface.co/datasets/AmazonScience/massive/blob/main/LICENSE",
"trainBoundary": "仅固定 revision 的官方 train split；dev/test 禁止训练。",
"synthetic": false,
"reason": "覆盖中英文助手意图与多领域表达，标签结构适合映射 assistantCommand、informationQuery 和 domain；需避免把源 intent 扩展为其他 taxonomy 字段的负例。"
},
{
"id": "CrossWOZ",
"status": "admitted",
"languages": [
"zh-CN"
],
"license": "Apache-2.0",
"commercialUse": "允许，须保留许可证和 NOTICE 要求。",
"url": "https://github.com/thu-coai/CrossWOZ",
"licenseUrl": "https://github.com/thu-coai/CrossWOZ/blob/master/LICENSE",
"trainBoundary": "仅 data/crosswoz/train.json.zip 的固定官方 revision；val/test 禁止训练。",
"synthetic": false,
"reason": "中文跨领域任务对话可补充 travel、dining、calendar 等边界；只抽取当前轮可独立判断的文本，依赖隐藏对话状态的记录保持 unknown 或丢弃。"
},
{
"id": "BiToD",
"status": "admitted",
"languages": [
"zh-CN",
"en"
],
"license": "Apache-2.0",
"commercialUse": "允许，须保留许可证和 NOTICE 要求。",
"url": "https://github.com/HLTCHKUST/BiToD",
"licenseUrl": "https://github.com/HLTCHKUST/BiToD/blob/main/LICENSE",
"trainBoundary": "仅官方 zh_train/en_train 数据；valid/test 和 cross-lingual evaluation split 禁止训练。",
"synthetic": false,
"reason": "官方提供中英双语和明确切分，可用于 travel、dining、calendar 与 assistant 路由映射；只保留脱离上下文仍有稳定语义的轮次。"
},
{
"id": "MultiDoGO",
"status": "admitted",
"languages": [
"en"
],
"license": "CDLA-Permissive-1.0",
"commercialUse": "允许，按数据许可证保留来源和许可记录。",
"url": "https://github.com/awslabs/multi-domain-goal-oriented-dialogues-dataset",
"licenseUrl": "https://github.com/awslabs/multi-domain-goal-oriented-dialogues-dataset/blob/master/LICENSE.txt",
"trainBoundary": "仅 data/paper_splits 下官方 train.tsv；dev/test 禁止训练。",
"synthetic": false,
"reason": "金融、媒体、软件等域的 turn-level intent 可补充英文边界与跨语言对照；只映射明确源标签，不推断其余 intent。"
},
{
"id": "Taskmaster-1",
"status": "admitted",
"languages": [
"en"
],
"license": "CC-BY-4.0（数据；代码 Apache-2.0）",
"commercialUse": "允许，数据须署名并记录修改。",
"url": "https://github.com/google-research-datasets/Taskmaster/tree/master/TM-1-2019",
"licenseUrl": "https://creativecommons.org/licenses/by/4.0/legalcode",
"trainBoundary": "只使用官方训练用途文件并固定清单；任何官方 evaluation/test 文件禁止训练。",
"synthetic": false,
"reason": "可补充服务预订、计划和确认边界；Wizard/self-dialogue 风格需单独标记 provenance，并过滤依赖多轮上下文的 utterance。"
},
{
"id": "SNIPS",
"status": "admitted",
"languages": [
"en"
],
"license": "CC0-1.0",
"commercialUse": "允许；保留数据集来源、固定 revision 和修改记录。",
"url": "https://github.com/sonos/nlu-benchmark",
"licenseUrl": "https://github.com/sonos/nlu-benchmark/blob/master/LICENSE",
"trainBoundary": "仅 sonos/nlu-benchmark 固定 revision 的 train_*_full.json；validate 文件和第三方镜像不得导入。",
"synthetic": false,
"reason": "可提供 assistantCommand 与媒体、天气、smartHome 等清晰边界；准入不延伸到来源和许可不一致的第三方 SNIPS 镜像。"
},
{
"id": "MInDS-14",
"status": "admitted",
"languages": [
"zh-CN",
"en-US",
"en-GB",
"en-AU"
],
"license": "CC-BY-4.0",
"commercialUse": "允许，须署名并记录转录文本的处理。",
"url": "https://huggingface.co/datasets/PolyAI/minds14",
"licenseUrl": "https://creativecommons.org/licenses/by/4.0/legalcode",
"trainBoundary": "上游每个 config 仅发布 train；这些记录只能产生训练候选，不能充当 OSGKeyboard 校准或评估真值。",
"synthetic": false,
"reason": "中英文银行意图可补充 finance、accountService 和 informationQuery；只使用转录文本，音频不进入本项目。"
},
{
"id": "GoEmotions",
"status": "admitted",
"languages": [
"en"
],
"license": "Apache-2.0",
"commercialUse": "允许，须保留许可证和 NOTICE；导入时保存固定 revision 的许可证证据。",
"url": "https://github.com/google-research/google-research/tree/master/goemotions",
"licenseUrl": "https://github.com/google-research/google-research/blob/master/LICENSE",
"trainBoundary": "仅官方 train.tsv；dev/test 禁止训练。",
"synthetic": false,
"reason": "只用于 sentiment 的已审计映射，不从情绪标签推断 complaint、blessing 或 replyableMessage；Reddit 文本须经过隐私与内容安全过滤。"
},
{
"id": "ASAP",
"status": "admitted",
"languages": [
"zh-CN"
],
"license": "Apache-2.0",
"commercialUse": "允许，须保留许可证和 notices。",
"url": "https://github.com/Meituan-Dianping/ASAP",
"licenseUrl": "https://github.com/Meituan-Dianping/ASAP/blob/master/LICENSE",
"trainBoundary": "仅 data/train.csv；dev/test 禁止训练。",
"synthetic": false,
"reason": "中文餐饮评价可补充 dining、sentiment 与显式 complaint 边界；负面 aspect 或低评分不能自动映射为 complaint。"
},
{
"id": "Restaurant8k",
"status": "admitted",
"languages": [
"en"
],
"license": "CC-BY-4.0",
"commercialUse": "允许，须署名并记录修改。",
"url": "https://github.com/PolyAI-LDN/task-specific-datasets/tree/master/span_extraction/restaurant8k",
"licenseUrl": "https://github.com/PolyAI-LDN/task-specific-datasets/blob/master/LICENSE",
"trainBoundary": "仅 train_0.json 作为完整官方训练集；其下采样副本不得重复导入，test.json 禁止训练。",
"synthetic": false,
"reason": "可补充 dining domain 和餐饮查询边界；span 标注只使经审计映射的字段 known，不使十二个 intent 全部已知。"
},
{
"id": "FormosaNLU-Synth-v1",
"status": "admitted",
"languages": [
"zh-TW"
],
"license": "CC-BY-4.0（合成数据及 MASSIVE zh-TW seed）",
"commercialUse": "允许，须同时署名 MASSIVE 与 FormosaNLU Synth，并说明合成和过滤修改。",
"url": "https://huggingface.co/datasets/steven0226/formosa-nlu-synth-v1",
"licenseUrl": "https://huggingface.co/datasets/steven0226/formosa-nlu-synth-v1/blob/main/LICENSE",
"trainBoundary": "仅固定 release manifest 对应的 train；不得作为校准、评估或人工 gold。",
"synthetic": true,
"maximumSampleWeight": 0.35,
"reason": "可补充繁体中文和台湾表达，但必须保留双重 attribution、生成 provenance 与 synthetic 标记；只声明生成契约保证的 knownLabels。"
}
],
"quarantinedSources": [
{
"id": "BANKING77",
"status": "quarantined",
"license": "CC-BY-4.0",
"commercialUse": "许可证允许商业使用，但当前不准进入训练。",
"url": "https://huggingface.co/datasets/PolyAI/banking77",
"reason": "与 MInDS-14 的 finance/accountService 覆盖高度重叠，且大量短查询会放大 informationQuery 与低信息 fragment 偏差；待完成去重、源域平衡和独立边界审计。"
},
{
"id": "ABCD",
"status": "quarantined",
"license": "MIT",
"commercialUse": "许可证允许商业使用，但当前不准进入训练。",
"url": "https://github.com/asappresearch/abcd",
"reason": "虚构零售客服的多轮 action dialogue 强依赖角色与上下文，直接抽取会污染单条 clipboard 的 task、confirmationDecision 和 systemNotification 边界。"
},
{
"id": "MultiWOZ",
"status": "quarantined",
"license": "MIT（官方仓库）",
"commercialUse": "许可证允许商业使用，但当前不准进入训练。",
"url": "https://github.com/budzianowski/multiwoz",
"reason": "版本多、历史标注错误和修订差异明显，且多轮状态依赖强；在固定唯一版本、验证官方 train 和完成 turn-level 质量审计前隔离。"
},
{
"id": "CLINC150",
"status": "quarantined",
"license": "CC-BY-4.0（UCI 官方分发）",
"commercialUse": "许可证允许商业使用，但当前不准进入 v6 新训练。",
"url": "https://archive.ics.uci.edu/dataset/570/clinc150",
"reason": "以虚拟助手和 OOS 检测为目标，短命令分布会主导新增路由标签；历史实验及 holdout 已使用相关文本，须先完成全局泄漏审计和旧产物隔离。"
},
{
"id": "CFPB",
"status": "quarantined",
"license": "CC0-1.0",
"commercialUse": "许可证允许商业使用，但真实投诉叙述当前不准进入训练。",
"url": "https://www.consumerfinance.gov/data-research/consumer-complaints/",
"reason": "真实消费者投诉可能包含敏感财务、身份和叙事隐私信息，且 2026 年官方停止主动发布投诉 narratives；即使是 CC0，也需法律、隐私和历史快照来源审查。"
},
{
"id": "openclaw-zh-greetings",
"status": "quarantined",
"license": "MIT（数据卡声明）",
"commercialUse": "许可声明表面允许，但当前证据不足以批准商业训练。",
"url": "https://huggingface.co/datasets/trytax/openclaw-zh-greetings",
"reason": "小型非官方示例集缺少稳定上游、版本化生成过程和逐条权利链；仓库/软件的 MIT 许可不能替代对数据文本本体的 provenance 审计。"
},
{
"id": "LCCC",
"status": "quarantined",
"license": "MIT 仓库标识与官方“仅限科研用途”说明并存",
"commercialUse": "禁止用于当前商业训练。",
"url": "https://github.com/thu-coai/CDial-GPT",
"reason": "官方 README 明确限定科研用途，语料来自抓取的微博对话，缺少完整底层内容权利与隐私链；只能在隔离研究队列中审阅。"
}
]
}
ModelTraining/ClipboardSemantics/corpus-registry-sources.json
+302
View File
@@ -0,0 +1,302 @@
{
"schemaVersion": 2,
"policy": "Only license-safe sources may produce train candidates. Any text seen in calibration or evaluation data is globally barred from training.",
"intentLabels": [
"task",
"question",
"invitation",
"complaint",
"scheduleNegotiation",
"confirmationDecision",
"followUpReminder",
"blessing",
"replyableMessage",
"assistantCommand",
"informationQuery",
"systemNotification"
],
"domains": [
"finance",
"travel",
"calendar",
"communication",
"media",
"smartHome",
"shopping",
"dining",
"health",
"weather",
"accountService",
"generalKnowledge"
],
"legacyFullyKnownIntentLabels": [
"task",
"question",
"invitation",
"complaint",
"scheduleNegotiation",
"confirmationDecision",
"followUpReminder",
"blessing",
"replyableMessage"
],
"sources": [
{
"id": "historical-nine-model-v1",
"path": "${REPO_ROOT}/ModelTraining/ClipboardSemantics/Generated/historical-nine-model-v1.jsonl",
"license": "OSGKeyboard project license",
"defaultUse": "by-split",
"sourceType": "project-generated",
"allIntentLabelsKnown": true,
"knownIntentLabels": ["task", "question", "invitation", "complaint", "scheduleNegotiation", "confirmationDecision", "followUpReminder", "blessing", "replyableMessage"],
"sentimentKnown": true,
"weight": 1.0,
"required": true
},
{
"id": "current-product-corpus",
"path": "${REPO_ROOT}/ModelTraining/ClipboardSemantics/clipboard_semantic_corpus.jsonl",
"license": "OSGKeyboard project license",
"defaultUse": "by-split",
"sourceType": "project-generated",
"allIntentLabelsKnown": true,
"knownIntentLabels": ["task", "question", "invitation", "complaint", "scheduleNegotiation", "confirmationDecision", "followUpReminder", "blessing", "replyableMessage"],
"sentimentKnown": true,
"weight": 1.0,
"required": true
},
{
"id": "product-corpus-16x",
"path": "${REPO_ROOT}/ModelTraining/ClipboardSemantics/Generated/product-corpus-16x.jsonl",
"license": "OSGKeyboard project license",
"defaultUse": "by-split",
"sourceType": "project-generated",
"allIntentLabelsKnown": true,
"knownIntentLabels": ["task", "question", "invitation", "complaint", "scheduleNegotiation", "confirmationDecision", "followUpReminder", "blessing", "replyableMessage"],
"sentimentKnown": true,
"weight": 1.0,
"required": true
},
{
"id": "licensed-open-training",
"path": "${REPO_ROOT}/ModelTraining/ClipboardSemantics/open-training-corpus.jsonl",
"license": "mixed per-record commercial licenses",
"defaultUse": "train",
"sourceType": "licensed-open-data",
"allIntentLabelsKnown": false,
"sentimentKnown": false,
"weight": 1.0,
"required": true
},
{
"id": "v6-boundary-synthetic",
"path": "${REPO_ROOT}/ModelTraining/ClipboardSemantics/v6-boundary-training-supplement.jsonl",
"license": "OSGKeyboard project license",
"defaultUse": "train",
"sourceType": "project-generated-synthetic",
"allIntentLabelsKnown": false,
"sentimentKnown": false,
"weight": 1.0,
"required": true
},
{
"id": "v6-migration-tier-a",
"path": "${REPO_ROOT}/ModelTraining/ClipboardSemantics/CorpusRegistry/Labels/V6Migration/tier-a.jsonl",
"license": "OSGKeyboard project license",
"defaultUse": "train",
"sourceType": "model-consensus",
"allIntentLabelsKnown": false,
"sentimentKnown": false,
"weight": 1.0,
"required": true
},
{
"id": "v6-migration-tier-b",
"path": "${REPO_ROOT}/ModelTraining/ClipboardSemantics/CorpusRegistry/Labels/V6Migration/tier-b.jsonl",
"license": "OSGKeyboard project license",
"defaultUse": "train",
"sourceType": "model-consensus",
"allIntentLabelsKnown": false,
"sentimentKnown": false,
"weight": 1.0,
"required": true
},
{
"id": "v6-migration-adjudicated",
"path": "${REPO_ROOT}/ModelTraining/ClipboardSemantics/CorpusRegistry/Labels/V6Migration/adjudicated.jsonl",
"license": "OSGKeyboard project license",
"defaultUse": "train",
"sourceType": "model-adjudicated",
"allIntentLabelsKnown": false,
"sentimentKnown": false,
"weight": 1.0,
"required": true
},
{
"id": "v6-blind-evaluation",
"path": "${REPO_ROOT}/ModelTraining/ClipboardSemantics/v6-blind-evaluation-corpus.jsonl",
"license": "OSGKeyboard project license",
"defaultUse": "by-split",
"sourceType": "frozen-holdout",
"allIntentLabelsKnown": false,
"sentimentKnown": false,
"weight": 1.0,
"required": true
},
{
"id": "blessing-synthetic-100k",
"path": "${REPO_ROOT}/ModelTraining/ClipboardSemantics/Generated/blessing-synthetic-100k.jsonl",
"license": "OSGKeyboard project license",
"defaultUse": "train",
"sourceType": "project-generated",
"allIntentLabelsKnown": false,
"sentimentKnown": false,
"weight": 0.35,
"required": true
},
{
"id": "consensus-silver-v1",
"path": "${REPO_ROOT}/ModelTraining/ClipboardSemantics/Consensus/consensus-silver.jsonl",
"license": "Apache-2.0",
"defaultUse": "by-split",
"sourceType": "model-consensus",
"allIntentLabelsKnown": false,
"sentimentKnown": false,
"weight": 1.0,
"required": false
},
{
"id": "random-holdout",
"path": "${REPO_ROOT}/ModelTraining/ClipboardSemantics/random-holdout-corpus.jsonl",
"license": "OSGKeyboard project license",
"defaultUse": "evaluation-only",
"sourceType": "frozen-holdout",
"allIntentLabelsKnown": true,
"knownIntentLabels": ["task", "question", "invitation", "complaint", "scheduleNegotiation", "confirmationDecision", "followUpReminder", "blessing", "replyableMessage"],
"sentimentKnown": true,
"weight": 1.0,
"required": true
},
{
"id": "fresh-metric-holdout",
"path": "${REPO_ROOT}/ModelTraining/ClipboardSemantics/fresh-metric-holdout-corpus.jsonl",
"license": "OSGKeyboard project license",
"defaultUse": "evaluation-only",
"sourceType": "frozen-holdout",
"allIntentLabelsKnown": true,
"knownIntentLabels": ["task", "question", "invitation", "complaint", "scheduleNegotiation", "confirmationDecision", "followUpReminder", "blessing", "replyableMessage"],
"sentimentKnown": true,
"weight": 1.0,
"required": true
},
{
"id": "targeted-confirmation-holdout",
"path": "${REPO_ROOT}/ModelTraining/ClipboardSemantics/targeted-confirmation-holdout-corpus.jsonl",
"license": "OSGKeyboard project license",
"defaultUse": "evaluation-only",
"sourceType": "frozen-holdout",
"allIntentLabelsKnown": true,
"knownIntentLabels": ["task", "question", "invitation", "complaint", "scheduleNegotiation", "confirmationDecision", "followUpReminder", "blessing", "replyableMessage"],
"sentimentKnown": true,
"weight": 1.0,
"required": true
},
{
"id": "targeted-final-holdout",
"path": "${REPO_ROOT}/ModelTraining/ClipboardSemantics/targeted-final-holdout-corpus.jsonl",
"license": "OSGKeyboard project license",
"defaultUse": "evaluation-only",
"sourceType": "frozen-holdout",
"allIntentLabelsKnown": true,
"knownIntentLabels": ["task", "question", "invitation", "complaint", "scheduleNegotiation", "confirmationDecision", "followUpReminder", "blessing", "replyableMessage"],
"sentimentKnown": true,
"weight": 1.0,
"required": true
},
{
"id": "targeted-release-holdout",
"path": "${REPO_ROOT}/ModelTraining/ClipboardSemantics/targeted-release-holdout-corpus.jsonl",
"license": "OSGKeyboard project license",
"defaultUse": "evaluation-only",
"sourceType": "frozen-holdout",
"allIntentLabelsKnown": true,
"knownIntentLabels": ["task", "question", "invitation", "complaint", "scheduleNegotiation", "confirmationDecision", "followUpReminder", "blessing", "replyableMessage"],
"sentimentKnown": true,
"weight": 1.0,
"required": true
},
{
"id": "online-real-holdout",
"path": "${REPO_ROOT}/ModelTraining/ClipboardSemantics/online-real-holdout-corpus.jsonl",
"license": "evaluation-only mixed provenance",
"defaultUse": "evaluation-only",
"sourceType": "frozen-holdout",
"allIntentLabelsKnown": true,
"knownIntentLabels": ["task", "question", "invitation", "complaint", "scheduleNegotiation", "confirmationDecision", "followUpReminder", "blessing", "replyableMessage"],
"sentimentKnown": true,
"weight": 1.0,
"required": true
},
{
"id": "comprehensive-online-holdout",
"path": "${REPO_ROOT}/ModelTraining/ClipboardSemantics/comprehensive-online-holdout-corpus.jsonl",
"license": "evaluation-only mixed provenance",
"defaultUse": "evaluation-only",
"sourceType": "frozen-holdout",
"allIntentLabelsKnown": true,
"knownIntentLabels": ["task", "question", "invitation", "complaint", "scheduleNegotiation", "confirmationDecision", "followUpReminder", "blessing", "replyableMessage"],
"sentimentKnown": true,
"weight": 1.0,
"required": true
},
{
"id": "blessing-benchmark-review",
"path": "${REPO_ROOT}/ModelTraining/ClipboardSemantics/BlessingBenchmark/review-queue.jsonl",
"license": "evaluation-only mixed provenance",
"defaultUse": "evaluation-only",
"sourceType": "blind-review-benchmark",
"allIntentLabelsKnown": false,
"sentimentKnown": false,
"weight": 1.0,
"sensitive": true,
"required": true
},
{
"id": "blessing-unseen-scenarios",
"path": "${REPO_ROOT}/OSGKeyboardTests/Fixtures/BlessingUnseenScenarios.json",
"license": "OSGKeyboard project license",
"defaultUse": "evaluation-only",
"sourceType": "frozen-holdout",
"allIntentLabelsKnown": false,
"sentimentKnown": false,
"weight": 1.0,
"required": false
}
],
"excludedSources": [
{
"id": "LCCC",
"reason": "Research-only terms and incomplete content-rights/privacy chain. It must not influence production corpus generation or training."
},
{
"id": "DailyDialog-EmpatheticDialogues-Switchboard",
"reason": "Non-commercial license restrictions."
},
{
"id": "CPED",
"reason": "Repository license does not establish commercial rights to the underlying television dialogue."
},
{
"id": "Tianji-Wishes-Birthday-Quotes",
"reason": "No sufficiently clear per-record source or generation-rights chain."
},
{
"id": "CFPB",
"reason": "No official train split; privacy-sensitive narratives overlap the frozen evaluation provenance."
},
{
"id": "CLINC150",
"reason": "Isolated from product training because its assistant taxonomy is not product-policy compatible."
}
]
}
ModelTraining/ClipboardSemantics/v6-blind-evaluation-report.json
+99
View File
@@ -0,0 +1,99 @@
{
"domains": {
"accountService": 14,
"calendar": 8,
"communication": 14,
"dining": 10,
"finance": 5,
"generalKnowledge": 10,
"health": 5,
"media": 1,
"shopping": 6,
"smartHome": 3,
"travel": 6,
"unknown": 38
},
"humanCoverage": {
"excludeDeviceCommandRecords": 11,
"overriddenRecords": 49,
"overridesByField": {
"ambiguous": 49,
"question": 49,
"replyableMessage": 49,
"task": 49
},
"records": 60
},
"inputSHA256": {
"holdout": "95ac25ff01113de05eb3c6b9dd9a7b38efa3fa1d434439097f604bb9f2de1a9b",
"humanLabels": "d6f5897227991a59420d2892e0d946c4fb4125d1cab37f4137deb3e1f3d1f688",
"primary:composer": "6221c8f6fa7a2fed7bb8c8d93f70eee1c1017f6a1dbf75487f90cb9789165644",
"primary:grok": "940ede21bfaaf30db36e931135cbb196240cdf96ff1eff36dbf490156738cb94",
"primary:luna": "63c525c05f146546603610bcb04d229d8ee415259aa5a7bdacb277715aef289b",
"reviewQueue": "7dd224c20dd1ee1da4c8ead6f19f99264fcb5a868e3588432b77ce233c6f500e",
"reviewer:claude": "fe34fea28628f5f84b41c838bfc59591feabd5851a21ba8e08d828811f637b26",
"reviewer:sol": "78f869bcd685cea9c579d488e5f588f21d67a1db1ca4fdf13d3ccb43713b1b9d"
},
"knownByField": {
"assistantCommand": 111,
"blessing": 120,
"complaint": 118,
"confirmationDecision": 113,
"domain": 82,
"followUpReminder": 120,
"informationQuery": 115,
"invitation": 120,
"question": 107,
"replyableMessage": 100,
"scheduleNegotiation": 112,
"sentiment": 107,
"systemNotification": 118,
"task": 104
},
"languages": {
"en": 60,
"zh-Hans": 60
},
"outputSHA256": "c800922eb24e9681e34d16d4328ffbdbfc3ba8a0e6b2538ea670d85dc3cf6b03",
"positiveByIntent": {
"assistantCommand": 6,
"blessing": 0,
"complaint": 4,
"confirmationDecision": 7,
"followUpReminder": 2,
"informationQuery": 10,
"invitation": 6,
"question": 23,
"replyableMessage": 63,
"scheduleNegotiation": 2,
"systemNotification": 8,
"task": 25
},
"recordCount": 120,
"schemaVersion": 1,
"sourceDataset": "product-policy-blind-holdout-v1",
"sourceLicense": "OSGKeyboard project license",
"sourceRevision": "v1",
"splits": {
"golden": 40,
"test": 40,
"validation": 40
},
"unresolvedByField": {
"ambiguous": 2,
"assistantCommand": 9,
"blessing": 0,
"complaint": 2,
"confirmationDecision": 7,
"domain": 38,
"followUpReminder": 0,
"informationQuery": 5,
"invitation": 0,
"question": 13,
"replyableMessage": 20,
"scheduleNegotiation": 8,
"sentiment": 13,
"systemNotification": 2,
"task": 16
}
}
ModelTraining/ClipboardSemantics/v6-boundary-training-supplement-summary.json
+121
View File
@@ -0,0 +1,121 @@
{
"corpusSHA256": "5b435a564e5dae111898f2ac743c9fa6cc862525e9c0d45c7358069b7567297f",
"counts": {
"byBoundaryTargetAndLanguage": {
"en": {
"assistantCommand": 1000,
"informationQuery": 1000,
"question": 200,
"replyableMessage": 200,
"systemNotification": 1000,
"task": 200
},
"zh-Hans": {
"assistantCommand": 1000,
"informationQuery": 1000,
"question": 200,
"replyableMessage": 200,
"systemNotification": 1000,
"task": 200
}
},
"byDomain": {
"accountService": 600,
"calendar": 600,
"communication": 600,
"dining": 600,
"finance": 600,
"generalKnowledge": 600,
"health": 600,
"media": 600,
"shopping": 600,
"smartHome": 600,
"travel": 600,
"weather": 600
},
"byIntent": {
"assistantCommand": 2000,
"informationQuery": 2000,
"question": 400,
"replyableMessage": 800,
"systemNotification": 2000,
"task": 400
},
"byIntentAndLanguage": {
"en": {
"assistantCommand": 1000,
"informationQuery": 1000,
"question": 200,
"replyableMessage": 400,
"systemNotification": 1000,
"task": 200
},
"zh-Hans": {
"assistantCommand": 1000,
"informationQuery": 1000,
"question": 200,
"replyableMessage": 400,
"systemNotification": 1000,
"task": 200
}
},
"byLanguage": {
"en": 3600,
"zh-Hans": 3600
},
"byTemplateFamily": {
"v6_assistantCommand_en_template_1": 253,
"v6_assistantCommand_en_template_2": 233,
"v6_assistantCommand_en_template_3": 250,
"v6_assistantCommand_en_template_4": 264,
"v6_assistantCommand_zh-Hans_template_1": 247,
"v6_assistantCommand_zh-Hans_template_2": 250,
"v6_assistantCommand_zh-Hans_template_3": 253,
"v6_assistantCommand_zh-Hans_template_4": 250,
"v6_informationQuery_en_template_1": 243,
"v6_informationQuery_en_template_2": 240,
"v6_informationQuery_en_template_3": 256,
"v6_informationQuery_en_template_4": 261,
"v6_informationQuery_zh-Hans_template_1": 246,
"v6_informationQuery_zh-Hans_template_2": 254,
"v6_informationQuery_zh-Hans_template_3": 245,
"v6_informationQuery_zh-Hans_template_4": 255,
"v6_question_en_template_1": 52,
"v6_question_en_template_2": 54,
"v6_question_en_template_3": 44,
"v6_question_en_template_4": 50,
"v6_question_zh-Hans_template_1": 54,
"v6_question_zh-Hans_template_2": 42,
"v6_question_zh-Hans_template_3": 46,
"v6_question_zh-Hans_template_4": 58,
"v6_replyableMessage_en_template_1": 57,
"v6_replyableMessage_en_template_2": 40,
"v6_replyableMessage_en_template_3": 53,
"v6_replyableMessage_en_template_4": 50,
"v6_replyableMessage_zh-Hans_template_1": 51,
"v6_replyableMessage_zh-Hans_template_2": 46,
"v6_replyableMessage_zh-Hans_template_3": 50,
"v6_replyableMessage_zh-Hans_template_4": 53,
"v6_systemNotification_en_template_1": 243,
"v6_systemNotification_en_template_2": 250,
"v6_systemNotification_en_template_3": 261,
"v6_systemNotification_en_template_4": 246,
"v6_systemNotification_zh-Hans_template_1": 252,
"v6_systemNotification_zh-Hans_template_2": 238,
"v6_systemNotification_zh-Hans_template_3": 245,
"v6_systemNotification_zh-Hans_template_4": 265,
"v6_task_en_template_1": 38,
"v6_task_en_template_2": 51,
"v6_task_en_template_3": 60,
"v6_task_en_template_4": 51,
"v6_task_zh-Hans_template_1": 41,
"v6_task_zh-Hans_template_2": 52,
"v6_task_zh-Hans_template_3": 48,
"v6_task_zh-Hans_template_4": 59
}
},
"excludedHoldoutOverlap": 0,
"recordCount": 7200,
"schemaVersion": 1,
"sourceRevision": "v6-boundary-templates-1"
}
ModelTraining/ClipboardSemantics/v6-release-gate-report.json
+111
View File
@@ -0,0 +1,111 @@
{
"allGatesPassed": false,
"candidate": "taxonomy-v6-expanded-maxEnt",
"corpus": {
"blindLabelPolicy": "Product-owner task/question/replyable labels are used for the first 60 records; other v6 fields require per-field multi-model consensus. Unknown fields are excluded.",
"blindRecords": 120,
"candidateCorpusRecords": 273746,
"humanLabeledBlindRecords": 60,
"registryCanonicalRecords": 327337,
"trainCandidates": 273626
},
"currentModelBlindBaseline": {
"accuracy": 0.8845,
"f1": 0.2326,
"precision": 0.2578,
"recall": 0.2974
},
"deploymentMode": "shadow/display",
"domainGoldenMetrics": {
"accuracy": 0.3704,
"macroF1": 0.3282,
"total": 27
},
"gates": {
"domainMacroF1": {
"actual": 0.3282,
"passed": false,
"required": 0.85
},
"evaluationIsolation": {
"exactOverlapCount": 0,
"passed": true,
"trainBarredByCalibrationCount": 1,
"trainBarredByEvaluationCount": 0
},
"newIntentMacroF1": {
"actual": 0.5915,
"passed": false,
"required": 0.9
},
"newIntentMinimumPrecision": {
"actual": 0.25,
"passed": false,
"required": 0.95
},
"oldNineNoRegression": {
"currentBlindMacroF1": 0.2326,
"passed": true,
"reason": "The candidate is additive and the production manifest and nine deployed model files were not replaced."
},
"runtimePerformance": {
"actual": {
"coldLoadMilliseconds": 49.2974,
"modelBytes": 570083,
"peakRSSDeltaBytes": 25001984,
"warmP95Milliseconds": 0.2498
},
"budgets": {
"coldLoadMilliseconds": 100,
"modelBytes": 2000000,
"peakRSSDeltaBytes": 41943040,
"warmP95Milliseconds": 1
},
"passed": true
}
},
"limitations": [
"Only 60 of 120 product blind records have product-owner labels.",
"The remaining fields are high-confidence model consensus, not human gold.",
"Per-language calibration has too few positive blind examples.",
"The current model has no heads for the three new intents or domain."
],
"newIntentGoldenMetrics": {
"assistantCommand": {
"accuracy": 0.9722,
"f1": 0.8,
"falseNegative": 0,
"falsePositive": 1,
"precision": 0.6667,
"recall": 1,
"total": 36,
"trueNegative": 33,
"truePositive": 2
},
"informationQuery": {
"accuracy": 0.7692,
"f1": 0.3077,
"falseNegative": 3,
"falsePositive": 6,
"precision": 0.25,
"recall": 0.4,
"total": 39,
"trueNegative": 28,
"truePositive": 2
},
"systemNotification": {
"accuracy": 0.95,
"f1": 0.6667,
"falseNegative": 2,
"falsePositive": 0,
"precision": 1,
"recall": 0.5,
"total": 40,
"trueNegative": 36,
"truePositive": 2
}
},
"productionManifestSHA256": "52cf3916ea18325fc1f4b51f3c1ee2d9f73c5633b227d3355606f5a1a1ea0c1c",
"releaseDecision": "keep-current-model",
"schemaVersion": 1
}