diff --git a/CHANGELOG.md b/CHANGELOG.md index 96abd58..5121fbc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Expanded clipboard semantics**: add fully local bilingual models for schedule negotiation, confirmed decisions, and follow-up reminders, with language-specific confidence thresholds, existing-skill routing, and reproducible random holdout evaluation. / **扩展剪贴板语义**：新增完全本地的中英双语日程协商、确认决策与跟进提醒模型，并加入分语言置信度阈值、现有技能路由及可复现的随机留出评估。 ### Changed +- **Smart reply center**: merge invitation, task, blessing, clarification, and empathy actions into Reply; local scene detection now offers three fixed, clearly labeled decisions whenever user intent must not be guessed, while ordinary chats retain optional tone choices. / **智能回复中心**：将邀约、任务、祝福、澄清与共情操作统一到“回复”；本地场景识别会在不可替用户猜测意图时始终提供三种固定且清晰标注的立场，普通聊天则继续使用可选的语气候选。 - **Clipboard intent precision**: specialize the local task and complaint models, add conservative language thresholds, and suppress complaint-only text from task routing unless it contains an explicit assignment. / **剪贴板意图精度**：专项优化本地任务与投诉模型，加入保守的分语言阈值，并在没有明确委派时阻止纯投诉文本进入任务路由。 - **Precision-first semantic routing**: add traceable multi-model consensus labels, local joint intent verifiers, confidence-margin abstention, layered source-isolated evaluation, and privacy-safe shadow disagreement counters while retaining current models until every acceptance gate passes. / **精度优先语义路由**：新增可追溯多模型共识银标、本地联合意图验证器、置信度与差值拒识、按来源隔离的分层评估及隐私安全的影子分歧计数；所有验收门槛通过前继续保留当前模型。 - **Iterative semantic research**: add reproducible twenty-round weakly supervised model selection, high-confidence self-training consensus, per-intent fine-tuning, explicit-evidence experiments, frozen-overlap gates, and macOS replay manifests; retain deployed models when every candidate underperforms release baselines. / **迭代语义研究**：新增可复现的二十轮弱监督模型选择、高置信自训练共识、分意图微调、显式证据实验、冻结集重叠门禁及 macOS 重放清单；所有候选低于发布基线时继续保留现有部署模型。 @@ -24,6 +25,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Personal style threshold**: allow generating a learned speaking style after 2,500 effective dictation characters instead of 5,000. / **专属风格门槛**：生成学习型说话风格所需的有效听写字符由 5,000 降至 2,500。 ### Fixed +- **Personal style generation**: derive every reviewed prompt through two-stage corpus evidence, apply concrete low-confidence ASR tendencies instead of replacing them with a neutral template, recover wrapped model JSON once, keep one provider configuration and a 45-second budget across both stages, and distinguish cancellation from timeout. / **专属风格生成**：每个待审阅 Prompt 均通过两阶段语料证据生成，并在证据较少时仍应用原始 ASR 中具体的低置信度表达倾向，而非替换为中性模板；同时支持一次模型 JSON 包装恢复，在两阶段固定同一服务配置与 45 秒预算，并区分主动取消和请求超时。 - **Durable Apple account sessions**: persist refresh operation identifiers before token rotation, reliably store and validate Apple user identifiers, require one-time reauthentication for unverifiable legacy sessions, and prevent failed Keychain deletion or concurrent account work from restoring a revoked login. / **可靠 Apple 账号会话**：在令牌轮换前持久化刷新操作标识，可靠存储并验证 Apple 用户标识，为无法验证的旧会话执行一次重新认证，并防止 Keychain 删除失败或并发账号操作恢复已撤销的登录态。 - **Account sign-in presentation**: use the concise reward message consistently and keep a uniform app surface beneath translucent Apple authorization sheets. / **账号登录呈现**：统一使用精简的积分奖励文案，并在半透明 Apple 授权弹窗下保持一致的 App 背景。 - **Recommended hotword details**: label Home recommendations by their actual source, render suggestion chips in adaptive monochrome, remove internal usage counts from dictionary rows, and anchor the iPhone confirmation popover to the selected hotword. / **推荐热词详情**：按真实来源标记首页推荐词，以自适应黑白配色显示推荐标签，移除词库列表中的内部使用次数，并让 iPhone 确认弹窗指向所选热词。 diff --git a/ModelTraining/ClipboardSemantics/README.md b/ModelTraining/ClipboardSemantics/README.md index 2cd5a71..ff37a07 100644 --- a/ModelTraining/ClipboardSemantics/README.md +++ b/ModelTraining/ClipboardSemantics/README.md @@ -26,6 +26,45 @@ unsupported labels, insufficient bilingual golden coverage, and content that resembles direct contact data or credentials. No user clipboard content is included. +## Taxonomy v6 candidate contract + +The deployed models above remain the historical nine-intent suite. The next +corpus taxonomy is defined by +`Consensus/labeling-instructions-v6.md`: it retains `task`, `question`, +`invitation`, `complaint`, `scheduleNegotiation`, `confirmationDecision`, +`followUpReminder`, `blessing`, and `replyableMessage`, then adds +`assistantCommand`, `informationQuery`, and `systemNotification`. Assistant, +search, and machine-notification text is therefore preserved as explicit +routing data instead of being flattened into legacy negatives or discarded. + +Every record also receives one primary domain or `unknown`: `finance`, +`travel`, `calendar`, `communication`, `media`, `smartHome`, `shopping`, +`dining`, `health`, `weather`, `accountService`, or `generalKnowledge`. +`Consensus/adjudication-instructions-v5.md` defines evidence-based resolution +for the expanded fields while retaining the existing queue format. + +Intent values remain three-state: `true`, `false`, or `unknown`. +`knownLabels` lists only fields the source actually annotates after an audited +mapping; an absent field is unknown and contributes neither a positive nor a +negative training example. External sources may produce candidates only from a +pinned upstream official `train` split under documented commercial-use terms. +Upstream dev/validation/test data and every local frozen holdout are barred from +training, including normalized near-duplicates. + +`chinese-corpus-candidate-audit-v1.json` records the first versioned source +decision. MASSIVE, CrossWOZ, BiToD, MultiDoGO, Taskmaster-1, SNIPS, MInDS-14, +GoEmotions, ASAP, Restaurant8k, and FormosaNLU are admitted to the candidate +audit pipeline subject to their per-source conditions. BANKING77, ABCD, +MultiWOZ, CLINC150, CFPB, `openclaw-zh-greetings`, and LCCC remain quarantined. +Admission is not automatic commercial training approval: revision pinning, +license evidence, attribution, privacy/content review, deterministic mapping, +and holdout deduplication still apply. + +Synthetic records are train-only, retain generation and upstream provenance, +use a sample weight no greater than `0.25`, and declare only contractually known +fields. They never enter calibration, evaluation, human gold, or policy-anchor +sets and cannot override conflicting human or licensed non-synthetic evidence. + ## Reproduce ```bash @@ -62,7 +101,8 @@ python3 Scripts/clipboard_semantics/apply_deployment_policy.py \ The reproducible open-data supplement uses official training splits from MASSIVE, CrossWOZ, GoEmotions, MultiDoGO, Taskmaster-1, CLINC150, CFPB, -and ASAP: +and ASAP. It also includes the MIT-licensed `openclaw-zh-greetings` labels and +the pinned MIT blessing templates from `SWHL/WeChat-AutoSendBless`: ```bash python3 Scripts/clipboard_semantics/generate_open_training_corpus.py @@ -84,11 +124,145 @@ CPED is also excluded because the repository license does not establish commercial rights to the underlying television dialogue; synthetic blessing datasets without a clear per-record rights chain are excluded as well. +## Broad blessing supplement + +`blessing-labeling-guidelines.md` defines the broad product boundary. The +dedicated generator creates equal numbers of positive examples and difficult +negatives such as greeting-only, blessing requests, received thanks, +celebration mentions, quotations, reported wishes, and sarcasm: + +```bash +python3 Scripts/clipboard_semantics/generate_blessing_training_corpus.py \ + --records-per-language 50000 +``` + +Every generated record declares only `blessing` in `knownLabels`, so unknown +clipboard intents are not treated as false. The generator rejects normalized +base-corpus and frozen-holdout overlap, duplicate text, and common PII shapes. + +LCCC may be mined only as an isolated research queue. Although its dataset card +declares MIT, the official CDial-GPT README limits the dataset and pretrained +models to research use. Its source is crawled Weibo dialogue without a complete +underlying content-rights or privacy chain. Download LCCC-base from the official +CDial-GPT link or the `silver/lccc` Hugging Face mirror, then run: + +```bash +python3 Scripts/clipboard_semantics/extract_lccc_blessing_candidates.py \ + /path/to/LCCC-base.zip \ + --output /tmp/lccc-blessing-candidates.jsonl \ + --manifest /tmp/lccc-blessing-candidates-manifest.json +``` + +The resulting unreviewed records are research-only and must not be merged into +a commercial training corpus without legal, privacy, and manual label review. + +## Blessing benchmark review + +Prepare a blind queue with 3,000 Chinese and 1,500 English records from the +frozen comprehensive holdout: + +```bash +python3 Scripts/clipboard_semantics/prepare_blessing_benchmark.py +``` + +Two different people independently complete `annotator-a.jsonl` and +`annotator-b.jsonl` without seeing `sealed-provenance.jsonl`. Finalization is +strict: incomplete labels, duplicate IDs, reused annotator identity, and +unadjudicated disagreement are fatal. + +```bash +python3 Scripts/clipboard_semantics/finalize_blessing_benchmark.py \ + --annotator-a-id reviewer-a \ + --annotator-b-id reviewer-b +``` + +`BlessingBenchmark/README.md` defines the stable positive and negative boundary +categories. The finalized calibration/test benchmark remains evaluation-only +and must never enter a training corpus. + ## Consensus silver data and joint verifiers -The precision-first pipeline prepares a license-traceable queue from public -training data, accepts only three-model agreement or source-supported -two-of-three agreement, and keeps conflicts out of training: +The corpus registry combines the preserved historical product corpus, current +and scaled product generators, licensed open data, and project-owned blessing +data without flattening provenance: + +```bash +python3 Scripts/clipboard_semantics/build_corpus_registry.py +``` + +`corpus-registry-sources.json` is the source-of-truth inventory. Exact duplicate +texts retain every source claim, while any occurrence in calibration or frozen +evaluation data globally bars that text from training. Labels use three states: +`true`, `false`, and `unknown`; a missing source annotation is never converted +to a negative label. LCCC and other research-only or unclear-rights sources are +explicitly excluded. + +The v2 blind-labeling pilot samples 1,000 distinct near-duplicate clusters. +Three primary models label every record without seeing source labels. Any +non-unanimous record is sent, still blind, to two review models: + +```bash +python3 Scripts/clipboard_semantics/merge_consensus_labels_v2.py \ + prepare-review \ + --queue ModelTraining/ClipboardSemantics/CorpusRegistry/labeling-pilot.jsonl \ + --primary sol=/path/to/primary-sol.jsonl \ + --primary grok=/path/to/primary-grok.jsonl \ + --primary codex=/path/to/primary-codex.jsonl \ + --output /path/to/review-queue.jsonl \ + --report /path/to/review-report.json +``` + +Primary unanimity produces Tier A silver data. Reviewed records require at +least four of five votes for every intent and sentiment field to produce Tier B +silver data at lower training weight. Remaining conflicts, ambiguity, and +positive quoted/meta cases enter a human adjudication queue and never train +automatically. `Consensus/labeling-instructions-v2.md` defines the shared +taxonomy and strict output schema. + +The frozen 2026-08-28 pilot used Sol, Grok, and Luna as primary labelers, then +Composer and Claude for blind conflict review. Of 1,000 +records, 81 reached Tier A, 290 reached Tier B, and 629 entered human review. +The 0.80 per-language/per-field kappa gate failed, so this pilot is not eligible +for automatic scale-up or model training. `replyableMessage`, `task`, +`question`, and Chinese coordination/blessing boundaries require adjudication +and instruction refinement first. + +The 629 unresolved records were then reviewed independently by Claude and Grok +using `Consensus/adjudication-instructions-v1.md`. A record reaches Tier C only +when both adjudicators select the same non-unknown value for every unresolved +field, quote exact supporting text, and report confidence of at least 0.90. +Only 21 records passed; 608 remain unresolved. Tier C keeps a 0.35 sample weight +and remains silver data rather than human gold. + +The 608-record remainder was then re-adjudicated with Sol 5.6 and Grok 4.6 +using the product-approved boundaries in +`Consensus/labeling-instructions-v3.md` and the disposition-aware schema in +`Consensus/adjudication-instructions-v2.md`. The approved rules exclude clear +device/assistant commands, separate invitation questions from information +questions, treat self-reminders as follow-up only, treat first-person needs as +implicit tasks, require explicit dissatisfaction for complaints, and require +explicit wishes or congratulations for blessings. At the unchanged 0.90 +two-model confidence gate, all seven policy-sensitive intent fields were +rechecked even when the old panel had agreed on them. In the final result, 202 +additional records reached Tier C, 109 clear device/assistant commands were +isolated, and 297 remained unresolved. A +deterministic language/field/severity-stratified sample of 60 records is the +human product-policy acceptance set; the new silver data must not be promoted +until that sample reaches 95% accuracy. The original per-language/per-field +kappa gate still applies before full-corpus labeling can scale up. + +The product owner subsequently labeled 30 high-information anchors for +`replyableMessage`, `task`, and `question`. These decisions are frozen in +`Consensus/product-policy-anchors-v1.json`, and the clarified taxonomy is +documented in `Consensus/labeling-instructions-v5.md`. On a blind replay, Grok +4.6 matched all 63 scored target-field decisions, while Luna 5.6 matched 62 of +63 (98.41%); both pass the 95% target-intent gate. Sol 5.6 was rejected for this +role because it forced low-information fragments into negative labels. The +target fields are eligible for focused re-adjudication, but the broader corpus +is still blocked by the independent ambiguity/disposition and kappa gates. + +The original verifier pipeline remains available for reproducing its earlier +three-model study: ```bash python3 Scripts/clipboard_semantics/generate_consensus_labels.py prepare @@ -165,6 +339,28 @@ memory checks still require macOS with Xcode 26+. A surrogate result can nominate a data/threshold policy for macOS replay, but cannot authorize automatic deployment. +## Taxonomy v6 candidate result + +The 2026-08-28 v6 run integrated 31,087 license-reviewed open-training +records, 7,200 low-weight bilingual boundary records, and 99 high-confidence +relabels from the former assistant-command exclusion queue. The registry +contains 327,337 canonical records and emits 273,626 train candidates. The +frozen 120-record bilingual product holdout has zero exact overlap with +training. + +The additive maxEnt candidate trained `assistantCommand`, `informationQuery`, +`systemNotification`, and the 12-way `domain` classifier. Runtime performance +passed: the four models total 570,083 bytes, load in 49.30 ms, have 0.250 ms +maximum warm p95 latency, and add 25,001,984 bytes of peak RSS in the macOS +benchmark process. + +Quality did not pass. Golden new-intent macro F1 is 0.5915, minimum intent +precision is 0.2500, and domain macro F1 is 0.3282, below the 0.90, 0.95, and +0.85 release gates. Only 60 blind records have product-owner labels; the +remaining v6 fields use per-field model consensus and are not human gold. +Therefore the candidate remains staging-only and the deployed models are not +replaced. See `v6-release-gate-report.json` for the machine-readable decision. + ## Deployment decision Only maxEnt models are trained and deployed because they are self-contained in @@ -172,11 +368,13 @@ the keyboard extension. Create ML BERT transfer models depend on `NLContextualEmbedding` assets that are not guaranteed to exist in a simulator or keyboard-extension runtime. -The deployed manifest remains schema version 2 until a verifier passes its +The deployed manifest remains schema version 2 until a candidate passes its frozen acceptance gates. Schema version 3 adds optional joint verifiers, per-language confidence thresholds, top-1/top-2 margins, and an explicit -`shadow` or `automatic` deployment mode. Consumers fall back to the current -binary routing behavior when verifiers are absent. +`shadow` or `automatic` deployment mode. Schema version 4 adds the three +display-only public intent heads and optional domain classifier. Consumers +accept schema versions 1 through 4 and fall back to the current binary routing +behavior when newer classifiers are absent. All ten deployed models pass the golden precision gate after deployment policy is applied. The preserved six-model and first nine-model reports are diff --git a/ModelTraining/ClipboardSemantics/open-training-sources.json b/ModelTraining/ClipboardSemantics/open-training-sources.json index 025d266..743dd04 100644 --- a/ModelTraining/ClipboardSemantics/open-training-sources.json +++ b/ModelTraining/ClipboardSemantics/open-training-sources.json @@ -1,9 +1,9 @@ { "baseCorpusRecords": 17692, - "combinedCorpusSHA256": "f111bc7425912213082e01038b52dd3d25fe21418d156f532da8f01f27fa16e4", - "combinedRecords": 21537, + "combinedCorpusSHA256": "24bbec3fb9b3cc6434582d660b03b37401abe1339185405115646675ec3b654a", + "combinedRecords": 48779, "excludedBaseOverlap": 6, - "excludedHoldoutOverlap": 72, + "excludedHoldoutOverlap": 425, "excludedSources": [ { "dataset": "MIDAS", @@ -36,60 +36,136 @@ { "dataset": "Verified Enron Intent / GitHub issue holdout source", "reason": "No clean official train split independent from the frozen holdout." + }, + { + "dataset": "CFPB", + "reason": "No official train split; the source is already represented in frozen evaluation data and contains privacy-sensitive narratives." + }, + { + "dataset": "CLINC150", + "reason": "Kept isolated from product training because its assistant intents are not aligned with the product-policy taxonomy." + }, + { + "dataset": "openclaw-zh-greetings / WeChat-AutoSendBless", + "reason": "Repository licenses do not establish a sufficiently clear, versioned rights chain for the underlying message templates." } ], - "openTrainingRecords": 3845, + "openTrainingRecords": 31087, "policy": "Licensed official training splits only. Per-record knownLabels prevent unannotated intents from becoming false negatives. Exact normalized text overlap with every local *holdout-corpus.jsonl file is excluded.", "schemaVersion": 1, "seed": 20260827, "sources": [ { - "dataset": "CrossWOZ", + "dataset": "AmazonScience/MASSIVE en-US", "knownLabelCounts": { - "question": 279, - "task": 258 + "assistantCommand": 3658, + "domain": 9824, + "informationQuery": 5307, + "sentiment": 126, + "task": 358 }, "languages": { - "zh-Hans": 544 + "en": 10810 + }, + "license": "CC-BY-4.0", + "records": 10810, + "revision": "1.1", + "url": "https://huggingface.co/datasets/AmazonScience/massive" + }, + { + "dataset": "AmazonScience/MASSIVE zh-CN", + "knownLabelCounts": { + "assistantCommand": 3272, + "domain": 9166, + "informationQuery": 5025, + "sentiment": 120, + "task": 348 + }, + "languages": { + "zh-Hans": 10099 + }, + "license": "CC-BY-4.0", + "records": 10099, + "revision": "1.1", + "url": "https://huggingface.co/datasets/AmazonScience/massive" + }, + { + "dataset": "CrossWOZ", + "knownLabelCounts": { + "domain": 476, + "informationQuery": 393 + }, + "languages": { + "zh-Hans": 599 }, "license": "Apache-2.0", - "records": 544, + "records": 599, "revision": "df82c9fdff91b9b130f2d6b89110d3870ba6260e", "url": "https://github.com/thu-coai/CrossWOZ/blob/df82c9fdff91b9b130f2d6b89110d3870ba6260e/data/crosswoz/train.json.zip" }, { - "dataset": "Google Research GoEmotions", + "dataset": "FormosaNLU Synth v1", "knownLabelCounts": { - "sentiment": 798 + "assistantCommand": 839, + "domain": 1747, + "informationQuery": 817, + "task": 119 }, "languages": { - "en": 1198 + "zh-Hans": 1988 + }, + "license": "CC-BY-4.0", + "records": 1988, + "revision": "03a337b61a200ab690994dca4dc31aa7f209800e", + "url": "https://huggingface.co/datasets/steven0226/formosa-nlu-synth-v1/resolve/03a337b61a200ab690994dca4dc31aa7f209800e/data/train.jsonl" + }, + { + "dataset": "Google Research GoEmotions", + "knownLabelCounts": { + "sentiment": 799 + }, + "languages": { + "en": 1199 }, "license": "Apache-2.0", - "records": 1198, + "records": 1199, "revision": "5d8f4ac97c873bde3a792ba4628f00bb9103d3e6", "url": "https://github.com/google-research/google-research/blob/5d8f4ac97c873bde3a792ba4628f00bb9103d3e6/goemotions/data/train.tsv" }, { "dataset": "Google Taskmaster-1", "knownLabelCounts": { - "confirmationDecision": 168, - "question": 138, - "scheduleNegotiation": 4, - "task": 90 + "domain": 631 }, "languages": { - "en": 638 + "en": 631 }, "license": "CC-BY-4.0", - "records": 638, + "records": 631, "revision": "d92cb6af3005f1dc09c39e75e7daf4a04905e00b", "url": "https://github.com/google-research-datasets/Taskmaster/tree/d92cb6af3005f1dc09c39e75e7daf4a04905e00b/TM-1-2019" }, + { + "dataset": "HLTCHKUST/BiToD", + "knownLabelCounts": { + "domain": 1339, + "informationQuery": 885, + "task": 355 + }, + "languages": { + "en": 681, + "zh-Hans": 658 + }, + "license": "Apache-2.0", + "records": 1339, + "revision": "a9bd74de9eecdc3d875cb4ebf6a6beaf9c30c2ff", + "url": "https://raw.githubusercontent.com/HLTCHKUST/BiToD/a9bd74de9eecdc3d875cb4ebf6a6beaf9c30c2ff/data/zh_train.json" + }, { "dataset": "Meituan-Dianping/ASAP", "knownLabelCounts": { "complaint": 500, + "domain": 1000, "sentiment": 1000 }, "languages": { @@ -103,31 +179,62 @@ { "dataset": "MultiDoGO", "knownLabelCounts": { - "confirmationDecision": 27, - "question": 83, - "task": 222 + "domain": 464, + "informationQuery": 61, + "task": 83 }, "languages": { - "en": 465 + "en": 464 }, "license": "CDLA-Permissive-1.0", - "records": 465, + "records": 464, "revision": "baa30639c4b271f394b81443c842193407cdf26d", - "url": "https://github.com/awslabs/multi-domain-goal-oriented-dialogues-dataset/blob/baa30639c4b271f394b81443c842193407cdf26d/data/paper_splits/splits_annotated_at_turn_level/software/train.tsv" + "url": "https://github.com/awslabs/multi-domain-goal-oriented-dialogues-dataset/blob/baa30639c4b271f394b81443c842193407cdf26d/data/paper_splits/splits_annotated_at_turn_level/airline/train.tsv" + }, + { + "dataset": "PolyAI MInDS-14 zh-CN", + "knownLabelCounts": { + "assistantCommand": 163, + "domain": 480, + "informationQuery": 179 + }, + "languages": { + "zh-Hans": 480 + }, + "license": "CC-BY-4.0", + "records": 480, + "revision": "40ce77cb32a384e4d50a568e1ec39ac804019d33", + "url": "https://huggingface.co/datasets/PolyAI/minds14/resolve/40ce77cb32a384e4d50a568e1ec39ac804019d33/zh-CN/train-00000-of-00001.parquet" + }, + { + "dataset": "PolyAI RESTAURANTS-8K", + "knownLabelCounts": { + "domain": 986 + }, + "languages": { + "en": 986 + }, + "license": "CC-BY-4.0", + "records": 986, + "revision": "57ec275d8078af65b7731c2a98be812d844a6d6b", + "url": "https://raw.githubusercontent.com/PolyAI-LDN/task-specific-datasets/57ec275d8078af65b7731c2a98be812d844a6d6b/span_extraction/restaurant8k/train_0.json" + }, + { + "dataset": "SNIPS NLU Benchmark", + "knownLabelCounts": { + "assistantCommand": 499, + "domain": 1492, + "informationQuery": 743, + "task": 250 + }, + "languages": { + "en": 1492 + }, + "license": "CC0-1.0", + "records": 1492, + "revision": "b86ac7f1577868c42158d0dec77db50956046696", + "url": "https://raw.githubusercontent.com/sonos/nlu-benchmark/b86ac7f1577868c42158d0dec77db50956046696/2017-06-custom-intent-engines/BookRestaurant/train_BookRestaurant_full.json" } ], - "unavailableSources": [ - { - "dataset": "MASSIVE", - "reason": "URLError: " - }, - { - "dataset": "CLINC150", - "reason": "URLError: " - }, - { - "dataset": "CFPB", - "reason": "URLError: " - } - ] + "unavailableSources": [] }