feat(app): add crash visibility, memory relief, and clipboard reply work
Squashes the working tree that had accumulated on cloud-agent. Two strands of work, kept in one commit at the maintainer's request. Crash and stability (this session): - Subscribe to MetricKit so crash, hang, CPU, and disk-write diagnostics are recorded on device. Keyboard-extension memory kills are the most likely way a user loses the keyboard mid-sentence and iOS does not classify them as crashes, so they never reach App Store Connect. - Upload those reports to POST /v1/analytics/diagnostics from internal builds only (local Debug and TestFlight), reusing the analytics installation identifier and skipping entirely when analytics are off. App Store builds keep everything on device. - Shed the Chinese engine, English lexicon, and in-flight pipeline work at the 40 MiB and 48 MiB budget thresholds instead of waiting for a system memory warning that jetsam often preempts. - Install audio taps with the input node's live format through one shared adaptive resampler, so an audio-route change mid-recording no longer raises an uncatchable format-mismatch exception. Covers the dictation preview and the Mac recorder, which still used the pattern the Flow capture path had already moved away from. - Harden the librime bridge: never pass a nil string across a nonnull boundary when a user dictionary holds malformed UTF-8, null-check the engine API vector, and distrust a candidate count without its array. - Guard the expanded candidate grid against a list that shrinks while the collection view is still laying out against the previous count. - Resolve localized text without the App Group so the screen shown when that container is missing renders its message instead of crashing on launch, which is the one situation it exists to handle. - Gate CI on the App Group and Keychain wiring that the keyboard extension hard-depends on, with a self-test proving the check fails. Clipboard, reply, and semantics (pre-existing working tree): - Unify the reply center, refresh keyboard AI features, and add native event and reminder exporters. - Expand clipboard semantic routing with authored corpora and a product-blind holdout, plus the scripts that build and validate them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
Executable
+264
@@ -0,0 +1,264 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Validate the App Group / Keychain wiring that the app hard-depends on.
|
||||
|
||||
Why this is a CI gate and not a unit test: `AppGroup.defaults`,
|
||||
`AppGroupStore`, `FlowSessionBridge` and `FlowSessionPolicy` all call
|
||||
`fatalError` when the App Group container is missing, by design — a silent
|
||||
`.standard` fallback would desync the keyboard extension from the host app and
|
||||
produce "I set my API key and nothing happens". The cost of that design is that
|
||||
a single typo or a dropped entitlement turns into a keyboard that crashes on
|
||||
every presentation for every user. Nothing else catches it: the failure needs a
|
||||
signed build on a real device.
|
||||
|
||||
Checks (all static, no Xcode required):
|
||||
|
||||
1. `AppGroup.identifier` in Swift matches `com.apple.security.application-groups`
|
||||
in every entitlements file that ships an App Group, and in `project.yml`.
|
||||
2. Both the iOS app and the keyboard extension declare the App Group — they
|
||||
are the two processes that must share one suite.
|
||||
3. `keychain-access-groups` lists `$(AppIdentifierPrefix)com.osgkeyboard.shared`
|
||||
FIRST in every target. `Keychain.swift` deliberately relies on the default
|
||||
access group (the first entry) instead of passing `kSecAttrAccessGroup`,
|
||||
so a reorder silently moves every stored API key to a different store.
|
||||
4. `project.yml` and the checked-in `.entitlements` files agree. XcodeGen
|
||||
regenerates the latter from the former, so a drifted `.entitlements` means
|
||||
someone hand-edited the generated output.
|
||||
|
||||
Run: python3 Scripts/check_app_group_wiring.py
|
||||
python3 Scripts/check_app_group_wiring.py --self-test
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import plistlib
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
REPO = Path(__file__).resolve().parent.parent
|
||||
|
||||
APP_GROUP_SWIFT = REPO / "OSGKeyboardShared/Constants/AppGroup.swift"
|
||||
PROJECT_YML = REPO / "project.yml"
|
||||
|
||||
# target label -> (entitlements path, must declare an App Group)
|
||||
TARGETS = {
|
||||
"OSGKeyboard (iOS app)": ("OSGKeyboard/OSGKeyboard.entitlements", True),
|
||||
"OSGKeyboardExt (keyboard)": ("OSGKeyboardExt/OSGKeyboardExt.entitlements", True),
|
||||
"OSGKeyboardMac (menu bar)": ("OSGKeyboardMac/OSGKeyboardMac.entitlements", False),
|
||||
}
|
||||
|
||||
APP_GROUP_KEY = "com.apple.security.application-groups"
|
||||
KEYCHAIN_KEY = "keychain-access-groups"
|
||||
EXPECTED_DEFAULT_KEYCHAIN_GROUP = "$(AppIdentifierPrefix)com.osgkeyboard.shared"
|
||||
|
||||
errors: list[str] = []
|
||||
|
||||
|
||||
def fail(message: str) -> None:
|
||||
errors.append(message)
|
||||
|
||||
|
||||
def swift_app_group_identifier() -> str | None:
|
||||
text = APP_GROUP_SWIFT.read_text(encoding="utf-8")
|
||||
match = re.search(
|
||||
r'public\s+static\s+let\s+identifier\s*=\s*"([^"]+)"', text
|
||||
)
|
||||
if not match:
|
||||
fail(f"{APP_GROUP_SWIFT.relative_to(REPO)}: could not find `AppGroup.identifier`")
|
||||
return None
|
||||
return match.group(1)
|
||||
|
||||
|
||||
def load_entitlements(path: Path) -> dict | None:
|
||||
if not path.exists():
|
||||
fail(f"{path.relative_to(REPO)}: entitlements file is missing")
|
||||
return None
|
||||
try:
|
||||
with path.open("rb") as handle:
|
||||
return plistlib.load(handle)
|
||||
except Exception as error: # noqa: BLE001 - surfaced verbatim to the developer
|
||||
fail(f"{path.relative_to(REPO)}: not a readable plist ({error})")
|
||||
return None
|
||||
|
||||
|
||||
def project_yml_blocks() -> dict[str, str]:
|
||||
"""Split `project.yml` into one text block per `entitlements:` stanza."""
|
||||
text = PROJECT_YML.read_text(encoding="utf-8")
|
||||
blocks: dict[str, str] = {}
|
||||
for match in re.finditer(
|
||||
r"^ entitlements:\n path:\s*(\S+)\n(.*?)(?=^ \w|\Z)",
|
||||
text,
|
||||
re.MULTILINE | re.DOTALL,
|
||||
):
|
||||
blocks[match.group(1)] = match.group(2)
|
||||
return blocks
|
||||
|
||||
|
||||
def yaml_list_after(block: str, key: str) -> list[str] | None:
|
||||
"""Reads a simple `key:` / `- value` list out of one entitlements stanza."""
|
||||
pattern = re.compile(rf"^\s*{re.escape(key)}:\s*\n((?:\s*(?:#.*)?\n|\s*-\s*\S.*\n)+)", re.MULTILINE)
|
||||
match = pattern.search(block)
|
||||
if not match:
|
||||
return None
|
||||
values = []
|
||||
for line in match.group(1).splitlines():
|
||||
stripped = line.strip()
|
||||
if not stripped or stripped.startswith("#"):
|
||||
continue
|
||||
if not stripped.startswith("- "):
|
||||
break
|
||||
values.append(stripped[2:].strip())
|
||||
return values
|
||||
|
||||
|
||||
def main() -> int:
|
||||
identifier = swift_app_group_identifier()
|
||||
if identifier is None:
|
||||
print("\n".join(f"✗ {error}" for error in errors))
|
||||
return 1
|
||||
|
||||
print(f"AppGroup.identifier = {identifier}")
|
||||
|
||||
yml_blocks = project_yml_blocks()
|
||||
|
||||
for label, (relative_path, requires_app_group) in TARGETS.items():
|
||||
before = len(errors)
|
||||
path = REPO / relative_path
|
||||
plist = load_entitlements(path)
|
||||
if plist is None:
|
||||
continue
|
||||
|
||||
groups = plist.get(APP_GROUP_KEY)
|
||||
if requires_app_group:
|
||||
if not groups:
|
||||
fail(f"{label}: entitlements declare no `{APP_GROUP_KEY}`")
|
||||
elif identifier not in groups:
|
||||
fail(
|
||||
f"{label}: entitlements declare {groups!r}, "
|
||||
f"which does not contain AppGroup.identifier ({identifier!r})"
|
||||
)
|
||||
|
||||
keychain = plist.get(KEYCHAIN_KEY)
|
||||
if not keychain:
|
||||
fail(f"{label}: entitlements declare no `{KEYCHAIN_KEY}`")
|
||||
elif keychain[0] != EXPECTED_DEFAULT_KEYCHAIN_GROUP:
|
||||
fail(
|
||||
f"{label}: `{KEYCHAIN_KEY}` starts with {keychain[0]!r}; "
|
||||
f"Keychain.swift relies on {EXPECTED_DEFAULT_KEYCHAIN_GROUP!r} "
|
||||
"being the FIRST (default) entry"
|
||||
)
|
||||
|
||||
block = yml_blocks.get(relative_path)
|
||||
if block is None:
|
||||
fail(f"{label}: project.yml has no `entitlements: path: {relative_path}` stanza")
|
||||
print(f" ✗ {label}")
|
||||
continue
|
||||
|
||||
yml_groups = yaml_list_after(block, APP_GROUP_KEY)
|
||||
if requires_app_group and (yml_groups or []) != (groups or []):
|
||||
fail(
|
||||
f"{label}: project.yml `{APP_GROUP_KEY}` ({yml_groups!r}) "
|
||||
f"disagrees with the generated entitlements ({groups!r})"
|
||||
)
|
||||
|
||||
yml_keychain = yaml_list_after(block, KEYCHAIN_KEY)
|
||||
if (yml_keychain or []) != (keychain or []):
|
||||
fail(
|
||||
f"{label}: project.yml `{KEYCHAIN_KEY}` ({yml_keychain!r}) "
|
||||
f"disagrees with the generated entitlements ({keychain!r})"
|
||||
)
|
||||
|
||||
if len(errors) == before:
|
||||
print(f" ✓ {label}")
|
||||
else:
|
||||
print(f" ✗ {label}")
|
||||
|
||||
if errors:
|
||||
print()
|
||||
for error in errors:
|
||||
print(f"✗ {error}")
|
||||
print(
|
||||
"\nThe App Group is a hard dependency: the keyboard extension calls "
|
||||
"fatalError() when its container is missing, so a mismatch here ships "
|
||||
"a keyboard that crashes on every presentation."
|
||||
)
|
||||
return 1
|
||||
|
||||
print("\nApp Group and Keychain wiring OK.")
|
||||
return 0
|
||||
|
||||
|
||||
def self_test() -> int:
|
||||
"""Proves the guard fails on the two regressions it exists to catch.
|
||||
|
||||
A checker that cannot fail is worse than no checker — it reads green
|
||||
forever while the thing it guards rots.
|
||||
"""
|
||||
global REPO, APP_GROUP_SWIFT, PROJECT_YML, errors
|
||||
|
||||
original_repo = REPO
|
||||
cases = [
|
||||
(
|
||||
"App Group typo in the keyboard extension",
|
||||
"OSGKeyboardExt/OSGKeyboardExt.entitlements",
|
||||
lambda text: text.replace(
|
||||
"group.com.osgkeyboard.shared", "group.com.osgkeyboard.shard"
|
||||
),
|
||||
),
|
||||
(
|
||||
"keychain-access-groups reordered in the iOS app",
|
||||
"OSGKeyboard/OSGKeyboard.entitlements",
|
||||
lambda text: text.replace(
|
||||
"<string>$(AppIdentifierPrefix)com.osgkeyboard.shared</string>\n"
|
||||
"\t\t<string>$(AppIdentifierPrefix)com.osgkeyboard.ios</string>",
|
||||
"<string>$(AppIdentifierPrefix)com.osgkeyboard.ios</string>\n"
|
||||
"\t\t<string>$(AppIdentifierPrefix)com.osgkeyboard.shared</string>",
|
||||
),
|
||||
),
|
||||
]
|
||||
|
||||
failures = 0
|
||||
for label, relative_path, mutate in cases:
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
sandbox = Path(temporary) / "repo"
|
||||
shutil.copytree(
|
||||
original_repo,
|
||||
sandbox,
|
||||
symlinks=True,
|
||||
ignore=shutil.ignore_patterns(
|
||||
".git", "build", "DerivedData", "ThirdParty", ".claude"
|
||||
),
|
||||
)
|
||||
target = sandbox / relative_path
|
||||
mutated = mutate(target.read_text(encoding="utf-8"))
|
||||
if mutated == target.read_text(encoding="utf-8"):
|
||||
print(f"✗ self-test could not apply the mutation for: {label}")
|
||||
failures += 1
|
||||
continue
|
||||
target.write_text(mutated, encoding="utf-8")
|
||||
|
||||
REPO = sandbox
|
||||
APP_GROUP_SWIFT = sandbox / "OSGKeyboardShared/Constants/AppGroup.swift"
|
||||
PROJECT_YML = sandbox / "project.yml"
|
||||
errors = []
|
||||
code = main()
|
||||
REPO = original_repo
|
||||
APP_GROUP_SWIFT = original_repo / "OSGKeyboardShared/Constants/AppGroup.swift"
|
||||
PROJECT_YML = original_repo / "project.yml"
|
||||
errors = []
|
||||
|
||||
if code == 0:
|
||||
print(f"✗ self-test: guard did NOT catch: {label}")
|
||||
failures += 1
|
||||
else:
|
||||
print(f"✓ self-test: guard catches: {label}")
|
||||
|
||||
return 1 if failures else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if "--self-test" in sys.argv[1:]:
|
||||
sys.exit(self_test())
|
||||
sys.exit(main())
|
||||
@@ -57,10 +57,61 @@ compare_pair "$APP_ZH_KEYS" "$APP_EN_KEYS" "App Localizable zh-Hans vs en"
|
||||
compare_pair "$EXT_EN_KEYS" "$EXT_ZH_KEYS" "Extension Keyboard en vs zh-Hans"
|
||||
compare_pair "$EXT_ZH_KEYS" "$EXT_EN_KEYS" "Extension Keyboard zh-Hans vs en"
|
||||
|
||||
# --- Host-app bare LocalizedStringKey check -------------------------------
|
||||
#
|
||||
# SwiftUI resolves a bare `Text("some.key")` through `Bundle.main`, whose
|
||||
# localization is fixed at launch from the *system* language. That silently
|
||||
# ignores the in-app language override (Settings → App Language), so such a
|
||||
# call site renders the system language no matter what the user picked.
|
||||
# `.environment(\.locale, …)` does NOT fix this — it only drives formatting.
|
||||
#
|
||||
# The host app must therefore go through `AppL10n.string` / `AppL10n.format`,
|
||||
# which resolve the `.lproj` sub-bundle manually. The keyboard extension and
|
||||
# the Mac app already do this via `ExtL10n` / `MacL10n`.
|
||||
KEY_RE='"[A-Za-z][A-Za-z0-9_]*(\.[A-Za-z0-9_+-]+)+"'
|
||||
BARE_RE="(\\.(navigationTitle|navigationBarTitle|alert|confirmationDialog|help|accessibilityLabel|accessibilityHint)|[^A-Za-z0-9_](Text|Button|Section|Label|Toggle|TextField|Picker|Stepper|Link|NavigationLink))\\([[:space:]]*${KEY_RE}"
|
||||
|
||||
bare_hits="$(
|
||||
grep -rnE "$BARE_RE|LocalizedStringKey\\([[:space:]]*${KEY_RE}" \
|
||||
--include='*.swift' "$ROOT/OSGKeyboard" 2>/dev/null \
|
||||
| grep -v '/Tests/' \
|
||||
| grep -vE '^[^:]+:[0-9]+:[[:space:]]*//' \
|
||||
|| true
|
||||
)"
|
||||
|
||||
if [[ -n "$bare_hits" ]]; then
|
||||
echo "❌ Host app uses bare LocalizedStringKey — these ignore the in-app language override:"
|
||||
echo "$bare_hits" | sed "s|^$ROOT/||" | sed 's/^/ /'
|
||||
echo ""
|
||||
echo " Wrap the key: Text(\"foo.bar\") → Text(AppL10n.string(\"foo.bar\"))"
|
||||
fail=1
|
||||
fi
|
||||
|
||||
# The host app must not reintroduce `LocalizedStringKey` at all: any value of
|
||||
# that type is resolved by SwiftUI through `Bundle.main` and so ignores the
|
||||
# in-app language override. Keys travel as `String` and are resolved at the
|
||||
# render site with `AppL10n.string`.
|
||||
lsk_hits="$(
|
||||
grep -rn 'LocalizedStringKey' --include='*.swift' "$ROOT/OSGKeyboard" 2>/dev/null \
|
||||
| grep -v '/Tests/' \
|
||||
| grep -vE '^[^:]+:[0-9]+:[[:space:]]*//' \
|
||||
|| true
|
||||
)"
|
||||
|
||||
if [[ -n "$lsk_hits" ]]; then
|
||||
echo "❌ Host app reintroduced LocalizedStringKey — it resolves via Bundle.main (system language):"
|
||||
echo "$lsk_hits" | sed "s|^$ROOT/||" | sed 's/^/ /'
|
||||
echo ""
|
||||
echo " Store the key as String and resolve at render: Text(AppL10n.string(key))"
|
||||
fail=1
|
||||
fi
|
||||
|
||||
if [[ "$fail" -ne 0 ]]; then
|
||||
echo ""
|
||||
echo "L10n key parity check failed."
|
||||
echo "L10n check failed."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "✅ L10n key parity check passed (Shared / App / Extension en ↔ zh-Hans)."
|
||||
echo "✅ Host app has no bare LocalizedStringKey call sites."
|
||||
echo "✅ Host app does not use LocalizedStringKey."
|
||||
|
||||
@@ -0,0 +1,219 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Split authored batches into train and evaluation corpora by scenario family.
|
||||
|
||||
Whole families move together. A family is one scenario (for example
|
||||
"invitation to a meal" or "received thanks for a blessing"), so a family-level
|
||||
split forces the evaluation set to test scenarios the model never trained on
|
||||
instead of paraphrases of them. Text overlap between the two sides is therefore
|
||||
impossible by construction, and the script verifies it anyway.
|
||||
|
||||
Positive and negative families are assigned separately so both sides keep a
|
||||
usable label balance.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import unicodedata
|
||||
from collections import Counter, defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
SEED = 20260903
|
||||
AUTHORED_DIRECTORY = Path("ModelTraining/ClipboardSemantics/Authored")
|
||||
INTENT_FIELDS = (
|
||||
"task",
|
||||
"question",
|
||||
"invitation",
|
||||
"complaint",
|
||||
"scheduleNegotiation",
|
||||
"confirmationDecision",
|
||||
"followUpReminder",
|
||||
"blessing",
|
||||
"replyable",
|
||||
)
|
||||
SOURCE_DATASET = "authored-v1"
|
||||
SOURCE_LICENSE = "OSGKeyboard project license"
|
||||
|
||||
|
||||
def parse_arguments() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("batches", nargs="*", type=Path)
|
||||
parser.add_argument("--seed", type=int, default=SEED)
|
||||
parser.add_argument(
|
||||
"--evaluation-family-fraction",
|
||||
type=float,
|
||||
default=0.30,
|
||||
help="Share of families in each polarity group reserved for evaluation.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--sample-weight",
|
||||
type=float,
|
||||
default=1.0,
|
||||
help="Training sample weight recorded on every authored train record.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--train-output",
|
||||
type=Path,
|
||||
default=AUTHORED_DIRECTORY / "authored-train-corpus.jsonl",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--evaluation-output",
|
||||
type=Path,
|
||||
default=AUTHORED_DIRECTORY / "authored-eval-corpus.jsonl",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--report",
|
||||
type=Path,
|
||||
default=AUTHORED_DIRECTORY / "authored-split-report.json",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def normalized_text(value: str) -> str:
|
||||
return " ".join(unicodedata.normalize("NFKC", value).split()).strip()
|
||||
|
||||
|
||||
def fingerprint(value: str) -> str:
|
||||
return normalized_text(value).casefold()
|
||||
|
||||
|
||||
def load_records(path: Path) -> list[dict]:
|
||||
return [
|
||||
json.loads(line)
|
||||
for line in path.read_text(encoding="utf-8").splitlines()
|
||||
if line.strip()
|
||||
]
|
||||
|
||||
|
||||
def polarity_group(family: str) -> str:
|
||||
"""Negative families carry a `neg` marker in their leading token."""
|
||||
|
||||
return "negative" if "neg_" in family else "positive"
|
||||
|
||||
|
||||
def intent_group(family: str) -> str:
|
||||
return family.split("_", 1)[0].removesuffix("neg")
|
||||
|
||||
|
||||
def family_priority(family: str, seed: int) -> str:
|
||||
return hashlib.sha256(f"{seed}|{family}".encode()).hexdigest()
|
||||
|
||||
|
||||
def assign_families(families: set[str], seed: int, fraction: float) -> dict[str, str]:
|
||||
"""Reserve a deterministic share of each intent/polarity cell for evaluation."""
|
||||
|
||||
cells: dict[tuple[str, str], list[str]] = defaultdict(list)
|
||||
for family in families:
|
||||
cells[(intent_group(family), polarity_group(family))].append(family)
|
||||
|
||||
assignment: dict[str, str] = {}
|
||||
for cell, values in sorted(cells.items()):
|
||||
ordered = sorted(values, key=lambda value: family_priority(value, seed))
|
||||
reserved = max(1, round(len(ordered) * fraction))
|
||||
for index, family in enumerate(ordered):
|
||||
assignment[family] = "evaluation" if index < reserved else "train"
|
||||
return assignment
|
||||
|
||||
|
||||
def main() -> None:
|
||||
arguments = parse_arguments()
|
||||
batches = arguments.batches or sorted(
|
||||
path
|
||||
for path in AUTHORED_DIRECTORY.glob("*.jsonl")
|
||||
if not path.name.startswith("authored-")
|
||||
)
|
||||
records: list[dict] = []
|
||||
for path in batches:
|
||||
records.extend(load_records(path))
|
||||
|
||||
assignment = assign_families(
|
||||
{record["family"] for record in records},
|
||||
arguments.seed,
|
||||
arguments.evaluation_family_fraction,
|
||||
)
|
||||
|
||||
train: list[dict] = []
|
||||
evaluation: list[dict] = []
|
||||
for record in records:
|
||||
target = assignment[record["family"]]
|
||||
emitted = {
|
||||
"id": record["id"],
|
||||
"text": normalized_text(record["text"]),
|
||||
"language": record["language"],
|
||||
"family": record["family"],
|
||||
"sentiment": record["sentiment"],
|
||||
**{field: record[field] for field in INTENT_FIELDS},
|
||||
"sourceDataset": SOURCE_DATASET,
|
||||
"sourceLicense": SOURCE_LICENSE,
|
||||
"provenance": "llm-authored",
|
||||
"knownLabels": list(INTENT_FIELDS),
|
||||
}
|
||||
if target == "train":
|
||||
emitted["split"] = "train"
|
||||
emitted["sampleWeight"] = arguments.sample_weight
|
||||
train.append(emitted)
|
||||
else:
|
||||
emitted["split"] = "authoredEvaluation"
|
||||
evaluation.append(emitted)
|
||||
|
||||
train_texts = {fingerprint(record["text"]) for record in train}
|
||||
evaluation_texts = {fingerprint(record["text"]) for record in evaluation}
|
||||
leaked = sorted(train_texts & evaluation_texts)
|
||||
|
||||
for path, values in (
|
||||
(arguments.train_output, train),
|
||||
(arguments.evaluation_output, evaluation),
|
||||
):
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(
|
||||
"\n".join(
|
||||
json.dumps(value, ensure_ascii=False, sort_keys=True)
|
||||
for value in values
|
||||
)
|
||||
+ "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
def positives(values: list[dict]) -> dict:
|
||||
return {
|
||||
field: sum(1 for value in values if value[field] is True)
|
||||
for field in INTENT_FIELDS
|
||||
}
|
||||
|
||||
report = {
|
||||
"schemaVersion": 1,
|
||||
"seed": arguments.seed,
|
||||
"batches": [str(path) for path in batches],
|
||||
"evaluationFamilyFraction": arguments.evaluation_family_fraction,
|
||||
"sampleWeight": arguments.sample_weight,
|
||||
"families": len(assignment),
|
||||
"familyAssignment": dict(sorted(assignment.items())),
|
||||
"train": {
|
||||
"records": len(train),
|
||||
"families": len({value["family"] for value in train}),
|
||||
"languages": dict(Counter(value["language"] for value in train)),
|
||||
"positives": positives(train),
|
||||
},
|
||||
"evaluation": {
|
||||
"records": len(evaluation),
|
||||
"families": len({value["family"] for value in evaluation}),
|
||||
"languages": dict(Counter(value["language"] for value in evaluation)),
|
||||
"positives": positives(evaluation),
|
||||
},
|
||||
"validation": {
|
||||
"trainEvaluationTextOverlap": len(leaked),
|
||||
"leakedExamples": leaked[:5],
|
||||
},
|
||||
}
|
||||
arguments.report.write_text(
|
||||
json.dumps(report, ensure_ascii=False, indent=2, sort_keys=True) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
print(json.dumps(report, ensure_ascii=False, indent=2, sort_keys=True))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,387 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Prepare the v2 product blind holdout with stratified, IPW-correctable sampling.
|
||||
|
||||
The v1 blind holdout has 120 records and as few as two positives per intent, so
|
||||
it cannot separate a real quality regression from sampling noise. This script
|
||||
builds a larger queue from real, clipboard-shaped licensed text.
|
||||
|
||||
Every eligible pool record is assigned to exactly one stratum by a rarest-first
|
||||
intent priority, so the design is a clean stratified sample: each emitted record
|
||||
carries `inclusionProbability` = selected/available for its stratum, and
|
||||
prevalence-corrected metrics follow from Horvitz-Thompson weighting. Rare
|
||||
intents are oversampled for statistical power without biasing reported rates.
|
||||
|
||||
Weak source labels drive stratification only. They are written to the sealed
|
||||
provenance file and never to the blind queue.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
import unicodedata
|
||||
from collections import Counter, defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
SEED = 20260902
|
||||
MODEL_TRAINING = Path("ModelTraining/ClipboardSemantics")
|
||||
OUTPUT_DIRECTORY = MODEL_TRAINING / "ProductBlindHoldoutV2"
|
||||
DEFAULT_SOURCES = (
|
||||
MODEL_TRAINING / "comprehensive-online-holdout-corpus.jsonl",
|
||||
MODEL_TRAINING / "online-real-holdout-corpus.jsonl",
|
||||
)
|
||||
|
||||
# Intents in rarest-first order. A record joins the first stratum it matches,
|
||||
# which is what gives the scarce intents their own guaranteed sampling cells.
|
||||
INTENT_PRIORITY = (
|
||||
"blessing",
|
||||
"confirmationDecision",
|
||||
"scheduleNegotiation",
|
||||
"invitation",
|
||||
"followUpReminder",
|
||||
"complaint",
|
||||
"task",
|
||||
"question",
|
||||
)
|
||||
REPLYABLE_STRATUM = "replyableOnly"
|
||||
NEGATIVE_STRATUM = "noWeakIntent"
|
||||
LANGUAGES = ("en", "zh-Hans")
|
||||
|
||||
# Families that are real text but not clipboard-shaped. CPED is television
|
||||
# dialogue and GoEmotions is short Reddit reaction text; both are dominated by
|
||||
# sub-15-character conversational turns that no user copies to a clipboard.
|
||||
EXCLUDED_FAMILY_PREFIXES = ("online_cped_", "online_goemotions_")
|
||||
|
||||
PII_PATTERN = re.compile(
|
||||
r"(?:[\w.+-]+@[\w.-]+\.\w+)|(?:\+?\d[\d ()-]{8,}\d)|"
|
||||
r"(?:\b\d{3}-\d{2}-\d{4}\b)",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
INTENT_FIELDS = (*INTENT_PRIORITY, "replyable")
|
||||
|
||||
|
||||
def parse_arguments() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--source", action="append", type=Path, default=[])
|
||||
parser.add_argument("--output-directory", type=Path, default=OUTPUT_DIRECTORY)
|
||||
parser.add_argument("--seed", type=int, default=SEED)
|
||||
parser.add_argument(
|
||||
"--minimum-characters",
|
||||
type=int,
|
||||
default=25,
|
||||
help="Clipboard-shape floor. Shorter real text is conversational turns.",
|
||||
)
|
||||
parser.add_argument("--maximum-characters", type=int, default=1_500)
|
||||
parser.add_argument(
|
||||
"--rare-stratum-target",
|
||||
type=int,
|
||||
default=50,
|
||||
help="Reviewed records to draw from each scarce intent stratum.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--common-stratum-target",
|
||||
type=int,
|
||||
default=120,
|
||||
help="Reviewed records to draw from each abundant stratum.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--common-strata",
|
||||
default="question,replyableOnly,noWeakIntent",
|
||||
help="Comma-separated strata that use --common-stratum-target.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--training-corpus",
|
||||
action="append",
|
||||
default=[],
|
||||
type=Path,
|
||||
help="Additional JSONL whose text must not appear in the queue.",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def normalized_text(value: str) -> str:
|
||||
value = unicodedata.normalize("NFKC", value.replace("\u0000", " "))
|
||||
return " ".join(value.split()).strip()
|
||||
|
||||
|
||||
def fingerprint(value: str) -> str:
|
||||
return normalized_text(value).casefold()
|
||||
|
||||
|
||||
def file_sha256(path: Path) -> str:
|
||||
return hashlib.sha256(path.read_bytes()).hexdigest()
|
||||
|
||||
|
||||
def load_records(path: Path) -> list[dict]:
|
||||
return [
|
||||
json.loads(line)
|
||||
for line in path.read_text(encoding="utf-8").splitlines()
|
||||
if line.strip()
|
||||
]
|
||||
|
||||
|
||||
def protected_fingerprints(paths: list[Path]) -> set[str]:
|
||||
values: set[str] = set()
|
||||
for path in paths:
|
||||
for record in load_records(path):
|
||||
values.add(fingerprint(record["text"]))
|
||||
return values
|
||||
|
||||
|
||||
def is_real_source(record: dict) -> bool:
|
||||
"""Reject template-generated families, which are not real user text."""
|
||||
|
||||
family = record.get("family") or ""
|
||||
return family.startswith("online_") and not family.startswith(
|
||||
EXCLUDED_FAMILY_PREFIXES
|
||||
)
|
||||
|
||||
|
||||
def stratum_of(record: dict) -> str:
|
||||
for intent in INTENT_PRIORITY:
|
||||
if record.get(intent) is True:
|
||||
return intent
|
||||
if record.get("replyable") is True:
|
||||
return REPLYABLE_STRATUM
|
||||
return NEGATIVE_STRATUM
|
||||
|
||||
|
||||
def stable_priority(record_id: str, seed: int, salt: str) -> bytes:
|
||||
return hashlib.sha256(f"{seed}|{salt}|{record_id}".encode()).digest()
|
||||
|
||||
|
||||
def eligible_pool(
|
||||
sources: list[Path],
|
||||
*,
|
||||
minimum_characters: int,
|
||||
maximum_characters: int,
|
||||
protected: set[str],
|
||||
) -> tuple[list[dict], Counter]:
|
||||
rejected: Counter = Counter()
|
||||
seen: set[str] = set()
|
||||
pool: list[dict] = []
|
||||
for path in sources:
|
||||
for record in load_records(path):
|
||||
if not is_real_source(record):
|
||||
rejected["notRealClipboardShapedSource"] += 1
|
||||
continue
|
||||
if record.get("language") not in LANGUAGES:
|
||||
rejected["unsupportedLanguage"] += 1
|
||||
continue
|
||||
text = normalized_text(record.get("text") or "")
|
||||
if not minimum_characters <= len(text) <= maximum_characters:
|
||||
rejected["lengthOutsideClipboardShape"] += 1
|
||||
continue
|
||||
key = fingerprint(text)
|
||||
if key in protected:
|
||||
rejected["overlapsProtectedCorpus"] += 1
|
||||
continue
|
||||
if key in seen:
|
||||
rejected["duplicateText"] += 1
|
||||
continue
|
||||
if PII_PATTERN.search(text):
|
||||
rejected["detectedPII"] += 1
|
||||
continue
|
||||
seen.add(key)
|
||||
candidate = dict(record)
|
||||
candidate["_text"] = text
|
||||
candidate["_stratum"] = stratum_of(record)
|
||||
pool.append(candidate)
|
||||
return pool, rejected
|
||||
|
||||
|
||||
def select(
|
||||
pool: list[dict],
|
||||
*,
|
||||
seed: int,
|
||||
rare_target: int,
|
||||
common_target: int,
|
||||
common_strata: set[str],
|
||||
) -> tuple[list[dict], list[dict]]:
|
||||
"""Draw each stratum independently and report unfilled cells."""
|
||||
|
||||
by_cell: dict[tuple[str, str], list[dict]] = defaultdict(list)
|
||||
for record in pool:
|
||||
by_cell[(record["language"], record["_stratum"])].append(record)
|
||||
|
||||
all_strata = (*INTENT_PRIORITY, REPLYABLE_STRATUM, NEGATIVE_STRATUM)
|
||||
selected: list[dict] = []
|
||||
gaps: list[dict] = []
|
||||
for language in LANGUAGES:
|
||||
for stratum in all_strata:
|
||||
target = common_target if stratum in common_strata else rare_target
|
||||
available = by_cell.get((language, stratum), [])
|
||||
drawn = sorted(
|
||||
available,
|
||||
key=lambda value: stable_priority(value["id"], seed, stratum),
|
||||
)[:target]
|
||||
probability = len(drawn) / len(available) if available else 0.0
|
||||
for record in drawn:
|
||||
record["_inclusionProbability"] = probability
|
||||
record["_stratumAvailable"] = len(available)
|
||||
record["_stratumSelected"] = len(drawn)
|
||||
selected.append(record)
|
||||
if len(drawn) < target:
|
||||
gaps.append(
|
||||
{
|
||||
"language": language,
|
||||
"stratum": stratum,
|
||||
"target": target,
|
||||
"available": len(available),
|
||||
"selected": len(drawn),
|
||||
"mustAuthor": target - len(drawn),
|
||||
}
|
||||
)
|
||||
return selected, gaps
|
||||
|
||||
|
||||
def write_jsonl(path: Path, records: list[dict]) -> None:
|
||||
path.write_text(
|
||||
"\n".join(
|
||||
json.dumps(record, ensure_ascii=False, sort_keys=True)
|
||||
for record in records
|
||||
)
|
||||
+ "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
arguments = parse_arguments()
|
||||
sources = arguments.source or list(DEFAULT_SOURCES)
|
||||
common_strata = {
|
||||
value.strip() for value in arguments.common_strata.split(",") if value.strip()
|
||||
}
|
||||
|
||||
protected = protected_fingerprints(arguments.training_corpus)
|
||||
pool, rejected = eligible_pool(
|
||||
sources,
|
||||
minimum_characters=arguments.minimum_characters,
|
||||
maximum_characters=arguments.maximum_characters,
|
||||
protected=protected,
|
||||
)
|
||||
selected, gaps = select(
|
||||
pool,
|
||||
seed=arguments.seed,
|
||||
rare_target=arguments.rare_stratum_target,
|
||||
common_target=arguments.common_stratum_target,
|
||||
common_strata=common_strata,
|
||||
)
|
||||
|
||||
output_directory = arguments.output_directory
|
||||
output_directory.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
queue: list[dict] = []
|
||||
provenance: list[dict] = []
|
||||
template: list[dict] = []
|
||||
for index, source in enumerate(
|
||||
sorted(
|
||||
selected,
|
||||
key=lambda value: stable_priority(value["id"], arguments.seed, "queue"),
|
||||
),
|
||||
start=1,
|
||||
):
|
||||
review_id = f"product-blind-v2-{index:05d}"
|
||||
queue.append(
|
||||
{
|
||||
"id": review_id,
|
||||
"text": source["_text"],
|
||||
"language": source["language"],
|
||||
"annotationStatus": "unreviewed",
|
||||
}
|
||||
)
|
||||
provenance.append(
|
||||
{
|
||||
"id": review_id,
|
||||
"language": source["language"],
|
||||
"sourceRecordID": source["id"],
|
||||
"sourceDataset": source.get("sourceDataset"),
|
||||
"sourceLicense": source.get("sourceLicense"),
|
||||
"sourceURL": source.get("sourceURL"),
|
||||
"family": source.get("family"),
|
||||
"samplingStratum": source["_stratum"],
|
||||
"inclusionProbability": source["_inclusionProbability"],
|
||||
"stratumAvailable": source["_stratumAvailable"],
|
||||
"stratumSelected": source["_stratumSelected"],
|
||||
"weakSourceLabels": {
|
||||
field: source.get(field) for field in INTENT_FIELDS
|
||||
},
|
||||
}
|
||||
)
|
||||
template.append(
|
||||
{
|
||||
"id": review_id,
|
||||
**{field: None for field in INTENT_FIELDS},
|
||||
"sentiment": None,
|
||||
"ambiguous": None,
|
||||
"confidence": None,
|
||||
"evidence": "",
|
||||
"notes": "",
|
||||
}
|
||||
)
|
||||
|
||||
queue_path = output_directory / "review-queue.jsonl"
|
||||
provenance_path = output_directory / "sealed-provenance.jsonl"
|
||||
write_jsonl(queue_path, queue)
|
||||
write_jsonl(provenance_path, provenance)
|
||||
write_jsonl(output_directory / "annotator-a.jsonl", template)
|
||||
write_jsonl(output_directory / "annotator-b.jsonl", template)
|
||||
|
||||
cell_counts: Counter = Counter(
|
||||
f"{value['language']}:{value['samplingStratum']}" for value in provenance
|
||||
)
|
||||
manifest = {
|
||||
"schemaVersion": 1,
|
||||
"seed": arguments.seed,
|
||||
"status": "awaiting-double-human-annotation",
|
||||
"humanReviewComplete": False,
|
||||
"policy": (
|
||||
"Evaluation-only stratified queue drawn from real licensed text. "
|
||||
"Weak source labels are stratification input only and are never "
|
||||
"gold. Never merge these records into training."
|
||||
),
|
||||
"design": (
|
||||
"Single-stage stratified sample. Each record belongs to exactly one "
|
||||
"(language, stratum) cell; divide by inclusionProbability for "
|
||||
"prevalence-corrected Horvitz-Thompson metrics."
|
||||
),
|
||||
"sources": [str(path) for path in sources],
|
||||
"clipboardShapeFilter": {
|
||||
"excludedFamilyPrefixes": list(EXCLUDED_FAMILY_PREFIXES),
|
||||
"minimumCharacters": arguments.minimum_characters,
|
||||
"maximumCharacters": arguments.maximum_characters,
|
||||
},
|
||||
"poolRecords": len(pool),
|
||||
"rejected": dict(rejected),
|
||||
"records": len(queue),
|
||||
"languages": dict(Counter(value["language"] for value in queue)),
|
||||
"cells": dict(sorted(cell_counts.items())),
|
||||
"unfilledCells": gaps,
|
||||
"authoringRequired": sum(gap["mustAuthor"] for gap in gaps),
|
||||
"validation": {
|
||||
"duplicateNormalizedTexts": len(queue)
|
||||
- len({fingerprint(value["text"]) for value in queue}),
|
||||
"configuredTrainingOverlap": sum(
|
||||
fingerprint(value["text"]) in protected for value in queue
|
||||
),
|
||||
"containsDetectedPII": any(
|
||||
PII_PATTERN.search(value["text"]) for value in queue
|
||||
),
|
||||
},
|
||||
"artifacts": {
|
||||
"reviewQueueSHA256": file_sha256(queue_path),
|
||||
"sealedProvenanceSHA256": file_sha256(provenance_path),
|
||||
},
|
||||
}
|
||||
(output_directory / "manifest.json").write_text(
|
||||
json.dumps(manifest, ensure_ascii=False, indent=2, sort_keys=True) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
print(json.dumps(manifest, ensure_ascii=False, indent=2, sort_keys=True))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,257 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Validate LLM-authored clipboard corpus batches before they enter any split.
|
||||
|
||||
Authored records exist because four socially scarce intents (blessing,
|
||||
invitation, confirmationDecision, scheduleNegotiation) have no commercially
|
||||
licensed real-text source. Their whole value is that they are not
|
||||
template-generated, so this script measures template-ness directly instead of
|
||||
trusting the author.
|
||||
|
||||
Checks:
|
||||
* schema, label states, and duplicate ids/text
|
||||
* PII shapes and frozen-holdout overlap
|
||||
* distinct-n diversity and the most repeated 4-gram
|
||||
* prefix concentration, which is how slot-filled templates leak a fingerprint
|
||||
* per-family label balance, so a family is never a pure label proxy
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
import unicodedata
|
||||
from collections import Counter, defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
MODEL_TRAINING = Path("ModelTraining/ClipboardSemantics")
|
||||
INTENT_FIELDS = (
|
||||
"task",
|
||||
"question",
|
||||
"invitation",
|
||||
"complaint",
|
||||
"scheduleNegotiation",
|
||||
"confirmationDecision",
|
||||
"followUpReminder",
|
||||
"blessing",
|
||||
"replyable",
|
||||
)
|
||||
SENTIMENTS = {"positive", "neutral", "negative"}
|
||||
REQUIRED_FIELDS = ("id", "text", "language", "family", "sentiment", *INTENT_FIELDS)
|
||||
LANGUAGES = {"en", "zh-Hans"}
|
||||
PII_PATTERN = re.compile(
|
||||
r"(?:[\w.+-]+@[\w.-]+\.\w+)|(?:\+?\d[\d ()-]{8,}\d)|"
|
||||
r"(?:\b\d{3}-\d{2}-\d{4}\b)",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
# Diversity floors. Template corpora sit far below these because slot filling
|
||||
# reuses one skeleton per family.
|
||||
MINIMUM_DISTINCT_4GRAM_RATIO = 0.80
|
||||
MAXIMUM_TOP_4GRAM_SHARE = 0.06
|
||||
MAXIMUM_PREFIX_SHARE = 0.10
|
||||
|
||||
|
||||
def parse_arguments() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("batches", nargs="+", type=Path)
|
||||
parser.add_argument(
|
||||
"--holdout",
|
||||
action="append",
|
||||
type=Path,
|
||||
default=[],
|
||||
help="Frozen holdout JSONL whose text must not be reused.",
|
||||
)
|
||||
parser.add_argument("--report", type=Path)
|
||||
parser.add_argument(
|
||||
"--compare-corpus",
|
||||
type=Path,
|
||||
default=MODEL_TRAINING / "clipboard_semantic_corpus.jsonl",
|
||||
help="Existing template corpus, scored with the same diversity metrics.",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def normalized_text(value: str) -> str:
|
||||
value = unicodedata.normalize("NFKC", value)
|
||||
return " ".join(value.split()).strip()
|
||||
|
||||
|
||||
def fingerprint(value: str) -> str:
|
||||
return normalized_text(value).casefold()
|
||||
|
||||
|
||||
def load_records(path: Path) -> list[dict]:
|
||||
return [
|
||||
json.loads(line)
|
||||
for line in path.read_text(encoding="utf-8").splitlines()
|
||||
if line.strip()
|
||||
]
|
||||
|
||||
|
||||
def character_ngrams(value: str, size: int) -> list[str]:
|
||||
text = normalized_text(value)
|
||||
return [text[index : index + size] for index in range(len(text) - size + 1)]
|
||||
|
||||
|
||||
def diversity_metrics(texts: list[str]) -> dict:
|
||||
"""Distinct-n and concentration statistics over character 4-grams."""
|
||||
|
||||
grams: Counter = Counter()
|
||||
for text in texts:
|
||||
grams.update(character_ngrams(text, 4))
|
||||
total = sum(grams.values())
|
||||
if not total:
|
||||
return {"distinct4GramRatio": 0.0, "topGramShare": 1.0, "topGram": ""}
|
||||
top_gram, top_count = grams.most_common(1)[0]
|
||||
prefixes = Counter(normalized_text(text)[:6] for text in texts)
|
||||
top_prefix, prefix_count = prefixes.most_common(1)[0]
|
||||
return {
|
||||
"distinct4GramRatio": round(len(grams) / total, 4),
|
||||
"topGramShare": round(top_count / total, 4),
|
||||
"topGram": top_gram,
|
||||
"topPrefix": top_prefix,
|
||||
"topPrefixShare": round(prefix_count / len(texts), 4),
|
||||
"totalGrams": total,
|
||||
"distinctGrams": len(grams),
|
||||
}
|
||||
|
||||
|
||||
def validate(records: list[dict], protected: set[str]) -> list[str]:
|
||||
errors: list[str] = []
|
||||
seen_ids: set[str] = set()
|
||||
seen_text: set[str] = set()
|
||||
for record in records:
|
||||
identifier = record.get("id", "<missing>")
|
||||
for field in REQUIRED_FIELDS:
|
||||
if field not in record:
|
||||
errors.append(f"{identifier}: missing field {field}")
|
||||
if identifier in seen_ids:
|
||||
errors.append(f"{identifier}: duplicate id")
|
||||
seen_ids.add(identifier)
|
||||
|
||||
text = normalized_text(record.get("text") or "")
|
||||
if not text:
|
||||
errors.append(f"{identifier}: empty text")
|
||||
continue
|
||||
key = fingerprint(text)
|
||||
if key in seen_text:
|
||||
errors.append(f"{identifier}: duplicate text")
|
||||
seen_text.add(key)
|
||||
if key in protected:
|
||||
errors.append(f"{identifier}: text overlaps a frozen holdout")
|
||||
if PII_PATTERN.search(text):
|
||||
errors.append(f"{identifier}: detected PII shape")
|
||||
if record.get("language") not in LANGUAGES:
|
||||
errors.append(f"{identifier}: unsupported language {record.get('language')}")
|
||||
if record.get("sentiment") not in SENTIMENTS:
|
||||
errors.append(f"{identifier}: bad sentiment {record.get('sentiment')}")
|
||||
for field in INTENT_FIELDS:
|
||||
if not isinstance(record.get(field), bool):
|
||||
errors.append(f"{identifier}: {field} must be a boolean")
|
||||
return errors
|
||||
|
||||
|
||||
def family_balance(records: list[dict]) -> dict:
|
||||
by_family: dict[str, list[dict]] = defaultdict(list)
|
||||
for record in records:
|
||||
by_family[record["family"]].append(record)
|
||||
return {
|
||||
family: {
|
||||
"records": len(values),
|
||||
"positives": {
|
||||
field: sum(1 for value in values if value.get(field) is True)
|
||||
for field in INTENT_FIELDS
|
||||
if any(value.get(field) is True for value in values)
|
||||
},
|
||||
}
|
||||
for family, values in sorted(by_family.items())
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
arguments = parse_arguments()
|
||||
records: list[dict] = []
|
||||
for path in arguments.batches:
|
||||
records.extend(load_records(path))
|
||||
|
||||
protected: set[str] = set()
|
||||
for path in arguments.holdout:
|
||||
for record in load_records(path):
|
||||
protected.add(fingerprint(record["text"]))
|
||||
|
||||
errors = validate(records, protected)
|
||||
texts = [record["text"] for record in records if record.get("text")]
|
||||
metrics = diversity_metrics(texts)
|
||||
|
||||
by_language = {
|
||||
language: diversity_metrics(
|
||||
[record["text"] for record in records if record.get("language") == language]
|
||||
)
|
||||
for language in sorted({record.get("language") for record in records} - {None})
|
||||
}
|
||||
|
||||
comparison = {}
|
||||
if arguments.compare_corpus and arguments.compare_corpus.is_file():
|
||||
compare_records = load_records(arguments.compare_corpus)
|
||||
for language in sorted(by_language):
|
||||
subset = [
|
||||
record["text"]
|
||||
for record in compare_records
|
||||
if record.get("language") == language
|
||||
]
|
||||
if subset:
|
||||
comparison[language] = diversity_metrics(subset)
|
||||
|
||||
gates: dict[str, dict] = {}
|
||||
for language, values in by_language.items():
|
||||
gates[language] = {
|
||||
"distinct4GramRatio": {
|
||||
"actual": values["distinct4GramRatio"],
|
||||
"required": MINIMUM_DISTINCT_4GRAM_RATIO,
|
||||
"passed": values["distinct4GramRatio"] >= MINIMUM_DISTINCT_4GRAM_RATIO,
|
||||
},
|
||||
"topGramShare": {
|
||||
"actual": values["topGramShare"],
|
||||
"maximum": MAXIMUM_TOP_4GRAM_SHARE,
|
||||
"passed": values["topGramShare"] <= MAXIMUM_TOP_4GRAM_SHARE,
|
||||
},
|
||||
"topPrefixShare": {
|
||||
"actual": values["topPrefixShare"],
|
||||
"maximum": MAXIMUM_PREFIX_SHARE,
|
||||
"passed": values["topPrefixShare"] <= MAXIMUM_PREFIX_SHARE,
|
||||
},
|
||||
}
|
||||
|
||||
all_passed = not errors and all(
|
||||
check["passed"] for language in gates.values() for check in language.values()
|
||||
)
|
||||
report = {
|
||||
"schemaVersion": 1,
|
||||
"batches": [str(path) for path in arguments.batches],
|
||||
"records": len(records),
|
||||
"languages": dict(Counter(record.get("language") for record in records)),
|
||||
"families": len({record.get("family") for record in records}),
|
||||
"positivesByIntent": {
|
||||
field: sum(1 for record in records if record.get(field) is True)
|
||||
for field in INTENT_FIELDS
|
||||
},
|
||||
"diversity": metrics,
|
||||
"diversityByLanguage": by_language,
|
||||
"templateCorpusComparison": comparison,
|
||||
"gates": gates,
|
||||
"familyBalance": family_balance(records),
|
||||
"errors": errors,
|
||||
"allChecksPassed": all_passed,
|
||||
}
|
||||
output = json.dumps(report, ensure_ascii=False, indent=2, sort_keys=True)
|
||||
if arguments.report:
|
||||
arguments.report.write_text(output + "\n", encoding="utf-8")
|
||||
print(output)
|
||||
return 0 if all_passed else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -1,273 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Build the companion Shortcut for the Extract Events skill.
|
||||
|
||||
Writes an unsigned binary plist that `shortcuts sign` can notarize.
|
||||
|
||||
Workflow (must materialize Shortcut Input as Text first — Split Text's
|
||||
WFInput/ExtensionInput binding finishes the run without iterating):
|
||||
|
||||
Text (Shortcut Input)
|
||||
→ Split Text (new lines)
|
||||
→ Repeat Each
|
||||
→ Split Text (|)
|
||||
→ Get items 1–4 (start, end, title, location)
|
||||
→ If end == ALLDAY
|
||||
→ Add New Event (all-day)
|
||||
Otherwise
|
||||
→ Add New Event (timed)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import plistlib
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
OUT_DIR = ROOT / "OSGKeyboard" / "Resources" / "Shortcuts"
|
||||
UNSIGNED = OUT_DIR / "OSGExtractEvents.unsigned.shortcut"
|
||||
SIGNED = OUT_DIR / "OSGExtractEvents.shortcut"
|
||||
SHORTCUT_NAME = "OSGExtractEvents"
|
||||
|
||||
OBJECT_REPLACEMENT = "\ufffc"
|
||||
|
||||
TEXT_UUID = "A1B2C3D0-1111-4E00-8A5A-FD7E1B0C396F"
|
||||
SPLIT_LINES_UUID = "A1B2C3D1-2222-4F11-9A6B-0E8F2C1D4A70"
|
||||
REPEAT_GROUP = "A1B2C3D2-3333-4E02-B3C4-91F0E6A7B812"
|
||||
REPEAT_START = "A1B2C3D3-4444-4F13-A4D5-02A1F7B8C923"
|
||||
SPLIT_FIELDS_UUID = "A1B2C3D4-5555-4014-B5E6-13B2A8C9D034"
|
||||
START_UUID = "A1B2C3D5-6666-4125-C6F7-24C3B9DAE145"
|
||||
END_UUID = "A1B2C3D6-7777-4236-D7A8-35D4CAEBF256"
|
||||
TITLE_UUID = "A1B2C3D7-8888-4347-E8B9-46E5DBF0C367"
|
||||
LOCATION_UUID = "A1B2C3D8-9999-4458-F9CA-57F6EC01D478"
|
||||
IF_GROUP = "A1B2C3D9-AAAA-4569-0ADB-6807FD12E589"
|
||||
IF_START = "A1B2C3DA-BBBB-467A-1BEC-79180E23F69A"
|
||||
EVENT_ALLDAY_UUID = "A1B2C3DB-CCCC-478B-2CFD-8A291F3407AB"
|
||||
IF_ELSE = "A1B2C3DC-DDDD-489C-3D0E-9B3A204518BC"
|
||||
EVENT_TIMED_UUID = "A1B2C3DD-EEEE-49AD-4E1F-AC4B315629CD"
|
||||
IF_END = "A1B2C3DE-FFFF-4ABE-5F20-BD5C42673ADE"
|
||||
REPEAT_END = "A1B2C3DF-0000-4BCF-6031-CE6D53784BEF"
|
||||
|
||||
|
||||
def attachment(value: dict) -> dict:
|
||||
return {"Value": value, "WFSerializationType": "WFTextTokenAttachment"}
|
||||
|
||||
|
||||
def token_string(attachment_value: dict) -> dict:
|
||||
return {
|
||||
"Value": {
|
||||
"attachmentsByRange": {"{0, 1}": attachment_value},
|
||||
"string": OBJECT_REPLACEMENT,
|
||||
},
|
||||
"WFSerializationType": "WFTextTokenString",
|
||||
}
|
||||
|
||||
|
||||
def shortcut_input_text() -> dict:
|
||||
return token_string({"Type": "ExtensionInput"})
|
||||
|
||||
|
||||
def action_output_text(output_name: str, output_uuid: str) -> dict:
|
||||
return token_string(
|
||||
{
|
||||
"Aggrandizements": [],
|
||||
"OutputName": output_name,
|
||||
"OutputUUID": output_uuid,
|
||||
"Type": "ActionOutput",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def action_output_attachment(output_name: str, output_uuid: str) -> dict:
|
||||
return attachment(
|
||||
{
|
||||
"Aggrandizements": [],
|
||||
"OutputName": output_name,
|
||||
"OutputUUID": output_uuid,
|
||||
"Type": "ActionOutput",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def named_variable_text(variable_name: str) -> dict:
|
||||
return token_string(
|
||||
{
|
||||
"Aggrandizements": [],
|
||||
"Type": "Variable",
|
||||
"VariableName": variable_name,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def if_input(output_name: str, output_uuid: str) -> dict:
|
||||
return {
|
||||
"Type": "Variable",
|
||||
"Variable": {
|
||||
"Value": {
|
||||
"Aggrandizements": [],
|
||||
"OutputName": output_name,
|
||||
"OutputUUID": output_uuid,
|
||||
"Type": "ActionOutput",
|
||||
},
|
||||
"WFSerializationType": "WFTextTokenAttachment",
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def get_item(uuid: str, name: str, index: int) -> dict:
|
||||
return {
|
||||
"WFWorkflowActionIdentifier": "is.workflow.actions.getitemfromlist",
|
||||
"WFWorkflowActionParameters": {
|
||||
"UUID": uuid,
|
||||
"CustomOutputName": name,
|
||||
"WFInput": action_output_attachment("字段", SPLIT_FIELDS_UUID),
|
||||
"WFItemSpecifier": "Item At Index",
|
||||
"WFItemIndex": index,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def add_event(
|
||||
uuid: str,
|
||||
*,
|
||||
all_day: bool,
|
||||
include_end: bool,
|
||||
) -> dict:
|
||||
parameters = {
|
||||
"UUID": uuid,
|
||||
"WFCalendarItemTitle": action_output_text("标题", TITLE_UUID),
|
||||
"WFCalendarItemLocation": action_output_text("地点", LOCATION_UUID),
|
||||
"WFCalendarItemDates": True,
|
||||
"WFCalendarItemStartDate": action_output_text("开始", START_UUID),
|
||||
"WFCalendarItemAllDay": all_day,
|
||||
"WFCalendarItemShowComposer": False,
|
||||
"ShowComposeSheet": False,
|
||||
"WFShowWhenRun": False,
|
||||
}
|
||||
if include_end:
|
||||
parameters["WFCalendarItemEndDate"] = action_output_text("结束", END_UUID)
|
||||
return {
|
||||
"WFWorkflowActionIdentifier": "is.workflow.actions.addnewevent",
|
||||
"WFWorkflowActionParameters": parameters,
|
||||
}
|
||||
|
||||
|
||||
def workflow() -> dict:
|
||||
return {
|
||||
"WFWorkflowClientVersion": "3600.0.4",
|
||||
"WFWorkflowClientRelease": "26.0",
|
||||
"WFWorkflowMinimumClientVersion": 900,
|
||||
"WFWorkflowMinimumClientVersionString": "900",
|
||||
"WFWorkflowName": SHORTCUT_NAME,
|
||||
"WFWorkflowHasShortcutInputVariables": True,
|
||||
"WFWorkflowHasOutputFallback": False,
|
||||
"WFWorkflowImportQuestions": [],
|
||||
"WFWorkflowTypes": [],
|
||||
"WFWorkflowInputContentItemClasses": [
|
||||
"WFStringContentItem",
|
||||
"WFRichTextContentItem",
|
||||
],
|
||||
"WFWorkflowOutputContentItemClasses": [],
|
||||
"WFWorkflowNoInputBehavior": {
|
||||
"Name": "WFWorkflowNoInputBehaviorAskForInput",
|
||||
"Parameters": {
|
||||
"ItemClass": "WFStringContentItem",
|
||||
},
|
||||
},
|
||||
"WFWorkflowIcon": {
|
||||
"WFWorkflowIconGlyphNumber": 59675,
|
||||
"WFWorkflowIconStartColor": 4286828031,
|
||||
},
|
||||
"WFWorkflowActions": [
|
||||
{
|
||||
"WFWorkflowActionIdentifier": "is.workflow.actions.gettext",
|
||||
"WFWorkflowActionParameters": {
|
||||
"UUID": TEXT_UUID,
|
||||
"CustomOutputName": "快捷指令文本",
|
||||
"WFTextActionText": shortcut_input_text(),
|
||||
},
|
||||
},
|
||||
{
|
||||
"WFWorkflowActionIdentifier": "is.workflow.actions.text.split",
|
||||
"WFWorkflowActionParameters": {
|
||||
"UUID": SPLIT_LINES_UUID,
|
||||
"CustomOutputName": "行",
|
||||
"text": action_output_text("快捷指令文本", TEXT_UUID),
|
||||
"WFTextSeparator": "New Lines",
|
||||
},
|
||||
},
|
||||
{
|
||||
"WFWorkflowActionIdentifier": "is.workflow.actions.repeat.each",
|
||||
"WFWorkflowActionParameters": {
|
||||
"UUID": REPEAT_START,
|
||||
"GroupingIdentifier": REPEAT_GROUP,
|
||||
"WFControlFlowMode": 0,
|
||||
"WFInput": action_output_attachment("行", SPLIT_LINES_UUID),
|
||||
},
|
||||
},
|
||||
{
|
||||
"WFWorkflowActionIdentifier": "is.workflow.actions.text.split",
|
||||
"WFWorkflowActionParameters": {
|
||||
"UUID": SPLIT_FIELDS_UUID,
|
||||
"CustomOutputName": "字段",
|
||||
"text": named_variable_text("Repeat Item"),
|
||||
"WFTextSeparator": "Custom",
|
||||
"WFTextCustomSeparator": "|",
|
||||
},
|
||||
},
|
||||
get_item(START_UUID, "开始", 1),
|
||||
get_item(END_UUID, "结束", 2),
|
||||
get_item(TITLE_UUID, "标题", 3),
|
||||
get_item(LOCATION_UUID, "地点", 4),
|
||||
{
|
||||
"WFWorkflowActionIdentifier": "is.workflow.actions.conditional",
|
||||
"WFWorkflowActionParameters": {
|
||||
"UUID": IF_START,
|
||||
"GroupingIdentifier": IF_GROUP,
|
||||
"WFControlFlowMode": 0,
|
||||
"WFCondition": 4,
|
||||
"WFConditionalActionString": "ALLDAY",
|
||||
"WFInput": if_input("结束", END_UUID),
|
||||
},
|
||||
},
|
||||
add_event(EVENT_ALLDAY_UUID, all_day=True, include_end=False),
|
||||
{
|
||||
"WFWorkflowActionIdentifier": "is.workflow.actions.conditional",
|
||||
"WFWorkflowActionParameters": {
|
||||
"UUID": IF_ELSE,
|
||||
"GroupingIdentifier": IF_GROUP,
|
||||
"WFControlFlowMode": 1,
|
||||
},
|
||||
},
|
||||
add_event(EVENT_TIMED_UUID, all_day=False, include_end=True),
|
||||
{
|
||||
"WFWorkflowActionIdentifier": "is.workflow.actions.conditional",
|
||||
"WFWorkflowActionParameters": {
|
||||
"UUID": IF_END,
|
||||
"GroupingIdentifier": IF_GROUP,
|
||||
"WFControlFlowMode": 2,
|
||||
},
|
||||
},
|
||||
{
|
||||
"WFWorkflowActionIdentifier": "is.workflow.actions.repeat.each",
|
||||
"WFWorkflowActionParameters": {
|
||||
"UUID": REPEAT_END,
|
||||
"GroupingIdentifier": REPEAT_GROUP,
|
||||
"WFControlFlowMode": 2,
|
||||
},
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
OUT_DIR.mkdir(parents=True, exist_ok=True)
|
||||
UNSIGNED.write_bytes(plistlib.dumps(workflow(), fmt=plistlib.FMT_BINARY))
|
||||
print(f"wrote {UNSIGNED}")
|
||||
print(
|
||||
"sign with: shortcuts sign --mode anyone "
|
||||
f"--input {UNSIGNED} --output {SIGNED}"
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,170 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Build the companion Shortcut for the Extract Tasks skill.
|
||||
|
||||
Writes an unsigned binary plist that `shortcuts sign` can notarize.
|
||||
|
||||
Workflow (must materialize Shortcut Input as Text first — Split Text's
|
||||
WFInput/ExtensionInput binding finishes the run without iterating):
|
||||
|
||||
Text (Shortcut Input)
|
||||
→ Split Text (new lines)
|
||||
→ Repeat Each
|
||||
→ Add New Reminder (title = Repeat Item, default list, no composer)
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import plistlib
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
OUT_DIR = ROOT / "OSGKeyboard" / "Resources" / "Shortcuts"
|
||||
UNSIGNED = OUT_DIR / "OSGExtractTodos.unsigned.shortcut"
|
||||
SIGNED = OUT_DIR / "OSGExtractTodos.shortcut"
|
||||
SHORTCUT_NAME = "OSGExtractTodos"
|
||||
|
||||
OBJECT_REPLACEMENT = "\ufffc"
|
||||
|
||||
TEXT_UUID = "6A2D0A80-3B1C-4E00-8A5A-FD7E1B0C396F"
|
||||
SPLIT_UUID = "7A3E1B90-4C2D-4F11-9A6B-0E8F2C1D4A70"
|
||||
REPEAT_GROUP = "2B9C8D11-55AA-4E02-B3C4-91F0E6A7B812"
|
||||
REPEAT_START = "3C0D9E22-66BB-4F13-A4D5-02A1F7B8C923"
|
||||
REMINDER_UUID = "4D1E0F33-77CC-4014-B5E6-13B2A8C9D034"
|
||||
REPEAT_END = "5E2F1044-88DD-4125-C6F7-24C3B9DAE145"
|
||||
|
||||
|
||||
def attachment(value: dict) -> dict:
|
||||
return {"Value": value, "WFSerializationType": "WFTextTokenAttachment"}
|
||||
|
||||
|
||||
def token_string(attachment_value: dict) -> dict:
|
||||
return {
|
||||
"Value": {
|
||||
"attachmentsByRange": {"{0, 1}": attachment_value},
|
||||
"string": OBJECT_REPLACEMENT,
|
||||
},
|
||||
"WFSerializationType": "WFTextTokenString",
|
||||
}
|
||||
|
||||
|
||||
def shortcut_input_text() -> dict:
|
||||
return token_string({"Type": "ExtensionInput"})
|
||||
|
||||
|
||||
def action_output_text(output_name: str, output_uuid: str) -> dict:
|
||||
return token_string(
|
||||
{
|
||||
"Aggrandizements": [],
|
||||
"OutputName": output_name,
|
||||
"OutputUUID": output_uuid,
|
||||
"Type": "ActionOutput",
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def named_variable_text(variable_name: str) -> dict:
|
||||
return token_string(
|
||||
{
|
||||
"Aggrandizements": [],
|
||||
"Type": "Variable",
|
||||
"VariableName": variable_name,
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
def workflow() -> dict:
|
||||
return {
|
||||
"WFWorkflowClientVersion": "3600.0.4",
|
||||
"WFWorkflowClientRelease": "26.0",
|
||||
"WFWorkflowMinimumClientVersion": 900,
|
||||
"WFWorkflowMinimumClientVersionString": "900",
|
||||
"WFWorkflowName": SHORTCUT_NAME,
|
||||
"WFWorkflowHasShortcutInputVariables": True,
|
||||
"WFWorkflowHasOutputFallback": False,
|
||||
"WFWorkflowImportQuestions": [],
|
||||
# Empty types: a normal library shortcut. QuickActions/ActionExtension
|
||||
# made `run-shortcut?name=` open Shortcuts without running it.
|
||||
"WFWorkflowTypes": [],
|
||||
"WFWorkflowInputContentItemClasses": [
|
||||
"WFStringContentItem",
|
||||
"WFRichTextContentItem",
|
||||
],
|
||||
"WFWorkflowOutputContentItemClasses": [],
|
||||
"WFWorkflowNoInputBehavior": {
|
||||
"Name": "WFWorkflowNoInputBehaviorAskForInput",
|
||||
"Parameters": {
|
||||
"ItemClass": "WFStringContentItem",
|
||||
},
|
||||
},
|
||||
"WFWorkflowIcon": {
|
||||
"WFWorkflowIconGlyphNumber": 59511,
|
||||
"WFWorkflowIconStartColor": 2071128575,
|
||||
},
|
||||
"WFWorkflowActions": [
|
||||
{
|
||||
"WFWorkflowActionIdentifier": "is.workflow.actions.gettext",
|
||||
"WFWorkflowActionParameters": {
|
||||
"UUID": TEXT_UUID,
|
||||
"CustomOutputName": "快捷指令文本",
|
||||
"WFTextActionText": shortcut_input_text(),
|
||||
},
|
||||
},
|
||||
{
|
||||
"WFWorkflowActionIdentifier": "is.workflow.actions.text.split",
|
||||
"WFWorkflowActionParameters": {
|
||||
"UUID": SPLIT_UUID,
|
||||
# Split Text reads `text`, not `WFInput`.
|
||||
"text": action_output_text("快捷指令文本", TEXT_UUID),
|
||||
"WFTextSeparator": "New Lines",
|
||||
},
|
||||
},
|
||||
{
|
||||
"WFWorkflowActionIdentifier": "is.workflow.actions.repeat.each",
|
||||
"WFWorkflowActionParameters": {
|
||||
"UUID": REPEAT_START,
|
||||
"GroupingIdentifier": REPEAT_GROUP,
|
||||
"WFControlFlowMode": 0,
|
||||
"WFInput": attachment(
|
||||
{
|
||||
"Aggrandizements": [],
|
||||
"OutputName": "Split Text",
|
||||
"OutputUUID": SPLIT_UUID,
|
||||
"Type": "ActionOutput",
|
||||
}
|
||||
),
|
||||
},
|
||||
},
|
||||
{
|
||||
"WFWorkflowActionIdentifier": "is.workflow.actions.addnewreminder",
|
||||
"WFWorkflowActionParameters": {
|
||||
"UUID": REMINDER_UUID,
|
||||
"WFCalendarItemTitle": named_variable_text("Repeat Item"),
|
||||
"WFCalendarItemShowComposer": False,
|
||||
"ShowComposeSheet": False,
|
||||
"WFShowWhenRun": False,
|
||||
},
|
||||
},
|
||||
{
|
||||
"WFWorkflowActionIdentifier": "is.workflow.actions.repeat.each",
|
||||
"WFWorkflowActionParameters": {
|
||||
"UUID": REPEAT_END,
|
||||
"GroupingIdentifier": REPEAT_GROUP,
|
||||
"WFControlFlowMode": 2,
|
||||
},
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
OUT_DIR.mkdir(parents=True, exist_ok=True)
|
||||
UNSIGNED.write_bytes(plistlib.dumps(workflow(), fmt=plistlib.FMT_BINARY))
|
||||
print(f"wrote {UNSIGNED}")
|
||||
print(
|
||||
"sign with: shortcuts sign --mode anyone "
|
||||
f"--input {UNSIGNED} --output {SIGNED}"
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user