OSGKeyboard/docs/TESTING.md

Testing

OSGKeyboard keeps one hermetic XCTest suite, organized so you can run all or a subset without duplicating cases.

Single source of truth

File	Role
Tests/suite-manifest.json	Atomic groups + presets. Each *Tests class appears in exactly one group.
Scripts/resolve_test_suite.py	Expand presets, validate membership vs on-disk files.
Scripts/run-tests.sh	Resolve → xcodebuild test -only-testing:… (iOS and/or Mac).

Helpers such as FakeUbiquitousKeyValueStore.swift are not listed (they are not XCTest classes).

Quick start (Mac + Xcode)

./Scripts/generate-xcodeproj.sh   # once / after project.yml changes
./Scripts/run-tests.sh list       # presets + groups
./Scripts/run-tests.sh validate   # every on-disk *Tests.swift is in the manifest once
./Scripts/run-tests.sh pr         # default CI / PR gate

Presets

Preset	Contains	When to use
pr	config, sync, polish, cloud_asr, local_asr, utterance, flow, keyboard	PR / default CI
all	pr + host_misc + pipeline_perf	Full iOS+Ext hermetic run before release
api	cloud_asr, polish	Online API contracts + polish/LLM stubs
asr	cloud_asr, local_asr, utterance	Transcription path
polish	polish	Polish only
keyboard	keyboard, flow	Keyboard + handoff/mic
flow	flow	Flow session only
sync	sync	iCloud sync only
perf	pipeline_perf	Hermetic voice→polish stage timings (stub ASR/LLM)
mac	mac	macOS host only (separate scheme)

live_api is reserved for optional live-network smoke and is empty / excluded from all and pr by design.

Atomic groups or mixes

./Scripts/run-tests.sh polish
./Scripts/run-tests.sh cloud_asr utterance
./Scripts/run-tests.sh perf
./Scripts/run-tests.sh api keyboard
DRY_RUN=1 ./Scripts/run-tests.sh all

Environment

Variable	Meaning
DESTINATION	iOS Simulator destination (default: platform=iOS Simulator,name=iPhone 17)
MAC_DESTINATION	macOS destination (default: platform=macOS)
CONFIGURATION	Debug (default) or Release
DRY_RUN=1	Print xcodebuild only
SKIP_GENERATE=1	Do not call generate-xcodeproj.sh even if the project is missing

CI

.github/workflows/ci.yml runs ./Scripts/run-tests.sh pr (critical path, including OSGKeyboardExtTests).

Adding a new test

1. Add the XCTest class under the correct target folder (OSGKeyboardTests, OSGKeyboardExtTests, or OSGKeyboardMacTests).
2. Register it in exactly one group in Tests/suite-manifest.json.
3. Run ./Scripts/run-tests.sh validate.
4. Prefer extending an existing group; only add a new group when the domain is genuinely new.

Layer note (avoid false “duplicates”)

• ChunkedUtterancePipelineTests (HostSupport pipeline orchestration) and FinalChunkRecoveryTests (Ext short/empty final recovery) cover different layers — both stay, both live in utterance.
• Cloud ASR runtime WebSocket clients and live_api smoke are future work inside cloud_asr / live_api, not a second parallel suite.
• Flow keyboard mic regressions (orange stuck / jetsam re-adopt / command seq) are covered by FlowKeyboardPolicies helpers in the flow group.
• Streaming provider event JSON / Volcengine frames are covered by CloudASRStreamingEventParsingTests in cloud_asr.
• Translation chip App Group poll clobber is covered by KeyboardTranslationConfigProtectionTests in keyboard.

Pipeline performance (perf / pipeline_perf)

Hermetic voice → chunk ASR → transcript guard → (optional batch fallback) → polish → App Group bridge deliver timings live in VoicePipelinePerformanceTests.

• Synthetic PCM + stub ASR/LLM only — no mic, no live network.
• Each run attaches a stage report (pcm_feed, chunk_asr, transcript_guard, batch_fallback?, polish, bridge_deliver, total_e2e) to the xcresult.
• Run alone: ./Scripts/run-tests.sh perf. Included in all, not in pr (keeps the PR gate free of timing-sensitive ceilings).

These measure harness/orchestration cost under stubs — not on-device SpeechAnalyzer or real LLM latency.

Physical-device Flow audio gate

Bluetooth HFP and PiP route transitions cannot be validated by the simulator. Before release, run FLOW_BLUETOOTH_TESTING.md and retain the filtered trace log with the release record.

Manual / non-XCTest checklists

Some product surfaces still use manual docs (StoreKit, personal-dictionary iCloud, etc.). Those are complementary; they are not duplicated into XCTest.