Rocky
31f5937a7f
Reduce extension memory pressure and delivery races while adding richer candidates, tactile feedback, and safer two-level creative polishing.
99 lines
4.7 KiB
Markdown
99 lines
4.7 KiB
Markdown
# Testing
|
|
|
|
OSGKeyboard keeps **one hermetic XCTest suite**, organized so you can run **all** or **a subset** without duplicating cases.
|
|
|
|
## Single source of truth
|
|
|
|
| File | Role |
|
|
|------|------|
|
|
| [`Tests/suite-manifest.json`](../Tests/suite-manifest.json) | Atomic groups + presets. Each `*Tests` class appears in **exactly one** group. |
|
|
| [`Scripts/resolve_test_suite.py`](../Scripts/resolve_test_suite.py) | Expand presets, validate membership vs on-disk files. |
|
|
| [`Scripts/run-tests.sh`](../Scripts/run-tests.sh) | Resolve → `xcodebuild test -only-testing:…` (iOS and/or Mac). |
|
|
|
|
Helpers such as `FakeUbiquitousKeyValueStore.swift` are not listed (they are not XCTest classes).
|
|
|
|
## Quick start (Mac + Xcode)
|
|
|
|
```bash
|
|
./Scripts/generate-xcodeproj.sh # once / after project.yml changes
|
|
./Scripts/run-tests.sh list # presets + groups
|
|
./Scripts/run-tests.sh validate # every on-disk *Tests.swift is in the manifest once
|
|
./Scripts/run-tests.sh pr # default CI / PR gate
|
|
```
|
|
|
|
### Presets
|
|
|
|
| Preset | Contains | When to use |
|
|
|--------|----------|-------------|
|
|
| `pr` | config, sync, polish, cloud_asr, local_asr, utterance, flow, keyboard | PR / default CI |
|
|
| `all` | `pr` + `host_misc` + `pipeline_perf` | Full iOS+Ext hermetic run before release |
|
|
| `api` | cloud_asr, polish | Online API contracts + polish/LLM stubs |
|
|
| `asr` | cloud_asr, local_asr, utterance | Transcription path |
|
|
| `polish` | polish | Polish only |
|
|
| `keyboard` | keyboard, **flow** | Keyboard + handoff/mic |
|
|
| `flow` | flow | Flow session only |
|
|
| `sync` | sync | iCloud sync only |
|
|
| `perf` | pipeline_perf | Hermetic voice→polish stage timings (stub ASR/LLM) |
|
|
| `mac` | mac | macOS host only (separate scheme) |
|
|
|
|
`live_api` is reserved for optional live-network smoke and is **empty / excluded** from `all` and `pr` by design.
|
|
|
|
### Atomic groups or mixes
|
|
|
|
```bash
|
|
./Scripts/run-tests.sh polish
|
|
./Scripts/run-tests.sh cloud_asr utterance
|
|
./Scripts/run-tests.sh perf
|
|
./Scripts/run-tests.sh api keyboard
|
|
DRY_RUN=1 ./Scripts/run-tests.sh all
|
|
```
|
|
|
|
### Environment
|
|
|
|
| Variable | Meaning |
|
|
|----------|---------|
|
|
| `DESTINATION` | iOS Simulator destination (default: `platform=iOS Simulator,name=iPhone 17`) |
|
|
| `MAC_DESTINATION` | macOS destination (default: `platform=macOS`) |
|
|
| `CONFIGURATION` | `Debug` (default) or `Release` |
|
|
| `DRY_RUN=1` | Print `xcodebuild` only |
|
|
| `SKIP_GENERATE=1` | Do not call `generate-xcodeproj.sh` even if the project is missing |
|
|
|
|
## CI
|
|
|
|
`.github/workflows/ci.yml` runs `./Scripts/run-tests.sh pr` (critical path, including `OSGKeyboardExtTests`).
|
|
|
|
## Adding a new test
|
|
|
|
1. Add the XCTest class under the correct target folder (`OSGKeyboardTests`, `OSGKeyboardExtTests`, or `OSGKeyboardMacTests`).
|
|
2. Register it in **exactly one** group in `Tests/suite-manifest.json`.
|
|
3. Run `./Scripts/run-tests.sh validate`.
|
|
4. Prefer extending an existing group; only add a new group when the domain is genuinely new.
|
|
|
|
## Layer note (avoid false “duplicates”)
|
|
|
|
- `ChunkedUtterancePipelineTests` (HostSupport pipeline orchestration) and `FinalChunkRecoveryTests` (Ext short/empty final recovery) cover **different layers** — both stay, both live in `utterance`.
|
|
- Cloud ASR **runtime** WebSocket clients and `live_api` smoke are future work inside `cloud_asr` / `live_api`, not a second parallel suite.
|
|
- Flow keyboard mic regressions (orange stuck / jetsam re-adopt / command seq) are covered by `FlowKeyboardPolicies` helpers in the `flow` group.
|
|
- Streaming provider event JSON / Volcengine frames are covered by `CloudASRStreamingEventParsingTests` in `cloud_asr`.
|
|
- Translation chip App Group poll clobber is covered by `KeyboardTranslationConfigProtectionTests` in `keyboard`.
|
|
|
|
## Pipeline performance (`perf` / `pipeline_perf`)
|
|
|
|
Hermetic **voice → chunk ASR → transcript guard → (optional batch fallback) → polish → App Group bridge deliver** timings live in `VoicePipelinePerformanceTests`.
|
|
|
|
- Synthetic PCM + stub ASR/LLM only — **no mic, no live network**.
|
|
- Each run attaches a stage report (`pcm_feed`, `chunk_asr`, `transcript_guard`, `batch_fallback?`, `polish`, `bridge_deliver`, `total_e2e`) to the xcresult.
|
|
- Run alone: `./Scripts/run-tests.sh perf`. Included in `all`, **not** in `pr` (keeps the PR gate free of timing-sensitive ceilings).
|
|
|
|
These measure harness/orchestration cost under stubs — not on-device SpeechAnalyzer or real LLM latency.
|
|
|
|
## Physical-device Flow audio gate
|
|
|
|
Bluetooth HFP and PiP route transitions cannot be validated by the simulator.
|
|
Before release, run [FLOW_BLUETOOTH_TESTING.md](FLOW_BLUETOOTH_TESTING.md) and
|
|
retain the filtered trace log with the release record.
|
|
|
|
## Manual / non-XCTest checklists
|
|
|
|
Some product surfaces still use manual docs (StoreKit, personal-dictionary iCloud, etc.). Those are complementary; they are not duplicated into XCTest.
|