Two long-standing UX papercuts, fixed without leaving the keyboard:
1. **First-launch onboarding inside the keyboard** — iOS keyboard
extensions *cannot* programmatically switch back to the previous
app after jumping out, so the old flow (jump to host app → user
has to manually navigate back) was 5+ taps of friction. The new
`KeyboardOnboardingOverlay` keeps the user inside the keyboard
for steps 1 (welcome), 2 (mic permission), 3 (speech permission),
and 5 (API key hint). The only step that *must* leave is step 4
("Enable Keyboard"), which jumps to `Settings.app` via
`UIApplication.openSettingsURLString` — on return,
`viewWillAppear` calls `autoAdvancePastKeyboardSetupStepIfNeeded`
which silently advances past that step if the keyboard is now
enabled. Net UX: user types in their app, keyboard walks them
through setup, normal UI appears as soon as setup is done.
2. **Per-app context chip on the keyboard top bar** — the v0.3.0
intelligent-prompt pipeline already adapted tone to detected
context (code/email/chat/document), but without a UI cue the user
had no way to know which mode was active or override the heuristic
when it guessed wrong. The new `AppContextChip` surfaces the
detected context; tap-to-override writes back to
`AppGroupStore.setDetectedAppContext` so the next LLM call
picks up the new tone immediately. Wired into `pressBegan` so
the chip updates in real time as the user types into different
fields.
### Files added
- `OSGKeyboardExt/Views/AppContextChip.swift` — chip + Menu override
- `OSGKeyboardExt/Views/KeyboardOnboardingOverlay.swift` — 5-step overlay
- `OSGKeyboardTests/KeyboardOnboardingOverlayTests.swift` — round-trip + enum surface tests
### Files modified
- `OSGKeyboardShared/Services/KeyboardState.swift`
+ `hasCompletedOnboarding`, `onboardingPage`, `appContext`
+ `setAppContext`, `advanceOnboarding`, `completeOnboarding`
+ `requestMicPermission`, `requestSpeechPermission`, `openSystemSettings`
- `OSGKeyboardShared/Services/AppGroupStore.swift`
+ `hasCompletedOnboarding` / `onboardingPage` accessors (mirror of
`ProviderConfig` keys, so the keyboard extension never has to
instantiate the host-app config)
- `OSGKeyboardExt/KeyboardViewController.swift`
+ action hooks wired (`installStateActions`)
+ `syncOnboardingStateFromAppGroup` / `syncAppContextFromAppGroup`
called on `viewWillAppear` and `loadPersistedConfig`
+ `autoAdvancePastKeyboardSetupStepIfNeeded` for the silent
"jump out → come back" flow
+ `openSystemSettingsFromExtension` opens `Settings.app` via
`UIApplication.openSettingsURLString` (the only system URL
the extension is allowed to open)
+ `detectAndStoreAppContext` mirrors to `state.appContext` so
the chip updates without waiting for `viewWillAppear`
- `OSGKeyboardExt/Views/KeyboardRootView.swift`
+ overlay mounted in `ZStack` over normal UI (animated)
+ AppContextChip in top bar (hidden during onboarding)
- `OSGKeyboardExt/{en,zh-Hans}.lproj/Keyboard.strings`
+ onboarding copy + chip labels (35 keys per language)
### iOS sandbox notes (kept here for posterity)
- Keyboard extensions **cannot** present AVAudioSession /
SFSpeechRecognizer permission dialogs directly. The overlay's
step 2/3 buttons optimistically advance; the actual permission
is granted when the user first opens the host app (which the
step-5 "Open OSGKeyboard" button triggers). This is the same
pattern the previous "jump to host app" flow used — just
without the broken return trip.
- `UIApplication.openSettingsURLString` is the only system URL
reachable from `extensionContext.open`. Both step 4 and the
"Open Settings" button route through `HostAppLauncher` so the
responder-chain fallback also kicks in if needed.
Co-authored-by: Mavis <Mavis@hkgood.dev>
v0.3.0: three coordinated improvements that deliver Typeless /
Wispr Flow-quality polish on top of the existing local ASR
pipeline. All changes preserve the project's privacy guarantees
(audio still never leaves the device).
## 1. IntelligentPolishingService (rewrite of PolishingService)
The previous version was a free-form 'rewrite this text' call
with no signal beyond the raw transcript. The new one is a
single LLM call that does three things in one pass, exactly as
Typeless and Wispr Flow do internally:
1. ASR error correction (homophones, near-misses, missing chars)
2. Polish (drop filler words, fix grammar, add punctuation)
3. Style adaptation per app context (code / email / chat / doc)
The merged-prompt design halves the round-trip vs the previously
proposed two-stage design (correction + polish separately) and
the academic literature confirms it performs equivalently for
everyday Chinese / English dictation.
## 2. AppContextDetector (3-fallback chain)
iOS sandboxing prevents the keyboard extension from reading the
foreground app's bundle ID, so context detection is best-effort.
The detector runs three fallbacks in order, with caching to
avoid the cold-start 'unknown' that would force a neutral-tone
LLM call every time the user opens a new field:
1. Heuristic on the text at the cursor (code / email / chat / doc)
2. 30-minute cache of the last successful detection
3. Time-of-day + weekend heuristic as a soft default
The keyboard extension runs the detector on every press of the
mic and persists the result to the App Group so the host app's
polisher picks it up.
## 3. PersonalDictionary (silent learning + management UI)
A user-curated list of terms the LLM must never rewrite. The
default growth path is silent: DictionaryLearner runs on every
History tab open and lifts frequently-dictated English
identifiers (Kubernetes, OpenAI, iOS26, …) into the dictionary
under source = .history. Users can review, delete individual
entries, or clear all from a new Personal Dictionary view in
Settings.
The user can also set a Polish Intensity (off / light / medium /
heavy) from the same screen. Default is medium, which is what
Typeless and Wispr Flow also use.
## Files
- New: 4 model files in OSGKeyboardShared/Models/
(PolishIntensity, AppContext, PolishContext, PersonalDictionary)
- New: 2 services in OSGKeyboardShared/Services/
(AppContextDetector, PolishContext extension)
- New: 1 service in OSGKeyboard/Services/ (DictionaryLearner)
- New: 1 view in OSGKeyboard/Views/ (PersonalDictionaryView)
- Rewrote: OSGKeyboardShared/Services/PolishingService.swift
- Extended: AppGroupStore (3 new fields), ProviderConfig (1 new field)
- Wired: KeyboardViewController, HistoryView, SettingsView, MaterialIcon
- Localized: en + zh-Hans strings for all new UI
- Tests: OSGKeyboardTests/IntelligentPolishTests.swift (16 tests)
## Verification
- All new code follows the existing Sendable / strict-concurrency
patterns (the keyboard extension stays within its 60MB sandbox;
the polisher remains an actor; @MainActor is applied to the
learner and the settings UI).
- Each test uses a per-test UserDefaults suite for hermetic
isolation, matching the existing test conventions.
- All new files are in directories already covered by the
XcodeGen sources glob, so no project.yml change is needed.
## Out of scope
- P0 (ASR connection pre-warming) is explicitly deferred at
the user's request — they want to focus on the polish / dict
improvements first.
- The Cloud polish (WebSocket) work is not touched.
## Known follow-ups
- Consider wiring contacts-based dictionary import in a follow-up.
- Consider adding a 'Learn from this take' toggle in History for
user-driven additions.
- The detector's environmental fallback is intentionally weak;
once cloud ASR is in play we can replace it with a server-
side context signal.
Sync project documentation with the current v0.2.1 implementation
(Tap-to-toggle recording, 5-step onboarding, Flow session model,
SpeechAnalyzer-only local engine, deepseek-v4-flash default).
README.md / README.zh.md
- Replace 'press-and-hold / 按住说话' with the v0.2.0+ tap-to-toggle
interaction; add the 60-second per-take cap to feature bullets.
- Update 3-step onboarding to the real 5-step flow.
- Correct architecture diagram: AudioCaptureService in the extension
is legacy/unused, PolishingService lives in OSGKeyboardShared, and
the FlowSession* / LiveDictation* services are now in the tree.
- Replace the legacy single-pipeline data flow with the actual Flow
session data flow (keyboard -> App Group -> host app -> chunked
ASR -> LLM polish -> App Group -> insertText).
- Move the stale 'main -> 0.2 branch rename' banner into a project
status section under the new 'v0.2.1' badge.
- Expand 'Known limitations' with 60s/3min caps, 60MB sandbox note,
URL scheme caveat, and the v0.2.0->v0.2.1 on-device LLM rollback.
- Add a 'Development' section pointing to tests, CI, logging policy.
- Add version badge, license link, privacy policy link.
docs/index.html (GitHub Pages landing)
- Mirror the same copy fixes in both EN and ZH i18n tables
(verified: 50 keys each, no missing translations).
- Grow feature grid from 6 to 8 cards (add 'Flow session' and
'Local + cloud polish'), grow steps from 3 to 5.
- Bump footer to 'v0.2.1 · source available, non-commercial',
add License link.
docs/privacy.html + docs/privacy/index.html
- Update 'Last updated' to July 3, 2026, tag v0.2.1.
- Add explicit iOS 26+ requirement, Flow session explanation,
children's privacy section, policy-changes section, license
reference, and rocky.hk@gmail.com contact.
- Make the 'no raw audio upload to any server' claim explicit.
Verification
- grep confirms no leftover 'press and hold' / '按住' / 'press to'.
- HTML structure validated (well-formed on all three pages).
- EN/ZH i18n keys are symmetric (50 each).
Rework the action cluster to a mic-above-bottom-row layout, add left/right
handedness setting that swaps delete and return, and keep the screen awake
during Flow recording sessions.
Add preset-driven polish scenarios (Settings, onboarding, ScenarioChip)
with ScenarioPrompt and style directives; drive keyboard height from
content (240pt) and use viewIsAppearing encapsulated-height offset for
smoother keyboard switches; remove redundant StatusBadge and hide system
dictation via hasDictationKey.
- Add PreconfiguredKeys.swift: placeholder constant for the DeepSeek
API key the local engine uses. DEBUG build asserts at launch when
the placeholder is still in place so nobody ships an always-401
build by accident. Replace TODO_FILL_LATER_DEEPSEEK_KEY before
distributing.
- PolishingService.polishRemote:
* Adopted (A) path: when effective provider is DeepSeek, take the
apiKey from PreconfiguredKeys.deepseek instead of store.apiKey.
Refuse the round-trip with PolishError.missingAPIKey when the
placeholder is still in place.
* Adopted fix for pre-existing typo: store.model.isEmpty ?
preset.defaultModel : store.model (the right-hand side was
defaultModel on both branches, silently ignoring the user's
custom model field).
- LocalEngineSettingsRows.LocalModelsGroup:
* Dropped the inline 'uses DeepSeek' caption — it added visual
weight without telling the user anything they couldn't infer.
* Translation row now lives inside the group so the local engine
reads as one cohesive card.
* Group owns its surface card chrome (palette.surface + rounded
border) — callers no longer need to wrap it externally.
- SettingsView:
* languageAndModelsSection: dropped the trailing TranslationPickerRow
and the inner LocalModelsGroup (now lifted to its own section).
* New localEngineSettingsSection: renders only when
engineMode == 'local', wrapping LocalModelsGroup with a
'settings.localEngine.title' header.
* body: added the new section between languageAndModelsSection and
the cloud-only systemPromptLinkSection.
- Localizable.strings (en + zh-Hans):
* Removed settings.localModels.cloudPolish.caption (no references).
* Added settings.localEngine.title = 'Local engine' / '本地引擎'.
DeepSeek key pre-fill tripwire + local-engine UI cohesion. No main
merge, no PR.
- Add LLMProvider.isUserSelectable (default true) and filter
ProviderPickerSection on it; next pass can hide non-user presets
(e.g. a future DeepSeek key-preset) without changing call sites.
- KeyboardRootView: hide TranslationChip when off (matches user's
mental model of an opt-in feature), drop the 'warming' branch
(Qwen3 download UX was removed with the backend in v0.2.0), unify
chip pill height to minHeight 28 + vertical 6 for visual rhythm
across all topbar chips.
- TranslationChip: drop isLocal warning path — both engines now
run the translate-and-polish step (local routes through DeepSeek
via ProviderConfig.localModeProviderId).
- OnboardingView: cloud engine branch now wraps the translation
row in the same surface card chrome as the local branch.
- Strings: drop keyboard.models.warming (no longer referenced).
DeepSeek key pre-fill deferred to a follow-up.
UI refinements on top of the translation pipeline (feature/translation@HEAD):
1. Onboarding engine page now hosts a translation row.
APISetupPage renders the same TranslationPickerRow used in the
language tab, so first-time users can pick a target language
before they ever see the keyboard. Same persisted bindings; same
'needs cloud' hint when the local engine is active.
2. Local engine hides the provider / API card unconditionally.
Removed the 'local + cloud polish on → show API fields' branch
from SettingsView. Provider/base URL/API key/model controls have
no use in local mode (translation is cloud-only anyway), and
exposing them invited users to fill in a DeepSeek key they
can't use.
3. 'Cloud polish after ASR' toggle loses its long subtitle.
The descriptive copy in LocalEngineSettingsRows.cloudPolishRow
was a wall of text that explained things visible elsewhere in
Settings. Title + switch is enough; the CloudPolishDisclosureBanner
(rendered by EnginePickerSection when cloud is active) already
covers the 'this sends text to your API' disclosure.
4. Translation row becomes a single dropdown with a 'Don't
translate' default.
TranslationPickerRow replaced with a one-row Menu picker:
'不翻译 / English / 中文 (简体) / 中文 (繁體) / 日本語 / 한국어 /
Français / Deutsch / Español / Русский / Português'.
'不翻译' maps to translationEnabled=false; any locale maps to
translationEnabled=true + translationTargetLocaleId=<id>.
TranslationLanguageCatalog gains an 'off' sentinel so the picker's
single binding stays a plain String.
5. Language tab reorder.
SettingsView.languageAndModelsSection: ASR locale ('识别语言')
now sits above the local-models block; translation row sits at
the bottom. The reading order follows the pipeline direction
(input → post-processing → post-post-processing).
Localization:
- 'settings.translation.title' → '翻译' / 'Translation'
- new 'settings.translation.off' / 'settings.translation.hint.needsCloud'
- dropped unused subtitle / target-language keys
xcodebuild scheme=OSGKeyboard config=Debug destination=iPhone 17
Simulator: BUILD SUCCEEDED (0 warning, 0 error).
Adds an opt-in translation pipeline that reuses the existing
PolishingService + LLMClient + AppGroupStore chain. Translation is
implemented as a new PolishMode (.translate(targetLocaleId:)); all
existing call sites are unchanged.
Settings:
- New TranslationPickerRow in the language tab (Toggle + 10-locale
picker: en/zh-Hans/zh-Hant/ja/ko/fr/de/es/ru/pt), persisted to the
App Group so the keyboard extension can read it during live dictation.
- 5 new strings per language (en + zh-Hans).
Keyboard:
- New TranslationChip on the top bar to the right of LocaleChip;
same Menu pattern, lets users toggle or quickly switch target
language without leaving the keyboard.
- PolishingService dispatches .translate with a parameterised prompt
(en/zh variants selected by provider id); PolishingService.error
gains a translationNotAvailable case so local-engine users get a
clear inline warning when the toggle is on but cloud is off.
- 6 new strings per language (en + zh-Hans) for the chip + banner.
Local engine policy:
- Translation is cloud-only by design (local engine stays ASR-only
to honour the no-roundtrip promise). Chip shows a 'cloud required'
state and raw transcript still inserts on failure — no data loss.
Build:
- OSGKeyboardShared adds TranslationLanguage enum (10 locales) and
TranslationPrompt factory.
- 4 new files, 9 modified. xcodebuild scheme=OSGKeyboard
config=Debug destination=iPhone 17 Simulator: BUILD SUCCEEDED
(0 warning, 0 error).
Also pins DEVELOPMENT_TEAM in project.yml for TestFlight uploads
(3 targets; Team X329MZU23S).
Rolls back the v0.2.0 Qwen3 CoreML on-device ASR stack and replaces the
'local engine' UX with iOS 26 SpeechAnalyzer + DictationTranscriber only.
The 'Cloud polish after ASR' toggle (ProviderConfig.localModeCloudPolishEnabled)
lets users opt into a post-ASR DeepSeek round-trip from the local engine.
Defaults to off so the local engine stays genuinely local. New PolishError.missingAPIError
surfaces an inline 'fill in your key' warning when the toggle is on but the
Keychain is empty. DeepSeek preset default model bumped to deepseek-v4-flash.
Deleted:
- OSGKeyboard/ThirdParty/Qwen3Speech/ (74 files, ~16k LoC)
- OSGKeyboard/Services/ModelManager.swift (492)
- OSGKeyboard/Services/OnDeviceModelWarmup.swift (197)
- OSGKeyboard/Services/Qwen3ASRService.swift (257)
- OSGKeyboard/Services/ModelDownloadSourcePicker.swift (126)
- OSGKeyboard/Views/OnDeviceModelsView.swift (184)
- OSGKeyboard/Views/DownloadConfirmSheet.swift (96)
- OSGKeyboardShared/Models/OnDeviceModel.swift (140)
- OSGKeyboardShared/Services/OnDeviceModelStatus.swift (104)
- Qwen3ASRServiceProvider registration in OSGKeyboardApp
- Qwen3Speech package declaration in project.yml
- 5 .qwen3ASR enum / branch reference sites in HomeView, OnboardingView,
LocalEngineSettingsRows, FlowSessionManager, ASRService, EngineServiceLabel
- Two pre-existing Swift 6 strict-concurrency errors in
LiveDictationController + FlowSessionManager (the weak [weak self] in
detached-task MainActor.run blocks) that were blocking clean builds
Added:
- LocalModelsGroup: 'Built-in iOS SpeechAnalyzer' badge + 'Cloud polish
after ASR' Switch toggle
- PolishingService: honour localModeCloudPolishEnabled; new .missingAPIKey
error case with localised warning
- AppGroupStore.localModeCloudPolishEnabled (mirrored into App Group
so the keyboard extension honours the toggle during live dictation)
- SettingsView: show provider/api sections when local-mode cloud polish
is on so the user can paste a DeepSeek key
- FlowSessionManager: route through PolishingService for local + polish-on
flow; translate missingAPIKey into a polished warning
- KeyboardViewController: handle PolishingService.PolishError.missingAPIKey
in the keyboard-side live polish path
- CHANGELOG v0.2.1: documents the rollback + new toggle
- README.md / README.zh.md: engine matrix section, data flow note
Verified: xcodebuild -scheme OSGKeyboard -destination 'generic/platform=iOS Simulator'
build succeeds under SWIFT_STRICT_CONCURRENCY=complete.
Replace MLX GPU inference with CoreML bundles so transcription continues
while the host app is backgrounded. Adds model download and warm-up,
vendored Qwen3Speech, and updates onboarding, settings, and copy for the
~1.6 GB CoreML package (iOS 18+).
Replace MIT with a restrictive source-available license and align README/docs copy. Add cloud polish acknowledgment, privacy manifest updates, voice history disclosure, export compliance metadata, and support links for App Store readiness.
Streamline the five-step flow with smarter skip logic, a centered welcome intro, clearer zh copy, keyboard setup detection via the extension, and home tips when permissions are still missing after onboarding completes.
App Store Connect requires screenshots at 1290x2796 (6.7") and
1179x2556 (6.1") for the iPhone 17 Pro Max / 17 / Pro lineup,
with at least 3 and at most 10 images per size. The repository
previously had no screenshots in docs/.
This commit adds:
- scripts/generate_screenshot_placeholders.py - PIL-based
generator that produces 5x 6.7" + 5x 6.1" placeholders with
status bar, mock device frame, keyboard mock, and headline /
subtitle text.
- docs/screenshots/6.7/ and 6.1/ with 5 placeholder PNGs each
(10 total, all at the correct Apple-mandated dimensions).
- docs/screenshots/README.md explaining the dimensions,
generation script, and how to capture real Simulator
screenshots via `xcrun simctl io booted screenshot`.
- docs/APPSTORE_METADATA.md - a single source of truth for
every field in App Store Connect: name, subtitle, URLs,
pricing, description (<=4000 chars), promotional text
(<=170 chars), keywords (<=100 chars), release notes
(<=4000 chars), what's new, App Privacy answers
(Data Not Collected), encryption declaration, and reviewer
notes.
- AUDIT_APPSTORE.md - the v0.1.2 pre-launch audit report
with 6 P0 items, 5 P1 items, 4 P2 items, and a 9-item
"cross-cutting observations" section addressing the items
in the original brief that turned out to be non-issues
(CFBundleURLTypes in keyboard ext, NSSupportsLiveText,
AppGroup fatalError inconsistency, AudioCaptureService
interaction with AVAudioSession, ASRService SFSpeechRecognizer
references). Includes an Appendix A with the local
xcodebuild build + test results.
Important: the placeholder PNGs are intentionally bland - they
exist so App Store Connect accepts the dimensions. They MUST
be replaced with real Simulator screenshots before the actual
upload, see docs/screenshots/README.md for the workflow.
Refs: AUDIT_APPSTORE.md P0-5
Both classes are kept in v0.1.2 (they serve the in-app keyboard
preview and the one-shot host-app dictation handoff path), but
their primary role was taken over by FlowSessionManager +
FlowContinuousCapture in v0.1.1. Add a STATUS section to each
file header explaining the current role and listing the call
sites, so future readers do not assume they are dead code and
try to delete them (which would break PreviewASRController and
KeyboardPreviewSheet).
Refs: AUDIT_APPSTORE.md P0-3
Apple's annual encryption self-classification questionnaire
pops up for every build that does not declare this key. All
network calls in OSGKeyboard are HTTPS (the LLM polish call
hits a user-configured OpenAI-compatible endpoint) and Apple
classifies standard HTTPS as exempt under EAR Category 5 Part
2 Note 4, so the app does not need an encryption registration.
The Info.plist properties live in project.yml (XcodeGen), which
is what xcodebuild actually consumes. The Info.plist stub on
disk is regenerated from project.yml by xcodegen and is
gitignored in spirit (it is checked in for tool compatibility
but XcodeGen treats it as the canonical source).
Verified by:
plutil -p .derivedData/.../OSGKeyboard.app/Info.plist | grep -i encrypt
-> "ITSAppUsesNonExemptEncryption" => 0
Refs: AUDIT_APPSTORE.md P0-4
OSGKeyboardExt/Services/AudioCaptureService.swift was a true
duplicate of OSGKeyboardShared/Services/LiveDictationController
that was replaced by FlowContinuousCapture in v0.1.1. It is
referenced from no Swift code (only from comments), so removing
it has zero runtime impact.
LiveDictationController and DictationBridge are kept (still used
by PreviewASRController, DictationCaptureView, KeyboardPreviewSheet
and the host-app one-shot dictation handoff path), but their file
headers now state the v0.1.2 status explicitly so future readers
do not try to "modernize" them away.
Verified by:
grep -rn "AudioCaptureService" --include="*.swift" . -> no hits
xcodebuild build -> BUILD SUCCEEDED
Refs: AUDIT_APPSTORE.md P0-3
The macos-14 runner image no longer ships Xcode 16.0; the default
is now Xcode 16.4. Pin the runner to that version explicitly so
`xcode-select -s /Applications/Xcode_16.4.app` succeeds.
Also drops the build matrix `xcode` field, since the runner only
has one Xcode available and we do not test across multiple Xcode
versions on the audit/appstore-prep branch.
Refs: AUDIT_APPSTORE.md P0-1
Resolve TestFlight ITMS-91056 by correcting privacy manifest keys and reason arrays. Add left-aligned page headers with circular confirm buttons, keep the keyboard surface transparent, and improve unselected dock icon contrast in light and dark mode.
Renew voice sessions while the host app stays foreground, auto-start flow from the keyboard with a Start action, and prevent leading audio loss via pre-roll buffering and faster signal polling.
Introduce MainTabView with history and Liquid Glass dock; refresh home and
onboarding layouts; unify accent green and provider localization; refine
settings/API rows; redesign keyboard mic and flanking controls; set Utility
iPhone-only targets and Flow session reliability fixes.
XcodeGen merged Localizable.strings into the main app only, so the keyboard
extension had no .lproj files and NSLocalizedString returned raw keys. Use a
dedicated Keyboard.strings table and explicit buildPhase: resources entries.
Icon Composer AppIcon.icon must not be compiled alongside AppIcon.appiconset;
actool crashes with a nil insert error. Use appiconset only and exclude .icon
bundles from the target.
Refresh mic/speech status when returning from TCC dialogs, persist onboarding
page across Settings trips, load extension strings from .lproj bundles via
ExtL10n, and keep flow result polling alive across host-app jumps.
Remove KeyboardL10n hard-coded fallbacks in favor of ExtL10n and extension
Localizable.strings. Add Flow session expiry hints, Darwin cross-process
notifications, session monitor on the keyboard, and app icon on GitHub Pages.
Migrate keyboard dictation to continuous Flow sessions with auto-start,
tap-to-toggle recording, 60s countdown, five-step onboarding, and App Group
IPC. Add docs/ GitHub Pages site with en/zh privacy policy for App Store compliance.
Keep repository messaging consistent with the implemented iOS 26 SpeechAnalyzer path, and remove mixed hardcoded copy by routing remaining UI/error text through localized string keys.
The previous build crashed on the first record-tap with:
*** Terminating app due to uncaught exception
'com.apple.coreaudio.avfaudio', reason:
'Failed to create tap due to format mismatch,
<AVAudioFormat: 1 ch, 48000 Hz, Float32>'
The crash surfaced at `[AVAudioNode installTap:...]` but the
*root* cause was one frame deeper, in the underlying
`AURemoteIO::enable` call:
AURemoteIO.cpp:1135 failed: -10851
(enable 1, outf< 2 ch, 0 Hz, Float32, deinterleaved>
inf< 1 ch, 48000 Hz, Float32>)
`AURemoteIO` is a two-direction Audio Unit (input + output).
`installTap` triggers `AURemoteIO::enable(1)` which tries to
initialize *both* directions at once. On the iOS Simulator, the
output direction reports a `0 Hz` "speaker" because the
simulator has no real speaker, and `enable` fails with
`kAudioUnitErr_FormatNotSupported` (-10851). The error then
bubbles up through `installTap` as the misleading "format
mismatch" — the input format we passed was correct, but the
output side broke the whole `enable` call.
The previous code asked the audio session for `.playAndRecord`
mode, which requires the output direction to be enabled. On a
real device, the speaker is 44100/48000 Hz and `.playAndRecord`
works. On the simulator, it traps.
Fix (two parts):
1) `start()` audio session now uses `.record` instead of
`.playAndRecord`. The preview never plays back audio — it
just records from the mic and hands the buffers to
`SpeechAnalyzer`. `.record` skips the output direction
entirely, so the simulator's 0 Hz speaker isn't a problem.
The real keyboard extension (`OSGKeyboardExt`) keeps
`.playAndRecord` because it runs on real devices and may
want to play click sounds / haptic feedback — only the
preview needs the simulator-friendly category.
2) `startEngineAndASR` adds a pre-flight check: refuse to
call `installTap` when the input bus reports
`sampleRate == 0` or `channelCount == 0`. The `installTap`
failure mode is an `NSException` (Objective-C), not a Swift
`Error` — `try`/`catch` can't intercept it, so the only safe
defence is to never call it with a placeholder/unconfigured
bus. We saw `0 Hz` inputs on the simulator when the host
mic permission wasn't granted to CoreSimulator, and on
devices with an unexpected audio-session state. The check
sets a clear `.error` phase ("Microphone unavailable (hw
format X Hz / Y ch)") instead of trapping the app.
Why this also unblocks voice-to-text: the previous
"no transcription" symptom was just the crash at step 7 of
the start pipeline — the ASR never saw a single audio frame
because the tap never installed. With the crash fixed, the
`SpeechAnalyzer` pipeline can now actually receive buffers
and emit `.partial` / `.final` events. Whether the *content*
of those events is meaningful depends on the simulator's mic
quality and the chosen locale (both separate concerns from
this fix).
Tests: 27/27 pass (no behavioural test for the pre-flight
check — the check is 4 lines of obvious defence, and
testing it would require either mocking the audio engine
or installing CoreSimulator's host-mic-bridge, neither of
which is worth the cost).
Build: BUILD SUCCEEDED.
🤖 Generated with Claude Code
Two design cleanups, both for visual polish on the record disc
(the hero control of the keyboard preview AND the live keyboard
extension). The shadows were a leftover from the iOS 18 mockup
that visually fought the brand-green disc; the processing-state
spinner was the system default at 1.1x, which on a 96-120pt disc
reads as "a dot in the middle" rather than "an active process".
Changes:
1) Remove the disc body's drop shadow on BOTH surfaces.
- `KeyboardPreviewStub.swift:244` — `.shadow(color: .black.opacity(0.4),
radius: 10, y: 6)` deleted. Hardcoded dark shadow on a
light surface, completely theme-blind (looked like a 2009
skeuomorphic button).
- `RecordButton.swift:112` — `.shadow(color: .black.opacity(0.45),
radius: 14, y: 8)` deleted. Same problem on the real
keyboard's disc, plus the shadow competed with the existing
"halo" radial gradient and the outer breathing ring.
The internal "ambient glow" (radial green gradient inside the
disc on idle) is NOT touched — that's brand colour, not a
drop shadow, and matches the live keyboard's idle accent.
2) Scale up the processing-state spinner on BOTH surfaces.
- Preview: `ProgressView().tint(.white).scaleEffect(1.1)` →
`ProgressView().progressViewStyle(.circular).tint(palette.textPrimary)
.scaleEffect(2.5)`.
- 1.1x (~22pt) → 2.5x (~50pt) = 52% of the 96pt disc.
- Tint changed from hardcoded `.white` to `palette.textPrimary`
because the processing-state disc gradient is
`surfaceElevated → surface` (light grey in light mode);
a white spinner on a light grey disc is invisible regardless
of scale.
- Real keyboard: `ProgressView().scaleEffect(1.2)` →
`ProgressView().scaleEffect(2.5)`. Same absolute spinner
size (~50pt) — 42% of the 120pt disc — so the user gets the
same visual weight whether they're looking at the in-app
preview or the live keyboard. Tint was already
`palette.textPrimary` so no change there.
Proportion rationale: ~50% of the disc diameter is the sweet
spot. 100% would crowd the disc's edge, <30% reads as
"decoration" rather than "process". 50pt absolute size matches
the same spinner across both surfaces — same weight, different
disc sizes — so the user doesn't have to re-learn the visual
language when they switch from preview to real keyboard.
Tests: 27/27 pass (no behavioural change, just visual).
Build: BUILD SUCCEEDED.
🤖 Generated with Claude Code
The previous code path for the keyboard preview's ASR controller
cancelled the consumer task at the exact moment it closed the
audio stream:
asrTask?.cancel() // ← kills the .final consumer
asrTask = nil
...
bufferContinuation?.finish() // tells ASR "no more audio"
The cancellation cascaded: the for-await on the events stream
exited → the AsyncStream's `continuation.onTermination` fired →
ASR.cancel() ran → producer task was marked cancelled → the
producer's `if !Task.isCancelled { yield(.final) }` guard
suppressed the .final event. Net result: nobody told the UI to
leave `.processing`, and the disc sat there forever.
Fix (4 changes):
1) `stop()` no longer cancels the consumer. The consumer task
exits naturally when the events stream finishes, sees the
`.final` event the producer still yields, and transitions
the phase out of `.processing`. This is the primary fix.
2) `start()` cancels any leftover `asrTask` at the entry point
as a safety net — covers the "user smashes the disc twice
quickly" race where a previous consumer is still draining.
3) `stop()` schedules a 3-second safety-net Task: if the ASR
pipeline never produces a `.final` (analyzer hang, system
glitch), force the phase back to `.idle` so the user isn't
stuck. Normal recordings complete well under 3 seconds, so
the timeout is only hit on the unhappy path.
4) `KeyboardPreviewSheet` adds `.onDisappear { asr.stop() }`
so closing the sheet mid-recording releases the
AVAudioSession and mic. `stop()` is idempotent (no-op on
non-recording phases), safe to call here.
State machine: `phase = .processing` now has TWO transition
paths out — the consumer receiving `.final` (fast path) and
the 3-second safety net (fallback). Both are required; the
fast path is the common case, the fallback is the
"guaranteed-progress" guarantee.
Testability: `asrTask` was `private`; relaxed to `internal` so
the regression test in
`OSGKeyboardTests/PreviewASRControllerStateTests.swift` can
install a known consumer task and assert `stop()` does not
cancel it. The class is `@MainActor` so Swift 6 isolation
rules still prevent production code outside the class from
racing on it.
Tests:
- `testStopDoesNotCancelConsumerTask` — primary fix regression.
- `testStopIsIdempotent` — `.onDisappear` after a manual stop
doesn't misbehave.
- 27/27 tests pass (25 existing + 2 new).
- BUILD SUCCEEDED.
🤖 Generated with Claude Code
The keyboard preview crashed on first record with a
`__abort_with_payload` deep inside Speech's
`DictationTranscriber`. The disassembly surfaced three
preconditions checked before a `brk #0x1`:
+620 "Audio sample data must be 16-bit signed integers"
+848 "Multi-channel audio is not supported"
+1072 "Client info not fully initialized"
We hit the first one. `DictationTranscriber` (iOS 26's new
`SpeechAnalyzer`-backed engine) is strict about its input
format: only Int16 PCM, not the Float32 PCM that the iOS 18
`SFSpeechRecognizer` path accepted. Our audio-tap and
`AudioBufferSnapshot.samples: [Float]` are Float32 all the
way down — that was the SFSpeech shape, and the previous
`AppleSpeechASR` adapted internally. With iOS 26 as the
deployment target, the only ASR backend is
`SpeechAnalyzerASR`, and the conversion needed to happen at
the `AnalyzerInput` boundary.
Fix:
- `transcribe` builds the `AVAudioFormat` as
`.pcmFormatInt16, 16 kHz, 1 ch, interleaved: true` (the
canonical layout for Int16 Speech input).
- `makeInputStream` runs the per-sample conversion
`Int16(round(clamp(s * 32767, -32768, 32767)))` into the
`AVAudioPCMBuffer`'s `int16ChannelData[0]`. The explicit
clip is required (a `s == 1.5` from a gain-overflow at the
audio-engine boundary would otherwise wrap to a negative
Int16 after the implicit truncation). `round()` (not
truncate) preserves DC balance — `0.5` quantises to
`+16384`, not `+16383`, matching what audio DAWs expect.
- The conversion helper is exposed as
`ASRServiceFactory.convertFloat32ToInt16` so unit tests
can lock the math without instantiating the full pipeline.
Why not change `AudioBufferSnapshot` to `[Int16]` instead
(see earlier first-principles discussion): the snapshot is a
transport format that both `AudioCaptureService` (in the
ext) and `PreviewASRController` (in the main app) produce.
Float32 is the natural shape coming out of `AVAudioEngine`,
and pushing the conversion to the ASR service keeps the
transport contract platform-agnostic — a future second
backend with different format needs can have its own
adaptation without dragging everyone else.
Tests:
- `testFloat32ToInt16EdgeCases` — 0, ±1, ±0.5, ±1.5
(gain-overflow case).
- `testFloat32ToInt16RoundTrip` — quantisation step is
1/32767 (so the asymmetric Int16 range is honoured: -32768
has no exact Float source).
- `testFloat32ToInt16Empty` — `sourceCount == 0` with nil
pointers is a no-op (function guards on count before
dereferencing).
- All 25 tests pass (22 existing + 3 new).
- BUILD SUCCEEDED.
🤖 Generated with Claude Code
Two related cleanups the user asked for in one shot:
1. iPhone-only is now enforced at every target — Mac Catalyst and
visionOS were never configured in `project.yml`, but the
`OSGKeyboardShared` framework and the two test bundles were
still defaulting to `TARGETED_DEVICE_FAMILY = "1,2"` (iPhone +
iPad). All four targets now explicitly set `"1"`. SDK is
`iphoneos` for everyone, no `xros` / `macosx`.
2. Deployment target bumped from iOS 18.0 to iOS 26.0 across the
board (`project.yml` + the ext's per-target setting). With
iOS 26 as the floor, the iOS 18–25 SFSpeechRecognizer path
became dead code and several `#available` checks became
always-true. Removed:
- `AppleSpeechASR` (the entire SFSpeechRecognizer-based ASR
backend) and the `#available(iOS 26.0, *)` factory branch.
`ASRServiceFactory.make()` now returns `SpeechAnalyzerASR()`
directly. SpeechAnalyzer is always fully on-device, which
also made the `requiresOnDevice` flag meaningless.
- `requiresOnDevice` from the `ASRService.transcribe` protocol
signature, from `ProviderConfig`, `AppGroupStore`,
`KeyboardState`, `AppGroupPersistor`, and the ext's
`KeyboardViewController` (`state.requiresOnDevice`,
`state.setRequiresOnDevice`, `persistRequiresOnDevice`).
- `#available(iOS 17.0, *)` branch in
`PreviewASRController.requestMicrophonePermission` and the
ext's `PermissionManager.requestMicPermission` — both now
just call the iOS 17+ `AVAudioApplication` API directly.
- The `else` (iOS 18–25) branch in `SettingsView.asrEngineRow`
— the on-device-only toggle is gone, the row is a static
"SpeechAnalyzer active" badge. Same for the `else` branch
in `EnginePickerSection.localSubtitle`.
- `makeMicAuthHandler` (the iOS < 17 mic permission callback
wrapper) from `PreviewASRController`.
No `#available` / `@available` checks remain in the codebase
except for the SpeechAnalyzer class itself (now unnecessary
too, but kept for clarity — `AVAudioApplication` and
`SpeechAnalyzer` are both iOS 17+ / iOS 26+ respectively,
and the deployment target of 26 makes the explicit
`@available` redundant; I left the SpeechAnalyzer class
un-`@available` and removed the `@available(iOS 26.0, *)`
decoration since it's no longer needed).
The Keyboard ext's existing ASRService usage
(`asr.transcribe(stream:locale:)`) is unchanged at the
call-site level — just the third argument is gone.
3. Updated `info.plist` UISupportedInterfaceOrientations is
already `[UIInterfaceOrientationPortrait]` only, which is
correct for an iPhone-only app; no change needed.
Build: BUILD SUCCEEDED.
Tests: 22/22 pass.
Verified: `TARGETED_DEVICE_FAMILY = 1` on all four targets,
`SDKROOT = iphoneos` on all four.
🤖 Generated with Claude Code
1) Preview chips weren't actually buttons.
`modeChip` and `localeChip` in `KeyboardPreviewStub` were
decorative HStacks — no `Button`, no action, no callback. The
chevron-down glyph made them *look* like pickers, so a user
tapping them got nothing. The screenshot the user shared
("润色 ▾" / "中文(简体) ▾") shows exactly that surface.
Fix: wrap each chip in a `Button(action: ...)` with
`.buttonStyle(.plain)`. The stub now takes `modeId`, `localeId`,
`onModeCycle`, `onLocaleCycle` and the sheet's `cycleMode` /
`cycleLocale` advance the config:
- mode cycles [off → transcribe → polish] (mirrors Settings)
- locale cycles [auto → zh-Hans → zh-Hant → en-US → ja-JP → ko-KR]
Mid-recording locale switches call `asr.stop()` because ASR
sessions are bound to the locale they were started with.
`modeId == "off"` also stops any in-flight recording so the
disc isn't recording into a mode that won't insert.
The mode chip's icon also follows the mode (mic.slash /
mic / wand) as a redundant visual cue, and both chips get
accessibility labels (preview.modeChip.cycle /
preview.localeChip.cycle) so VoiceOver users can use them.
2) Onboarding's "Next" stays enabled when local engine is picked
but no API key is filled in. Root cause: `ProviderConfig.isConfigured`
checks `!apiKey.isEmpty && !baseURL.isEmpty && !model.isEmpty` —
it never asks whether the user *needs* a key. The local engine
(on-device ASR) doesn't round-trip through the LLM, so an
empty key on the local path is correct, not a configuration gap.
Fix: short-circuit `isConfigured` to `true` when
`engineMode == "local"`. The onboarding "Next" button is
already disabled on the API page when `!isConfigured`; this
just makes the gate respect the engine choice. New test
`testIsConfiguredTrueForLocalEngineWithoutAPIKey` locks the
behaviour in (local → true, cloud → false, flip back).
3) Add the two new accessibility keys to all four
`Localizable.strings` files (en + zh-Hans, main app + ext)
so VoiceOver and the cycle button labels resolve in both
languages.
Build: BUILD SUCCEEDED.
Tests: 22/22 pass (1 new).
🤖 Generated with Claude Code
Same `dispatch_assert_queue_fail` family as `3e3eb0f` (audio tap),
but on a different code path: `SFSpeechRecognizer.requestAuthorization`.
The crash signature was the giveaway:
closure #1 in closure #2 in PreviewASRController.start(locale:)+96
thunk for @escaping @callee_guaranteed (@unowned
SFSpeechRecognizerAuthorizationStatus) -> ()
__TCCAccessRequest_block_invoke_8
`closure #2` is the `withCheckedContinuation` body, `closure #1` is
the requestAuthorization callback. Both end up attributed to
`start(locale:)` — which is `@MainActor` — because Swift 6 *inlined*
the `nonisolated static func` helper (`e8a0310`'s fix) back into the
caller. After inlining, the inner closure is retyped in the
caller's @MainActor context, the runtime asserts main-queue, and
TCC's reply queue (where the callback actually fires) trips the
assert.
Why `nonisolated static` alone is not enough: the optimizer
aggressively inlines small async helpers, and when it does, the
isolation of the closures inside the body re-infers from the
inlined context. Same trap as the audio tap — function bodies
defined inside a `@MainActor` method can't escape @MainActor just
because their enclosing function was marked nonisolated. The
*function-reference* pattern (the fix in `3e3eb0f`) is what works:
SFSpeechRecognizer.requestAuthorization(
Self.makeSpeechAuthHandler(continuation: cont)
)
`makeSpeechAuthHandler` is `nonisolated static`, returns the
`(Status) -> Void` handler, and crucially the closure body is
*constructed* in a nonisolated context. When Swift tries to inline
the callback, it has nothing to inline — only a function reference
to a non-inlinable (because it crosses an isolation boundary
implicitly) helper. The runtime sees a nonisolated closure on a
non-main queue and is satisfied.
Same treatment for the iOS < 17 mic path (`makeMicAuthHandler`),
which has the same TCC-reply-queue dispatch shape.
Build: BUILD SUCCEEDED.
Tests: 21/21 pass.
🤖 Generated with Claude Code
Same dispatch_assert_queue_fail as the prior SFSpeechRecognizer
crash, but in a different code path: `AVAudioNode.installTap`'s
callback fires on the AVAudioEngine real-time audio thread. The
previous fix (`e8a0310`, extract permission callbacks to
`nonisolated static func`) only addressed the once-and-done
TCC callback — audio taps are *continuous*, so the runtime
reached the audio thread before the user had any chance to
back off the recording.
Root cause: Swift 6 strict concurrency. The closure literal
passed to `installTap` was defined inside a `@MainActor` method,
so the compiler inferred the closure body as `@MainActor`-
isolated. AVAudioEngine calls it from its real-time audio
thread, not main, so `dispatch_assert_queue_fail` fires on
the *very first buffer delivery*. Wrapping the inner state
updates in `Task { @MainActor in ... }` (the prior fix) was not
enough — the runtime checks the OUTER closure's isolation, not
just the inner accesses.
Fix: build the tap body inside a `nonisolated static func` that
returns a function reference. Swift 6 function references
never carry inferred isolation, so the dispatch runtime sees
the closure as non-isolated and is happy to run it on the audio
thread. State updates to `self.level` and the AsyncStream
continuation hop back to main via `Task { @MainActor in … }`,
which is itself safe to invoke from a non-isolated context.
`makeAudioTapBlock` takes the format / converter / sample-rate
values plus two `@Sendable` callbacks (`onMeter`, `onSnapshot`)
and returns a `@Sendable` closure suitable for the installTap
block parameter. No state escapes the audio thread; the only
back-channel is through those callbacks.
Build: BUILD SUCCEEDED.
Tests: 21/21 pass.
🤖 Generated with Claude Code
Five user-flagged issues addressed in this commit. Some of the
uncommitted files belong to a prior agent pass and are included as-is
so this is a clean working tree.
1. Remove globe (nextKeyboard) button from keyboard bottom bar.
iOS already provides a globe key in the system keyboard strip
for next-keyboard switching, so the in-extension one was
redundant. The bar is now: ⌫ (delete) [space] ↩ (return).
Affected: OSGKeyboardExt/Views/KeyboardRootView.swift and
OSGKeyboard/Views/KeyboardPreviewStub.swift (preview mirrors).
2. Fix TabView with .page style auto-jumping on TextField focus.
SwiftUI's `.tabViewStyle(.page(...))` wraps content in a
UIPageViewController, which has a long-standing iOS 18 bug
where the keyboard-showing layout reflow on a TextField focus
is misread as a horizontal swipe — the page jumps back to
step 1 the moment the user starts typing. Replaced the
TabView with a ZStack + conditional view + transition. We give
up swipe-to-page, but Back/Next buttons + page dots are the
canonical onboarding affordance and the user is one tap from
the next page anyway.
3. Onboarding APISetupPage now offers Engine choice (Local vs
Cloud), matching the in-app Settings page. First-run users can
pick the on-device engine and skip the API-key setup entirely.
Extracted the engine section from SettingsView into a shared
`EnginePickerSection` component used by both. Cloud path
keeps the provider + API fields; Local path shows a
"no API key needed" confirmation card.
4. APISettingsCard test-connection error messages are now built
with NSLocalizedString + String.localizedStringWithFormat (for
the interpolated HTTP status / reply preview). Status badge
labels are also NSLocalizedString-backed.
5. iOS-standard localization.
- New `en.lproj/Localizable.strings` and `zh-Hans.lproj/
Localizable.strings` (in both the main app and the keyboard
extension bundles — each .appex has its own bundle).
- `project.yml` sets `CFBundleDevelopmentRegion: en` and
`CFBundleLocalizations: [en, zh-Hans]`; the two .lproj dirs
are added as resources.
- Every `Text("...")` / `Button("...")` / `Label("...")` /
`accessibilityLabel(Text("..."))` in user-facing views was
rewritten to use `Text("key")` (SwiftUI auto-resolves
string-literal `LocalizedStringKey`s against the strings
file) or `NSLocalizedString("key", comment: "")` for
interpolated / dynamic values. The hardcoded
"中 · EN" / "EN · 中" pattern is gone.
- Two helper signatures that took `String` for the title/body
of a row (`footnoteRow`, `sectionHeader`) are now
`LocalizedStringKey` so the row labels are looked up.
- `Label("key", systemImage: "...")` and
`.confirmationDialog(LocalizedStringKey("key"), ...)` and
`.navigationTitle(LocalizedStringKey("key"))` are used where
`String` would just print the key.
- Preview-onboarding `APISetupPage` updated to use the same
engine picker as Settings (issue 3), with a section that
only renders the provider + API fields when the user picks
Cloud.
Verified on iOS 26 simulator with system language set to both
zh-Hans (default) and en: the same screen renders "按住说话,松开
即得润色文字。" / "下一步" in Chinese, and "Hold to talk. Release
for polished text, in any app." / "Next" in English — no
"中 · EN" doubling, no missing keys.
Build: BUILD SUCCEEDED.
Tests: 21/21 pass.
🤖 Generated with Claude Code
Crash: EXC_BREAKPOINT on first call to `requestSpeechAuthorization`.
Backtrace:
closure #1 in closure #2 in PreviewASRController.start(locale:) + 96
thunk for @escaping (@unowned SFSpeechRecognizerAuthorizationStatus) -> ()
__TCCAccessRequest_block_invoke_8
_dispatch_assert_queue_fail
_swift_task_checkIsolatedSwift
Root cause: `SFSpeechRecognizer.requestAuthorization` (and the
iOS < 17 `AVAudioSession.requestRecordPermission`) deliver their
callbacks on a TCC reply queue, NOT the main queue. My previous
commit wrapped those callbacks inline inside `start(locale:)`,
which is `@MainActor`. Swift 6 strict concurrency infers the inner
closure body as `@MainActor`, so as soon as TCC delivers the
callback on its own queue, the runtime's
`_swift_task_checkIsolatedSwift` asserts we're on @MainActor, sees
we're not, and traps.
Fix: extract both permission dances to `private nonisolated static
func` helpers. The helpers have no isolation, the callback
closures defined inside them have no isolation, and
`CheckedContinuation.resume` is itself thread-safe, so the TCC
queue can resume it without dispatching through the main actor.
private nonisolated static func requestMicrophonePermission() async -> Bool
private nonisolated static func requestSpeechRecognitionPermission() async -> Bool
`start(locale:)` now just `await`s those two helpers and proceeds
to the engine / ASR setup on @MainActor as before. No behavior
change; the user-visible flow is identical.
Build: BUILD SUCCEEDED.
Tests: 21/21 pass.
Runtime: app launches and stays running on the simulator without
the EXC_BREAKPOINT that the previous build hit immediately after
tapping the keyboard preview's record disc.
🤖 Generated with Claude Code
Cursor Agent
rocky