Long-press mic runs ASR as an instruction over eligible clipboard text,
with host-confirm recording UI, min-record gate, and light ASR prewarm.
Bump CURRENT_PROJECT_VERSION to 53.
Custom polish styles can opt in to emotion-matched emoji (default off), with
prompt-level opt-in detection so paste-only styles keep model-added emoji.
Also include Volcengine API-Key ASR auth, voice-processing capture, PiP flash
fix, and related keyboard Shift/haptics reliability work.
Redeploy an osg_personal import table on dictionary add/delete/sync so
Chinese, English typing, and ASR share one curated lexicon (next keyboard open).
Retain the external target app for menu-bar paste and polish context, retry
failed middle ASR chunks once, and stop false-positive path/number violations.
Use redacted cursor neighborhood and pause-aware chunks for more natural polish,
validate protected terms with retry/local fallback, and structure bilingual prompts
for consistency and provider prefix caching.
Harden polish so question drafts stay questions, add local density
routing with style-specific degrade, expand fun style packs, and add
end-to-end FlowTrace logging plus offline guard eval scripts.
Unify Bailian/Volcengine/OpenAI realtime streaming, ABE polish routing with
fun styles, and a shared card-page Settings hierarchy; bump to 1.1 (build 32).
- Add FlowKeepAliveMode (Dynamic Island vs PiP) with iCloud sync
- Settings: keep-alive picker; inactivity timeout only for Live Activity
- PiP: waveform sample-buffer controller, mic released between utterances
- PiP sessions have no idle expiry; user closing PiP ends the session
- FlowSessionManager branches hostReady, session start, and utterance paths
- UIBackgroundModes picture-in-picture; bilingual strings and tests
Co-authored-by: Rocky <hkgood@users.noreply.github.com>
- Accumulate utterance PCM in FlowContinuousCapture for batch retry
- Run full-utterance transcribeChunk when stitched final lags partial
- Refactor Mac MLX tail drain to shared FlowUtteranceEndCoordinator
- Add FlowUtterancePCMStore, UtteranceBatchFallbackPolicy, and tests
Co-authored-by: Rocky <hkgood@users.noreply.github.com>
- Add FlowUtteranceEndCoordinator with 350ms silence drain and 150ms post-roll
- Extend FinalChunkRecovery for short/empty final chunks in chunked pipeline
- Snapshot partial at mic stop and guard final transcript in FlowSessionManager
- Unify tail drain presets (iosFlow/macMLX) and expand diagnostics
Co-authored-by: Rocky <hkgood@users.noreply.github.com>
Add shared prompt composition, custom style management, iCloud sync, and native iOS/macOS selection interfaces so users can keep a consistent writing voice across devices.
Replace the Sherpa offline pipeline with native MLX streaming, resilient model downloads, live transcript previews, and supporting tests and documentation.
Add System/Light/Dark preference for iPhone and iPad, restore the iPad
sidebar brand mark, ship a DEBUG seed-demo URL for rich placeholder data,
and refresh GitHub Pages with the device composite, full-bleed pale-green
hero wash, and transparent Mac screenshot chrome.
Add optional StoreKit support tip (features stay free), redesign the
landing page with zh/en + light/dark and product shots, and ship
iPhone/iPad/Mac App Store screenshot sets.
Fix cold-start overlay recursion that overflowed the main-thread stack when
recording began while the ready overlay was still up; also remove temporary
on-screen Flow DEBUG panels after the orange-mic investigation, and land the
macOS overlay/catalog/layout polish plus related Flow recovery hardening.
End Live Activities and release the audio session synchronously on
applicationWillTerminate, and continue macOS local-model install,
onboarding, and settings polish on this branch.
Adds a bundled local ASR model catalog for the macOS app with one-click
Sherpa Qwen3 / SenseVoice downloads (pause/resume, inline actions) and a
shared model storage directory used by MLX Qwen3. Fixes the light-mode
sidebar material and makes the menu-bar icon follow the system appearance
with a refreshed status mark. Renames the built product to OSGKeyboard.app.
Bumps version to 0.5.2 (build 19).
Introduce a standalone macOS menu-bar app (OSGKeyboardMac) that reuses the
platform-agnostic OSGKeyboardShared core: record -> cloud/local ASR -> polish
-> insert. Local mode uses Qwen3-ASR via mlx-swift-asr (macOS 15+, Apple
Silicon); iOS targets stay zero-SPM.
Harden iCloud sync for multi-device correctness:
- Per-field settings merge (appSettings.v2) so concurrent edits no longer
clobber each other's unrelated fields.
- Per-device usage statistics (G-Counter) that sum instead of max().
- Tombstoned dictionary/history merge so deletes propagate and entries can't
resurrect.
- API keys replicate via iCloud Keychain, never iCloud KVS JSON; pulling a
legacy blob without key fields no longer wipes local Keychain entries.
- Add a low-risk "Sync Now" action in Settings.
Fix Flow keyboard mic state: stay orange until the host publishes a real ready
contract, share a single MicVoiceAvailability gate, and self-heal stale
cross-process heartbeat jitter instead of getting stuck.
Extract shared storage (SpeechHistoryStore/UsageStatisticsStore,
ConfigurationStore) into OSGKeyboardShared and add tests for the new
sync/merge logic.
Mirror the personal dictionary through iCloud Key-Value Store with
deterministic merge rules, main-app-only sync UI, and App Group as the
keyboard runtime cache. Add cloud-engine ASR with dictionary bias and
expand the bundled custom language model lexicon.
- fix(keyboard): drop unusable extensionContext.open; report the real
result of UIApplication.open and cancel the start watchdog immediately
on failure with clear "open OSGKeyboard" guidance instead of a 30s spin
- feat(capture): recover from audio route changes (AirPods/wired headset
plug-unplug) and phone-call interruptions by rebuilding the engine/tap
against the live hardware format
- feat(home): add a "Start" button in the Home footer when a session is
inactive and permissions are granted (no-jump manual restart)
- feat(hostapps): expand the return whitelist to 46 apps (Signal, Zoom,
Instagram, X, Douyin, Zhihu, Bilibili, Things, Firefox, ...) with
en/zh-Hans names and synced LSApplicationQueriesSchemes
- feat(island): use a "ready" checkmark instead of a mic glyph while idle
and set a staleDate so an orphaned Live Activity fades after a force-quit
- feat(coldstart): use the template OSG brand mark instead of a system
waveform glyph in the cold-start overlay
- perf(keyboard): rely on Darwin notifications as the primary session
signal and drop the fallback poll from 1s to 3s
- fix(keyboard): remove the redundant "Start" text button next to the
inactive-session hint on the keyboard
- docs: correct the per-take cap to 210s (3.5 min) and document that
force-quitting no longer resurrects the session (tracker + READMEs)
- P0/P1: tail-drain state machine in FlowContinuousCapture with converter
flush; FlowSessionManager awaits drain before finalize
- P2: FlowCaptureTailDrain policy/tracker, pipeline diagnostics, unit tests
- P3: short final chunk merged re-transcription in ChunkedUtterancePipeline
- P4: UtteranceTranscriptStitcher composedSafely fallback; preview path parity
in LiveDictationController
Co-authored-by: Rocky <hkgood@users.noreply.github.com>
Reduce perceived latency from key release to final text:
- Adaptive chunking: 2.5s first chunk + 5s follow-ups so short
utterances start on-device recognition while still recording.
- Session-level ASR warmup and audio-format cache reuse to remove
per-utterance cold-start of SpeechAnalyzer.
- Mirror live pipelined partials to the keyboard transcript line via
a new flow.transcriptionPartial App Group key + Darwin ping.
Also commits the accumulated custom language model, Flow session,
keyboard extension restructure, and Xiaomi MiMo provider work in
progress on this branch.
Batch of in-progress app work from the working tree.
- feat(keyboard): CursorNavigation + CursorDragPad for caret movement;
KeyboardSoundFeedback for system key click sounds
- feat(dictionary): DictionaryAliasGenerator + PersonalDictionaryEntrySheet;
TranscriptPostProcessor quality gate; retire DictionaryLearner
- feat(ui): TabBarVisibility handling; drop PageHeaderRow /
PageHeaderConfirmButton; refresh views and localizable strings
- fix(security): move the hardcoded DeepSeek key out of
PreconfiguredKeys.swift into a gitignored PreconfiguredKeys.local.swift
(seeded from .example by generate-xcodeproj.sh)
- docs(agents): add Conventional Commits versioning + bilingual changelog rules
- chore(gitignore): ignore PreconfiguredKeys.local.swift, .cache/, pycache
Custom language model / lexicon work stays on
feature/custom-language-model-asr. Changelog bullets added under
[Unreleased]; no version bump.
Use ThemedRoot in preview instead of inaccessible ThemePalette().
Add setPersonalDictionary(_:) for let AppGroupStore bindings; use
it from PersonalDictionaryView and DictionaryLearner.
Co-authored-by: Rocky <hkgood@users.noreply.github.com>
AppGroupStore is a struct; property setters are mutating and cannot
be called on a let binding. Add setOnboardingPage and
setHasCompletedOnboarding helpers (like setDetectedAppContext).
Co-authored-by: Rocky <hkgood@users.noreply.github.com>
contains(where:) on String iterates Characters; comparing each to
multi-character strings like "\n " is a type error. Use
tail.contains("\n ") || tail.contains("\t") instead.
Co-authored-by: Rocky <hkgood@users.noreply.github.com>
Two long-standing UX papercuts, fixed without leaving the keyboard:
1. **First-launch onboarding inside the keyboard** — iOS keyboard
extensions *cannot* programmatically switch back to the previous
app after jumping out, so the old flow (jump to host app → user
has to manually navigate back) was 5+ taps of friction. The new
`KeyboardOnboardingOverlay` keeps the user inside the keyboard
for steps 1 (welcome), 2 (mic permission), 3 (speech permission),
and 5 (API key hint). The only step that *must* leave is step 4
("Enable Keyboard"), which jumps to `Settings.app` via
`UIApplication.openSettingsURLString` — on return,
`viewWillAppear` calls `autoAdvancePastKeyboardSetupStepIfNeeded`
which silently advances past that step if the keyboard is now
enabled. Net UX: user types in their app, keyboard walks them
through setup, normal UI appears as soon as setup is done.
2. **Per-app context chip on the keyboard top bar** — the v0.3.0
intelligent-prompt pipeline already adapted tone to detected
context (code/email/chat/document), but without a UI cue the user
had no way to know which mode was active or override the heuristic
when it guessed wrong. The new `AppContextChip` surfaces the
detected context; tap-to-override writes back to
`AppGroupStore.setDetectedAppContext` so the next LLM call
picks up the new tone immediately. Wired into `pressBegan` so
the chip updates in real time as the user types into different
fields.
### Files added
- `OSGKeyboardExt/Views/AppContextChip.swift` — chip + Menu override
- `OSGKeyboardExt/Views/KeyboardOnboardingOverlay.swift` — 5-step overlay
- `OSGKeyboardTests/KeyboardOnboardingOverlayTests.swift` — round-trip + enum surface tests
### Files modified
- `OSGKeyboardShared/Services/KeyboardState.swift`
+ `hasCompletedOnboarding`, `onboardingPage`, `appContext`
+ `setAppContext`, `advanceOnboarding`, `completeOnboarding`
+ `requestMicPermission`, `requestSpeechPermission`, `openSystemSettings`
- `OSGKeyboardShared/Services/AppGroupStore.swift`
+ `hasCompletedOnboarding` / `onboardingPage` accessors (mirror of
`ProviderConfig` keys, so the keyboard extension never has to
instantiate the host-app config)
- `OSGKeyboardExt/KeyboardViewController.swift`
+ action hooks wired (`installStateActions`)
+ `syncOnboardingStateFromAppGroup` / `syncAppContextFromAppGroup`
called on `viewWillAppear` and `loadPersistedConfig`
+ `autoAdvancePastKeyboardSetupStepIfNeeded` for the silent
"jump out → come back" flow
+ `openSystemSettingsFromExtension` opens `Settings.app` via
`UIApplication.openSettingsURLString` (the only system URL
the extension is allowed to open)
+ `detectAndStoreAppContext` mirrors to `state.appContext` so
the chip updates without waiting for `viewWillAppear`
- `OSGKeyboardExt/Views/KeyboardRootView.swift`
+ overlay mounted in `ZStack` over normal UI (animated)
+ AppContextChip in top bar (hidden during onboarding)
- `OSGKeyboardExt/{en,zh-Hans}.lproj/Keyboard.strings`
+ onboarding copy + chip labels (35 keys per language)
### iOS sandbox notes (kept here for posterity)
- Keyboard extensions **cannot** present AVAudioSession /
SFSpeechRecognizer permission dialogs directly. The overlay's
step 2/3 buttons optimistically advance; the actual permission
is granted when the user first opens the host app (which the
step-5 "Open OSGKeyboard" button triggers). This is the same
pattern the previous "jump to host app" flow used — just
without the broken return trip.
- `UIApplication.openSettingsURLString` is the only system URL
reachable from `extensionContext.open`. Both step 4 and the
"Open Settings" button route through `HostAppLauncher` so the
responder-chain fallback also kicks in if needed.
Co-authored-by: Mavis <Mavis@hkgood.dev>
v0.3.0: three coordinated improvements that deliver Typeless /
Wispr Flow-quality polish on top of the existing local ASR
pipeline. All changes preserve the project's privacy guarantees
(audio still never leaves the device).
## 1. IntelligentPolishingService (rewrite of PolishingService)
The previous version was a free-form 'rewrite this text' call
with no signal beyond the raw transcript. The new one is a
single LLM call that does three things in one pass, exactly as
Typeless and Wispr Flow do internally:
1. ASR error correction (homophones, near-misses, missing chars)
2. Polish (drop filler words, fix grammar, add punctuation)
3. Style adaptation per app context (code / email / chat / doc)
The merged-prompt design halves the round-trip vs the previously
proposed two-stage design (correction + polish separately) and
the academic literature confirms it performs equivalently for
everyday Chinese / English dictation.
## 2. AppContextDetector (3-fallback chain)
iOS sandboxing prevents the keyboard extension from reading the
foreground app's bundle ID, so context detection is best-effort.
The detector runs three fallbacks in order, with caching to
avoid the cold-start 'unknown' that would force a neutral-tone
LLM call every time the user opens a new field:
1. Heuristic on the text at the cursor (code / email / chat / doc)
2. 30-minute cache of the last successful detection
3. Time-of-day + weekend heuristic as a soft default
The keyboard extension runs the detector on every press of the
mic and persists the result to the App Group so the host app's
polisher picks it up.
## 3. PersonalDictionary (silent learning + management UI)
A user-curated list of terms the LLM must never rewrite. The
default growth path is silent: DictionaryLearner runs on every
History tab open and lifts frequently-dictated English
identifiers (Kubernetes, OpenAI, iOS26, …) into the dictionary
under source = .history. Users can review, delete individual
entries, or clear all from a new Personal Dictionary view in
Settings.
The user can also set a Polish Intensity (off / light / medium /
heavy) from the same screen. Default is medium, which is what
Typeless and Wispr Flow also use.
## Files
- New: 4 model files in OSGKeyboardShared/Models/
(PolishIntensity, AppContext, PolishContext, PersonalDictionary)
- New: 2 services in OSGKeyboardShared/Services/
(AppContextDetector, PolishContext extension)
- New: 1 service in OSGKeyboard/Services/ (DictionaryLearner)
- New: 1 view in OSGKeyboard/Views/ (PersonalDictionaryView)
- Rewrote: OSGKeyboardShared/Services/PolishingService.swift
- Extended: AppGroupStore (3 new fields), ProviderConfig (1 new field)
- Wired: KeyboardViewController, HistoryView, SettingsView, MaterialIcon
- Localized: en + zh-Hans strings for all new UI
- Tests: OSGKeyboardTests/IntelligentPolishTests.swift (16 tests)
## Verification
- All new code follows the existing Sendable / strict-concurrency
patterns (the keyboard extension stays within its 60MB sandbox;
the polisher remains an actor; @MainActor is applied to the
learner and the settings UI).
- Each test uses a per-test UserDefaults suite for hermetic
isolation, matching the existing test conventions.
- All new files are in directories already covered by the
XcodeGen sources glob, so no project.yml change is needed.
## Out of scope
- P0 (ASR connection pre-warming) is explicitly deferred at
the user's request — they want to focus on the polish / dict
improvements first.
- The Cloud polish (WebSocket) work is not touched.
## Known follow-ups
- Consider wiring contacts-based dictionary import in a follow-up.
- Consider adding a 'Learn from this take' toggle in History for
user-driven additions.
- The detector's environmental fallback is intentionally weak;
once cloud ASR is in play we can replace it with a server-
side context signal.
Rework the action cluster to a mic-above-bottom-row layout, add left/right
handedness setting that swaps delete and return, and keep the screen awake
during Flow recording sessions.
Add preset-driven polish scenarios (Settings, onboarding, ScenarioChip)
with ScenarioPrompt and style directives; drive keyboard height from
content (240pt) and use viewIsAppearing encapsulated-height offset for
smoother keyboard switches; remove redundant StatusBadge and hide system
dictation via hasDictationKey.
- Add PreconfiguredKeys.swift: placeholder constant for the DeepSeek
API key the local engine uses. DEBUG build asserts at launch when
the placeholder is still in place so nobody ships an always-401
build by accident. Replace TODO_FILL_LATER_DEEPSEEK_KEY before
distributing.
- PolishingService.polishRemote:
* Adopted (A) path: when effective provider is DeepSeek, take the
apiKey from PreconfiguredKeys.deepseek instead of store.apiKey.
Refuse the round-trip with PolishError.missingAPIKey when the
placeholder is still in place.
* Adopted fix for pre-existing typo: store.model.isEmpty ?
preset.defaultModel : store.model (the right-hand side was
defaultModel on both branches, silently ignoring the user's
custom model field).
- LocalEngineSettingsRows.LocalModelsGroup:
* Dropped the inline 'uses DeepSeek' caption — it added visual
weight without telling the user anything they couldn't infer.
* Translation row now lives inside the group so the local engine
reads as one cohesive card.
* Group owns its surface card chrome (palette.surface + rounded
border) — callers no longer need to wrap it externally.
- SettingsView:
* languageAndModelsSection: dropped the trailing TranslationPickerRow
and the inner LocalModelsGroup (now lifted to its own section).
* New localEngineSettingsSection: renders only when
engineMode == 'local', wrapping LocalModelsGroup with a
'settings.localEngine.title' header.
* body: added the new section between languageAndModelsSection and
the cloud-only systemPromptLinkSection.
- Localizable.strings (en + zh-Hans):
* Removed settings.localModels.cloudPolish.caption (no references).
* Added settings.localEngine.title = 'Local engine' / '本地引擎'.
DeepSeek key pre-fill tripwire + local-engine UI cohesion. No main
merge, no PR.
Adds an opt-in translation pipeline that reuses the existing
PolishingService + LLMClient + AppGroupStore chain. Translation is
implemented as a new PolishMode (.translate(targetLocaleId:)); all
existing call sites are unchanged.
Settings:
- New TranslationPickerRow in the language tab (Toggle + 10-locale
picker: en/zh-Hans/zh-Hant/ja/ko/fr/de/es/ru/pt), persisted to the
App Group so the keyboard extension can read it during live dictation.
- 5 new strings per language (en + zh-Hans).
Keyboard:
- New TranslationChip on the top bar to the right of LocaleChip;
same Menu pattern, lets users toggle or quickly switch target
language without leaving the keyboard.
- PolishingService dispatches .translate with a parameterised prompt
(en/zh variants selected by provider id); PolishingService.error
gains a translationNotAvailable case so local-engine users get a
clear inline warning when the toggle is on but cloud is off.
- 6 new strings per language (en + zh-Hans) for the chip + banner.
Local engine policy:
- Translation is cloud-only by design (local engine stays ASR-only
to honour the no-roundtrip promise). Chip shows a 'cloud required'
state and raw transcript still inserts on failure — no data loss.
Build:
- OSGKeyboardShared adds TranslationLanguage enum (10 locales) and
TranslationPrompt factory.
- 4 new files, 9 modified. xcodebuild scheme=OSGKeyboard
config=Debug destination=iPhone 17 Simulator: BUILD SUCCEEDED
(0 warning, 0 error).
Also pins DEVELOPMENT_TEAM in project.yml for TestFlight uploads
(3 targets; Team X329MZU23S).
Rolls back the v0.2.0 Qwen3 CoreML on-device ASR stack and replaces the
'local engine' UX with iOS 26 SpeechAnalyzer + DictationTranscriber only.
The 'Cloud polish after ASR' toggle (ProviderConfig.localModeCloudPolishEnabled)
lets users opt into a post-ASR DeepSeek round-trip from the local engine.
Defaults to off so the local engine stays genuinely local. New PolishError.missingAPIError
surfaces an inline 'fill in your key' warning when the toggle is on but the
Keychain is empty. DeepSeek preset default model bumped to deepseek-v4-flash.
Deleted:
- OSGKeyboard/ThirdParty/Qwen3Speech/ (74 files, ~16k LoC)
- OSGKeyboard/Services/ModelManager.swift (492)
- OSGKeyboard/Services/OnDeviceModelWarmup.swift (197)
- OSGKeyboard/Services/Qwen3ASRService.swift (257)
- OSGKeyboard/Services/ModelDownloadSourcePicker.swift (126)
- OSGKeyboard/Views/OnDeviceModelsView.swift (184)
- OSGKeyboard/Views/DownloadConfirmSheet.swift (96)
- OSGKeyboardShared/Models/OnDeviceModel.swift (140)
- OSGKeyboardShared/Services/OnDeviceModelStatus.swift (104)
- Qwen3ASRServiceProvider registration in OSGKeyboardApp
- Qwen3Speech package declaration in project.yml
- 5 .qwen3ASR enum / branch reference sites in HomeView, OnboardingView,
LocalEngineSettingsRows, FlowSessionManager, ASRService, EngineServiceLabel
- Two pre-existing Swift 6 strict-concurrency errors in
LiveDictationController + FlowSessionManager (the weak [weak self] in
detached-task MainActor.run blocks) that were blocking clean builds
Added:
- LocalModelsGroup: 'Built-in iOS SpeechAnalyzer' badge + 'Cloud polish
after ASR' Switch toggle
- PolishingService: honour localModeCloudPolishEnabled; new .missingAPIKey
error case with localised warning
- AppGroupStore.localModeCloudPolishEnabled (mirrored into App Group
so the keyboard extension honours the toggle during live dictation)
- SettingsView: show provider/api sections when local-mode cloud polish
is on so the user can paste a DeepSeek key
- FlowSessionManager: route through PolishingService for local + polish-on
flow; translate missingAPIKey into a polished warning
- KeyboardViewController: handle PolishingService.PolishError.missingAPIKey
in the keyboard-side live polish path
- CHANGELOG v0.2.1: documents the rollback + new toggle
- README.md / README.zh.md: engine matrix section, data flow note
Verified: xcodebuild -scheme OSGKeyboard -destination 'generic/platform=iOS Simulator'
build succeeds under SWIFT_STRICT_CONCURRENCY=complete.
Replace MLX GPU inference with CoreML bundles so transcription continues
while the host app is backgrounded. Adds model download and warm-up,
vendored Qwen3Speech, and updates onboarding, settings, and copy for the
~1.6 GB CoreML package (iOS 18+).
Rocky
rocky