Custom polish styles can opt in to emotion-matched emoji (default off), with
prompt-level opt-in detection so paste-only styles keep model-added emoji.
Also include Volcengine API-Key ASR auth, voice-processing capture, PiP flash
fix, and related keyboard Shift/haptics reliability work.
Redeploy an osg_personal import table on dictionary add/delete/sync so
Chinese, English typing, and ASR share one curated lexicon (next keyboard open).
Retain the external target app for menu-bar paste and polish context, retry
failed middle ASR chunks once, and stop false-positive path/number violations.
Use redacted cursor neighborhood and pause-aware chunks for more natural polish,
validate protected terms with retry/local fallback, and structure bilingual prompts
for consistency and provider prefix caching.
Harden polish so question drafts stay questions, add local density
routing with style-specific degrade, expand fun style packs, and add
end-to-end FlowTrace logging plus offline guard eval scripts.
Unify Bailian/Volcengine/OpenAI realtime streaming, ABE polish routing with
fun styles, and a shared card-page Settings hierarchy; bump to 1.1 (build 32).
- Add FlowKeepAliveMode (Dynamic Island vs PiP) with iCloud sync
- Settings: keep-alive picker; inactivity timeout only for Live Activity
- PiP: waveform sample-buffer controller, mic released between utterances
- PiP sessions have no idle expiry; user closing PiP ends the session
- FlowSessionManager branches hostReady, session start, and utterance paths
- UIBackgroundModes picture-in-picture; bilingual strings and tests
Co-authored-by: Rocky <hkgood@users.noreply.github.com>
- Accumulate utterance PCM in FlowContinuousCapture for batch retry
- Run full-utterance transcribeChunk when stitched final lags partial
- Refactor Mac MLX tail drain to shared FlowUtteranceEndCoordinator
- Add FlowUtterancePCMStore, UtteranceBatchFallbackPolicy, and tests
Co-authored-by: Rocky <hkgood@users.noreply.github.com>
- Add FlowUtteranceEndCoordinator with 350ms silence drain and 150ms post-roll
- Extend FinalChunkRecovery for short/empty final chunks in chunked pipeline
- Snapshot partial at mic stop and guard final transcript in FlowSessionManager
- Unify tail drain presets (iosFlow/macMLX) and expand diagnostics
Co-authored-by: Rocky <hkgood@users.noreply.github.com>
Add shared prompt composition, custom style management, iCloud sync, and native iOS/macOS selection interfaces so users can keep a consistent writing voice across devices.
Use compliant permission prompts, align cloud ASR privacy disclosures, and default Flow sessions to a shorter inactivity window for safer resubmission.
Replace the Sherpa offline pipeline with native MLX streaming, resilient model downloads, live transcript previews, and supporting tests and documentation.
Add System/Light/Dark preference for iPhone and iPad, restore the iPad
sidebar brand mark, ship a DEBUG seed-demo URL for rich placeholder data,
and refresh GitHub Pages with the device composite, full-bleed pale-green
hero wash, and transparent Mac screenshot chrome.
Add optional StoreKit support tip (features stay free), redesign the
landing page with zh/en + light/dark and product shots, and ship
iPhone/iPad/Mac App Store screenshot sets.
Fix cold-start overlay recursion that overflowed the main-thread stack when
recording began while the ready overlay was still up; also remove temporary
on-screen Flow DEBUG panels after the orange-mic investigation, and land the
macOS overlay/catalog/layout polish plus related Flow recovery hardening.
End Live Activities and release the audio session synchronously on
applicationWillTerminate, and continue macOS local-model install,
onboarding, and settings polish on this branch.
Adds a bundled local ASR model catalog for the macOS app with one-click
Sherpa Qwen3 / SenseVoice downloads (pause/resume, inline actions) and a
shared model storage directory used by MLX Qwen3. Fixes the light-mode
sidebar material and makes the menu-bar icon follow the system appearance
with a refreshed status mark. Renames the built product to OSGKeyboard.app.
Bumps version to 0.5.2 (build 19).
Introduce a standalone macOS menu-bar app (OSGKeyboardMac) that reuses the
platform-agnostic OSGKeyboardShared core: record -> cloud/local ASR -> polish
-> insert. Local mode uses Qwen3-ASR via mlx-swift-asr (macOS 15+, Apple
Silicon); iOS targets stay zero-SPM.
Harden iCloud sync for multi-device correctness:
- Per-field settings merge (appSettings.v2) so concurrent edits no longer
clobber each other's unrelated fields.
- Per-device usage statistics (G-Counter) that sum instead of max().
- Tombstoned dictionary/history merge so deletes propagate and entries can't
resurrect.
- API keys replicate via iCloud Keychain, never iCloud KVS JSON; pulling a
legacy blob without key fields no longer wipes local Keychain entries.
- Add a low-risk "Sync Now" action in Settings.
Fix Flow keyboard mic state: stay orange until the host publishes a real ready
contract, share a single MicVoiceAvailability gate, and self-heal stale
cross-process heartbeat jitter instead of getting stuck.
Extract shared storage (SpeechHistoryStore/UsageStatisticsStore,
ConfigurationStore) into OSGKeyboardShared and add tests for the new
sync/merge logic.
Mirror the personal dictionary through iCloud Key-Value Store with
deterministic merge rules, main-app-only sync UI, and App Group as the
keyboard runtime cache. Add cloud-engine ASR with dictionary bias and
expand the bundled custom language model lexicon.
- fix(keyboard): drop unusable extensionContext.open; report the real
result of UIApplication.open and cancel the start watchdog immediately
on failure with clear "open OSGKeyboard" guidance instead of a 30s spin
- feat(capture): recover from audio route changes (AirPods/wired headset
plug-unplug) and phone-call interruptions by rebuilding the engine/tap
against the live hardware format
- feat(home): add a "Start" button in the Home footer when a session is
inactive and permissions are granted (no-jump manual restart)
- feat(hostapps): expand the return whitelist to 46 apps (Signal, Zoom,
Instagram, X, Douyin, Zhihu, Bilibili, Things, Firefox, ...) with
en/zh-Hans names and synced LSApplicationQueriesSchemes
- feat(island): use a "ready" checkmark instead of a mic glyph while idle
and set a staleDate so an orphaned Live Activity fades after a force-quit
- feat(coldstart): use the template OSG brand mark instead of a system
waveform glyph in the cold-start overlay
- perf(keyboard): rely on Darwin notifications as the primary session
signal and drop the fallback poll from 1s to 3s
- fix(keyboard): remove the redundant "Start" text button next to the
inactive-session hint on the keyboard
- docs: correct the per-take cap to 210s (3.5 min) and document that
force-quitting no longer resurrects the session (tracker + READMEs)
- P0/P1: tail-drain state machine in FlowContinuousCapture with converter
flush; FlowSessionManager awaits drain before finalize
- P2: FlowCaptureTailDrain policy/tracker, pipeline diagnostics, unit tests
- P3: short final chunk merged re-transcription in ChunkedUtterancePipeline
- P4: UtteranceTranscriptStitcher composedSafely fallback; preview path parity
in LiveDictationController
Co-authored-by: Rocky <hkgood@users.noreply.github.com>
Reduce perceived latency from key release to final text:
- Adaptive chunking: 2.5s first chunk + 5s follow-ups so short
utterances start on-device recognition while still recording.
- Session-level ASR warmup and audio-format cache reuse to remove
per-utterance cold-start of SpeechAnalyzer.
- Mirror live pipelined partials to the keyboard transcript line via
a new flow.transcriptionPartial App Group key + Darwin ping.
Also commits the accumulated custom language model, Flow session,
keyboard extension restructure, and Xiaomi MiMo provider work in
progress on this branch.
Batch of in-progress app work from the working tree.
- feat(keyboard): CursorNavigation + CursorDragPad for caret movement;
KeyboardSoundFeedback for system key click sounds
- feat(dictionary): DictionaryAliasGenerator + PersonalDictionaryEntrySheet;
TranscriptPostProcessor quality gate; retire DictionaryLearner
- feat(ui): TabBarVisibility handling; drop PageHeaderRow /
PageHeaderConfirmButton; refresh views and localizable strings
- fix(security): move the hardcoded DeepSeek key out of
PreconfiguredKeys.swift into a gitignored PreconfiguredKeys.local.swift
(seeded from .example by generate-xcodeproj.sh)
- docs(agents): add Conventional Commits versioning + bilingual changelog rules
- chore(gitignore): ignore PreconfiguredKeys.local.swift, .cache/, pycache
Custom language model / lexicon work stays on
feature/custom-language-model-asr. Changelog bullets added under
[Unreleased]; no version bump.
Use ThemedRoot in preview instead of inaccessible ThemePalette().
Add setPersonalDictionary(_:) for let AppGroupStore bindings; use
it from PersonalDictionaryView and DictionaryLearner.
Co-authored-by: Rocky <hkgood@users.noreply.github.com>
AppGroupStore is a struct; property setters are mutating and cannot
be called on a let binding. Add setOnboardingPage and
setHasCompletedOnboarding helpers (like setDetectedAppContext).
Co-authored-by: Rocky <hkgood@users.noreply.github.com>
contains(where:) on String iterates Characters; comparing each to
multi-character strings like "\n " is a type error. Use
tail.contains("\n ") || tail.contains("\t") instead.
Co-authored-by: Rocky <hkgood@users.noreply.github.com>
Use 'version: Int = 1' instead of invalid 'version: 1' so Swift
synthesizes the default initializer and PersonalDictionary() compiles.
Co-authored-by: Rocky <hkgood@users.noreply.github.com>
Two long-standing UX papercuts, fixed without leaving the keyboard:
1. **First-launch onboarding inside the keyboard** — iOS keyboard
extensions *cannot* programmatically switch back to the previous
app after jumping out, so the old flow (jump to host app → user
has to manually navigate back) was 5+ taps of friction. The new
`KeyboardOnboardingOverlay` keeps the user inside the keyboard
for steps 1 (welcome), 2 (mic permission), 3 (speech permission),
and 5 (API key hint). The only step that *must* leave is step 4
("Enable Keyboard"), which jumps to `Settings.app` via
`UIApplication.openSettingsURLString` — on return,
`viewWillAppear` calls `autoAdvancePastKeyboardSetupStepIfNeeded`
which silently advances past that step if the keyboard is now
enabled. Net UX: user types in their app, keyboard walks them
through setup, normal UI appears as soon as setup is done.
2. **Per-app context chip on the keyboard top bar** — the v0.3.0
intelligent-prompt pipeline already adapted tone to detected
context (code/email/chat/document), but without a UI cue the user
had no way to know which mode was active or override the heuristic
when it guessed wrong. The new `AppContextChip` surfaces the
detected context; tap-to-override writes back to
`AppGroupStore.setDetectedAppContext` so the next LLM call
picks up the new tone immediately. Wired into `pressBegan` so
the chip updates in real time as the user types into different
fields.
### Files added
- `OSGKeyboardExt/Views/AppContextChip.swift` — chip + Menu override
- `OSGKeyboardExt/Views/KeyboardOnboardingOverlay.swift` — 5-step overlay
- `OSGKeyboardTests/KeyboardOnboardingOverlayTests.swift` — round-trip + enum surface tests
### Files modified
- `OSGKeyboardShared/Services/KeyboardState.swift`
+ `hasCompletedOnboarding`, `onboardingPage`, `appContext`
+ `setAppContext`, `advanceOnboarding`, `completeOnboarding`
+ `requestMicPermission`, `requestSpeechPermission`, `openSystemSettings`
- `OSGKeyboardShared/Services/AppGroupStore.swift`
+ `hasCompletedOnboarding` / `onboardingPage` accessors (mirror of
`ProviderConfig` keys, so the keyboard extension never has to
instantiate the host-app config)
- `OSGKeyboardExt/KeyboardViewController.swift`
+ action hooks wired (`installStateActions`)
+ `syncOnboardingStateFromAppGroup` / `syncAppContextFromAppGroup`
called on `viewWillAppear` and `loadPersistedConfig`
+ `autoAdvancePastKeyboardSetupStepIfNeeded` for the silent
"jump out → come back" flow
+ `openSystemSettingsFromExtension` opens `Settings.app` via
`UIApplication.openSettingsURLString` (the only system URL
the extension is allowed to open)
+ `detectAndStoreAppContext` mirrors to `state.appContext` so
the chip updates without waiting for `viewWillAppear`
- `OSGKeyboardExt/Views/KeyboardRootView.swift`
+ overlay mounted in `ZStack` over normal UI (animated)
+ AppContextChip in top bar (hidden during onboarding)
- `OSGKeyboardExt/{en,zh-Hans}.lproj/Keyboard.strings`
+ onboarding copy + chip labels (35 keys per language)
### iOS sandbox notes (kept here for posterity)
- Keyboard extensions **cannot** present AVAudioSession /
SFSpeechRecognizer permission dialogs directly. The overlay's
step 2/3 buttons optimistically advance; the actual permission
is granted when the user first opens the host app (which the
step-5 "Open OSGKeyboard" button triggers). This is the same
pattern the previous "jump to host app" flow used — just
without the broken return trip.
- `UIApplication.openSettingsURLString` is the only system URL
reachable from `extensionContext.open`. Both step 4 and the
"Open Settings" button route through `HostAppLauncher` so the
responder-chain fallback also kicks in if needed.
Co-authored-by: Mavis <Mavis@hkgood.dev>
v0.3.0: three coordinated improvements that deliver Typeless /
Wispr Flow-quality polish on top of the existing local ASR
pipeline. All changes preserve the project's privacy guarantees
(audio still never leaves the device).
## 1. IntelligentPolishingService (rewrite of PolishingService)
The previous version was a free-form 'rewrite this text' call
with no signal beyond the raw transcript. The new one is a
single LLM call that does three things in one pass, exactly as
Typeless and Wispr Flow do internally:
1. ASR error correction (homophones, near-misses, missing chars)
2. Polish (drop filler words, fix grammar, add punctuation)
3. Style adaptation per app context (code / email / chat / doc)
The merged-prompt design halves the round-trip vs the previously
proposed two-stage design (correction + polish separately) and
the academic literature confirms it performs equivalently for
everyday Chinese / English dictation.
## 2. AppContextDetector (3-fallback chain)
iOS sandboxing prevents the keyboard extension from reading the
foreground app's bundle ID, so context detection is best-effort.
The detector runs three fallbacks in order, with caching to
avoid the cold-start 'unknown' that would force a neutral-tone
LLM call every time the user opens a new field:
1. Heuristic on the text at the cursor (code / email / chat / doc)
2. 30-minute cache of the last successful detection
3. Time-of-day + weekend heuristic as a soft default
The keyboard extension runs the detector on every press of the
mic and persists the result to the App Group so the host app's
polisher picks it up.
## 3. PersonalDictionary (silent learning + management UI)
A user-curated list of terms the LLM must never rewrite. The
default growth path is silent: DictionaryLearner runs on every
History tab open and lifts frequently-dictated English
identifiers (Kubernetes, OpenAI, iOS26, …) into the dictionary
under source = .history. Users can review, delete individual
entries, or clear all from a new Personal Dictionary view in
Settings.
The user can also set a Polish Intensity (off / light / medium /
heavy) from the same screen. Default is medium, which is what
Typeless and Wispr Flow also use.
## Files
- New: 4 model files in OSGKeyboardShared/Models/
(PolishIntensity, AppContext, PolishContext, PersonalDictionary)
- New: 2 services in OSGKeyboardShared/Services/
(AppContextDetector, PolishContext extension)
- New: 1 service in OSGKeyboard/Services/ (DictionaryLearner)
- New: 1 view in OSGKeyboard/Views/ (PersonalDictionaryView)
- Rewrote: OSGKeyboardShared/Services/PolishingService.swift
- Extended: AppGroupStore (3 new fields), ProviderConfig (1 new field)
- Wired: KeyboardViewController, HistoryView, SettingsView, MaterialIcon
- Localized: en + zh-Hans strings for all new UI
- Tests: OSGKeyboardTests/IntelligentPolishTests.swift (16 tests)
## Verification
- All new code follows the existing Sendable / strict-concurrency
patterns (the keyboard extension stays within its 60MB sandbox;
the polisher remains an actor; @MainActor is applied to the
learner and the settings UI).
- Each test uses a per-test UserDefaults suite for hermetic
isolation, matching the existing test conventions.
- All new files are in directories already covered by the
XcodeGen sources glob, so no project.yml change is needed.
## Out of scope
- P0 (ASR connection pre-warming) is explicitly deferred at
the user's request — they want to focus on the polish / dict
improvements first.
- The Cloud polish (WebSocket) work is not touched.
## Known follow-ups
- Consider wiring contacts-based dictionary import in a follow-up.
- Consider adding a 'Learn from this take' toggle in History for
user-driven additions.
- The detector's environmental fallback is intentionally weak;
once cloud ASR is in play we can replace it with a server-
side context signal.
Rework the action cluster to a mic-above-bottom-row layout, add left/right
handedness setting that swaps delete and return, and keep the screen awake
during Flow recording sessions.
Add preset-driven polish scenarios (Settings, onboarding, ScenarioChip)
with ScenarioPrompt and style directives; drive keyboard height from
content (240pt) and use viewIsAppearing encapsulated-height offset for
smoother keyboard switches; remove redundant StatusBadge and hide system
dictation via hasDictationKey.
- Add PreconfiguredKeys.swift: placeholder constant for the DeepSeek
API key the local engine uses. DEBUG build asserts at launch when
the placeholder is still in place so nobody ships an always-401
build by accident. Replace TODO_FILL_LATER_DEEPSEEK_KEY before
distributing.
- PolishingService.polishRemote:
* Adopted (A) path: when effective provider is DeepSeek, take the
apiKey from PreconfiguredKeys.deepseek instead of store.apiKey.
Refuse the round-trip with PolishError.missingAPIKey when the
placeholder is still in place.
* Adopted fix for pre-existing typo: store.model.isEmpty ?
preset.defaultModel : store.model (the right-hand side was
defaultModel on both branches, silently ignoring the user's
custom model field).
- LocalEngineSettingsRows.LocalModelsGroup:
* Dropped the inline 'uses DeepSeek' caption — it added visual
weight without telling the user anything they couldn't infer.
* Translation row now lives inside the group so the local engine
reads as one cohesive card.
* Group owns its surface card chrome (palette.surface + rounded
border) — callers no longer need to wrap it externally.
- SettingsView:
* languageAndModelsSection: dropped the trailing TranslationPickerRow
and the inner LocalModelsGroup (now lifted to its own section).
* New localEngineSettingsSection: renders only when
engineMode == 'local', wrapping LocalModelsGroup with a
'settings.localEngine.title' header.
* body: added the new section between languageAndModelsSection and
the cloud-only systemPromptLinkSection.
- Localizable.strings (en + zh-Hans):
* Removed settings.localModels.cloudPolish.caption (no references).
* Added settings.localEngine.title = 'Local engine' / '本地引擎'.
DeepSeek key pre-fill tripwire + local-engine UI cohesion. No main
merge, no PR.
- Add LLMProvider.isUserSelectable (default true) and filter
ProviderPickerSection on it; next pass can hide non-user presets
(e.g. a future DeepSeek key-preset) without changing call sites.
- KeyboardRootView: hide TranslationChip when off (matches user's
mental model of an opt-in feature), drop the 'warming' branch
(Qwen3 download UX was removed with the backend in v0.2.0), unify
chip pill height to minHeight 28 + vertical 6 for visual rhythm
across all topbar chips.
- TranslationChip: drop isLocal warning path — both engines now
run the translate-and-polish step (local routes through DeepSeek
via ProviderConfig.localModeProviderId).
- OnboardingView: cloud engine branch now wraps the translation
row in the same surface card chrome as the local branch.
- Strings: drop keyboard.models.warming (no longer referenced).
DeepSeek key pre-fill deferred to a follow-up.
Rocky
rocky