- Keep recording after a streaming ASR error and retry the captured PCM
through batch ASR before reporting a failure.
- Treat an expired ASR wait as a stall so batch fallback still runs.
- Prime the managed Volcengine stream on mic touch and adopt it only while
fresh and open.
- Auto-scroll the live transcript to the newest recognized text.
Squashes the working tree that had accumulated on cloud-agent. Two
strands of work, kept in one commit at the maintainer's request.
Crash and stability (this session):
- Subscribe to MetricKit so crash, hang, CPU, and disk-write diagnostics
are recorded on device. Keyboard-extension memory kills are the most
likely way a user loses the keyboard mid-sentence and iOS does not
classify them as crashes, so they never reach App Store Connect.
- Upload those reports to POST /v1/analytics/diagnostics from internal
builds only (local Debug and TestFlight), reusing the analytics
installation identifier and skipping entirely when analytics are off.
App Store builds keep everything on device.
- Shed the Chinese engine, English lexicon, and in-flight pipeline work
at the 40 MiB and 48 MiB budget thresholds instead of waiting for a
system memory warning that jetsam often preempts.
- Install audio taps with the input node's live format through one
shared adaptive resampler, so an audio-route change mid-recording no
longer raises an uncatchable format-mismatch exception. Covers the
dictation preview and the Mac recorder, which still used the pattern
the Flow capture path had already moved away from.
- Harden the librime bridge: never pass a nil string across a nonnull
boundary when a user dictionary holds malformed UTF-8, null-check the
engine API vector, and distrust a candidate count without its array.
- Guard the expanded candidate grid against a list that shrinks while
the collection view is still laying out against the previous count.
- Resolve localized text without the App Group so the screen shown when
that container is missing renders its message instead of crashing on
launch, which is the one situation it exists to handle.
- Gate CI on the App Group and Keychain wiring that the keyboard
extension hard-depends on, with a self-test proving the check fails.
Clipboard, reply, and semantics (pre-existing working tree):
- Unify the reply center, refresh keyboard AI features, and add native
event and reminder exporters.
- Expand clipboard semantic routing with authored corpora and a
product-blind holdout, plus the scripts that build and validate them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Unify PiP recovery across startup and foreground transitions, surface actionable status, and add privacy-safe analytics plus the refreshed onboarding and account experience. Update documentation assets and advance the release build to 84.
Introduce optional Apple account-backed credits with scoped gateway access while preserving local and BYOK paths. Refresh assistant behavior, tests, privacy disclosures, docs, and the website for the 2.0 experience.
Rotate AI idle suggestions with optional remote packs, move history/dictionary onto self-sizing Home preview cards, harden clipboard capture/prompting, and simplify keyboard chrome by dropping most liquid-glass shadows.
Add the AI keyboard tab, Agent settings, and user-owned LLM key path for 1.7.0, including streaming answers and web-search transports without the built-in DeepSeek fallback.
Adapt typing/voice surfaces for iPad width and height, add the system globe
key and last-input editing flow, harden host-only Rime deployment, and remove
clipboard voice commands. Bump build to 61.
Stabilize clipboard long-press prepare/resume across paste alerts and cold start, add an in-app release-notes sheet with remote bilingual HTML, localize typing input settings, and bump build to 55.
Long-press mic runs ASR as an instruction over eligible clipboard text,
with host-confirm recording UI, min-record gate, and light ASR prewarm.
Bump CURRENT_PROJECT_VERSION to 53.
Custom polish styles can opt in to emotion-matched emoji (default off), with
prompt-level opt-in detection so paste-only styles keep model-added emoji.
Also include Volcengine API-Key ASR auth, voice-processing capture, PiP flash
fix, and related keyboard Shift/haptics reliability work.
Use redacted cursor neighborhood and pause-aware chunks for more natural polish,
validate protected terms with retry/local fallback, and structure bilingual prompts
for consistency and provider prefix caching.
Harden polish so question drafts stay questions, add local density
routing with style-specific degrade, expand fun style packs, and add
end-to-end FlowTrace logging plus offline guard eval scripts.
Unify Bailian/Volcengine/OpenAI realtime streaming, ABE polish routing with
fun styles, and a shared card-page Settings hierarchy; bump to 1.1 (build 32).
- Add FlowKeepAliveMode (Dynamic Island vs PiP) with iCloud sync
- Settings: keep-alive picker; inactivity timeout only for Live Activity
- PiP: waveform sample-buffer controller, mic released between utterances
- PiP sessions have no idle expiry; user closing PiP ends the session
- FlowSessionManager branches hostReady, session start, and utterance paths
- UIBackgroundModes picture-in-picture; bilingual strings and tests
Co-authored-by: Rocky <hkgood@users.noreply.github.com>
- Accumulate utterance PCM in FlowContinuousCapture for batch retry
- Run full-utterance transcribeChunk when stitched final lags partial
- Refactor Mac MLX tail drain to shared FlowUtteranceEndCoordinator
- Add FlowUtterancePCMStore, UtteranceBatchFallbackPolicy, and tests
Co-authored-by: Rocky <hkgood@users.noreply.github.com>
- Add FlowUtteranceEndCoordinator with 350ms silence drain and 150ms post-roll
- Extend FinalChunkRecovery for short/empty final chunks in chunked pipeline
- Snapshot partial at mic stop and guard final transcript in FlowSessionManager
- Unify tail drain presets (iosFlow/macMLX) and expand diagnostics
Co-authored-by: Rocky <hkgood@users.noreply.github.com>
Fix cold-start overlay recursion that overflowed the main-thread stack when
recording began while the ready overlay was still up; also remove temporary
on-screen Flow DEBUG panels after the orange-mic investigation, and land the
macOS overlay/catalog/layout polish plus related Flow recovery hardening.
End Live Activities and release the audio session synchronously on
applicationWillTerminate, and continue macOS local-model install,
onboarding, and settings polish on this branch.
Adds a bundled local ASR model catalog for the macOS app with one-click
Sherpa Qwen3 / SenseVoice downloads (pause/resume, inline actions) and a
shared model storage directory used by MLX Qwen3. Fixes the light-mode
sidebar material and makes the menu-bar icon follow the system appearance
with a refreshed status mark. Renames the built product to OSGKeyboard.app.
Bumps version to 0.5.2 (build 19).
Introduce a standalone macOS menu-bar app (OSGKeyboardMac) that reuses the
platform-agnostic OSGKeyboardShared core: record -> cloud/local ASR -> polish
-> insert. Local mode uses Qwen3-ASR via mlx-swift-asr (macOS 15+, Apple
Silicon); iOS targets stay zero-SPM.
Harden iCloud sync for multi-device correctness:
- Per-field settings merge (appSettings.v2) so concurrent edits no longer
clobber each other's unrelated fields.
- Per-device usage statistics (G-Counter) that sum instead of max().
- Tombstoned dictionary/history merge so deletes propagate and entries can't
resurrect.
- API keys replicate via iCloud Keychain, never iCloud KVS JSON; pulling a
legacy blob without key fields no longer wipes local Keychain entries.
- Add a low-risk "Sync Now" action in Settings.
Fix Flow keyboard mic state: stay orange until the host publishes a real ready
contract, share a single MicVoiceAvailability gate, and self-heal stale
cross-process heartbeat jitter instead of getting stuck.
Extract shared storage (SpeechHistoryStore/UsageStatisticsStore,
ConfigurationStore) into OSGKeyboardShared and add tests for the new
sync/merge logic.
Mirror the personal dictionary through iCloud Key-Value Store with
deterministic merge rules, main-app-only sync UI, and App Group as the
keyboard runtime cache. Add cloud-engine ASR with dictionary bias and
expand the bundled custom language model lexicon.
- fix(keyboard): drop unusable extensionContext.open; report the real
result of UIApplication.open and cancel the start watchdog immediately
on failure with clear "open OSGKeyboard" guidance instead of a 30s spin
- feat(capture): recover from audio route changes (AirPods/wired headset
plug-unplug) and phone-call interruptions by rebuilding the engine/tap
against the live hardware format
- feat(home): add a "Start" button in the Home footer when a session is
inactive and permissions are granted (no-jump manual restart)
- feat(hostapps): expand the return whitelist to 46 apps (Signal, Zoom,
Instagram, X, Douyin, Zhihu, Bilibili, Things, Firefox, ...) with
en/zh-Hans names and synced LSApplicationQueriesSchemes
- feat(island): use a "ready" checkmark instead of a mic glyph while idle
and set a staleDate so an orphaned Live Activity fades after a force-quit
- feat(coldstart): use the template OSG brand mark instead of a system
waveform glyph in the cold-start overlay
- perf(keyboard): rely on Darwin notifications as the primary session
signal and drop the fallback poll from 1s to 3s
- fix(keyboard): remove the redundant "Start" text button next to the
inactive-session hint on the keyboard
- docs: correct the per-take cap to 210s (3.5 min) and document that
force-quitting no longer resurrects the session (tracker + READMEs)
- Add OSGKeyboardLiveActivity widget extension (ActivityKit) with branded
compactLeading mark in the Dynamic Island during Flow sessions
- Wire FlowLiveActivityController into FlowSessionManager lifecycle
- Set keyboard PrimaryLanguage to mis in project.yml, Info.plist, and at runtime
- Enable NSSupportsLiveActivities on main app and widget extension targets
Co-authored-by: Rocky <hkgood@users.noreply.github.com>
- P0/P1: tail-drain state machine in FlowContinuousCapture with converter
flush; FlowSessionManager awaits drain before finalize
- P2: FlowCaptureTailDrain policy/tracker, pipeline diagnostics, unit tests
- P3: short final chunk merged re-transcription in ChunkedUtterancePipeline
- P4: UtteranceTranscriptStitcher composedSafely fallback; preview path parity
in LiveDictationController
Co-authored-by: Rocky <hkgood@users.noreply.github.com>
Reduce perceived latency from key release to final text:
- Adaptive chunking: 2.5s first chunk + 5s follow-ups so short
utterances start on-device recognition while still recording.
- Session-level ASR warmup and audio-format cache reuse to remove
per-utterance cold-start of SpeechAnalyzer.
- Mirror live pipelined partials to the keyboard transcript line via
a new flow.transcriptionPartial App Group key + Darwin ping.
Also commits the accumulated custom language model, Flow session,
keyboard extension restructure, and Xiaomi MiMo provider work in
progress on this branch.
Batch of in-progress app work from the working tree.
- feat(keyboard): CursorNavigation + CursorDragPad for caret movement;
KeyboardSoundFeedback for system key click sounds
- feat(dictionary): DictionaryAliasGenerator + PersonalDictionaryEntrySheet;
TranscriptPostProcessor quality gate; retire DictionaryLearner
- feat(ui): TabBarVisibility handling; drop PageHeaderRow /
PageHeaderConfirmButton; refresh views and localizable strings
- fix(security): move the hardcoded DeepSeek key out of
PreconfiguredKeys.swift into a gitignored PreconfiguredKeys.local.swift
(seeded from .example by generate-xcodeproj.sh)
- docs(agents): add Conventional Commits versioning + bilingual changelog rules
- chore(gitignore): ignore PreconfiguredKeys.local.swift, .cache/, pycache
Custom language model / lexicon work stays on
feature/custom-language-model-asr. Changelog bullets added under
[Unreleased]; no version bump.
Track dictation duration, dictation/translation character counts, and
personal dictionary size. Record metrics when Flow finalizes utterances.
Co-authored-by: Rocky <hkgood@users.noreply.github.com>
Rework the action cluster to a mic-above-bottom-row layout, add left/right
handedness setting that swaps delete and return, and keep the screen awake
during Flow recording sessions.
Add preset-driven polish scenarios (Settings, onboarding, ScenarioChip)
with ScenarioPrompt and style directives; drive keyboard height from
content (240pt) and use viewIsAppearing encapsulated-height offset for
smoother keyboard switches; remove redundant StatusBadge and hide system
dictation via hasDictationKey.
Rolls back the v0.2.0 Qwen3 CoreML on-device ASR stack and replaces the
'local engine' UX with iOS 26 SpeechAnalyzer + DictationTranscriber only.
The 'Cloud polish after ASR' toggle (ProviderConfig.localModeCloudPolishEnabled)
lets users opt into a post-ASR DeepSeek round-trip from the local engine.
Defaults to off so the local engine stays genuinely local. New PolishError.missingAPIError
surfaces an inline 'fill in your key' warning when the toggle is on but the
Keychain is empty. DeepSeek preset default model bumped to deepseek-v4-flash.
Deleted:
- OSGKeyboard/ThirdParty/Qwen3Speech/ (74 files, ~16k LoC)
- OSGKeyboard/Services/ModelManager.swift (492)
- OSGKeyboard/Services/OnDeviceModelWarmup.swift (197)
- OSGKeyboard/Services/Qwen3ASRService.swift (257)
- OSGKeyboard/Services/ModelDownloadSourcePicker.swift (126)
- OSGKeyboard/Views/OnDeviceModelsView.swift (184)
- OSGKeyboard/Views/DownloadConfirmSheet.swift (96)
- OSGKeyboardShared/Models/OnDeviceModel.swift (140)
- OSGKeyboardShared/Services/OnDeviceModelStatus.swift (104)
- Qwen3ASRServiceProvider registration in OSGKeyboardApp
- Qwen3Speech package declaration in project.yml
- 5 .qwen3ASR enum / branch reference sites in HomeView, OnboardingView,
LocalEngineSettingsRows, FlowSessionManager, ASRService, EngineServiceLabel
- Two pre-existing Swift 6 strict-concurrency errors in
LiveDictationController + FlowSessionManager (the weak [weak self] in
detached-task MainActor.run blocks) that were blocking clean builds
Added:
- LocalModelsGroup: 'Built-in iOS SpeechAnalyzer' badge + 'Cloud polish
after ASR' Switch toggle
- PolishingService: honour localModeCloudPolishEnabled; new .missingAPIKey
error case with localised warning
- AppGroupStore.localModeCloudPolishEnabled (mirrored into App Group
so the keyboard extension honours the toggle during live dictation)
- SettingsView: show provider/api sections when local-mode cloud polish
is on so the user can paste a DeepSeek key
- FlowSessionManager: route through PolishingService for local + polish-on
flow; translate missingAPIKey into a polished warning
- KeyboardViewController: handle PolishingService.PolishError.missingAPIKey
in the keyboard-side live polish path
- CHANGELOG v0.2.1: documents the rollback + new toggle
- README.md / README.zh.md: engine matrix section, data flow note
Verified: xcodebuild -scheme OSGKeyboard -destination 'generic/platform=iOS Simulator'
build succeeds under SWIFT_STRICT_CONCURRENCY=complete.
Replace MLX GPU inference with CoreML bundles so transcription continues
while the host app is backgrounded. Adds model download and warm-up,
vendored Qwen3Speech, and updates onboarding, settings, and copy for the
~1.6 GB CoreML package (iOS 18+).
Renew voice sessions while the host app stays foreground, auto-start flow from the keyboard with a Start action, and prevent leading audio loss via pre-roll buffering and faster signal polling.
Introduce MainTabView with history and Liquid Glass dock; refresh home and
onboarding layouts; unify accent green and provider localization; refine
settings/API rows; redesign keyboard mic and flanking controls; set Utility
iPhone-only targets and Flow session reliability fixes.
Remove KeyboardL10n hard-coded fallbacks in favor of ExtL10n and extension
Localizable.strings. Add Flow session expiry hints, Darwin cross-process
notifications, session monitor on the keyboard, and app icon on GitHub Pages.
Migrate keyboard dictation to continuous Flow sessions with auto-start,
tap-to-toggle recording, 60s countdown, five-step onboarding, and App Group
IPC. Add docs/ GitHub Pages site with en/zh privacy policy for App Store compliance.
rocky
