Version: CURRENT_PROJECT_VERSION 130 -> 131 (project.yml, AGENTS.md, READMEs,
docs/llms.txt).
fix(flow): host-app identification now compiles in every build configuration
(Debug, TestFlight, Release). The arbiter probe and HostAppIdentity were
DEBUG-only, so TestFlight/App Store builds never knew the host bundle id and
could not auto-return after a startflow handoff. Debug-only logging and the
Home test card are unchanged.
Also includes the pending work in the tree:
- Home: single account/setup card surface (HomeAccountSurfacePolicy/Views)
- Wide screens: voice key row and centred mic placement, iPhone Duo layout
- Keyboard: clipboard skill ranking and AI skill layout updates
- Debug: account session handoff to the Simulator (+ import script)
- Docs: portfolio assets, privacy policy URL, site/README refresh
- KeyboardState moves from ObservableObject/@Published to @Observable, so a view re-renders only for the properties it reads. Previously unpublished state and injected action hooks are @ObservationIgnored.
- KeyboardViewController replaces the $phase/$surface Combine sinks with Observations loops tied to the same cancellables.
- AIKeyboardView reads the 200 ms mic level and live ASR partials in small child views (AssistantWaveform, LiveTranscriptReader), so those ticks no longer rebuild the whole assistant surface.
- Tests observe changes with withObservationTracking instead of Combine publishers.
- Keep recording after a streaming ASR error and retry the captured PCM
through batch ASR before reporting a failure.
- Treat an expired ASR wait as a stall so batch fallback still runs.
- Prime the managed Volcengine stream on mic touch and adopt it only while
fresh and open.
- Auto-scroll the live transcript to the newest recognized text.
- record analytics/usage only in the host app; the extension no longer opens the
SQLite queues, so hiding the keyboard cannot hold WAL locks (0xdead10cc)
- sync clipboard history writes across processes so deletes made on one side
are not reverted by the other
- guard memory-relief reentrancy on controller and telemetry sides to break
the recursive crash loop
- resolve credentials off the main thread and migrate legacy keys once per
install, so Keychain reads cannot be caught in the suspend window
- gate Volcengine ASR ping completion to once (SIGTRAP on double resume)
- treat same-text captures as the same logical copy and dedupe auto actions
by content fingerprint within 30s, stopping Universal Clipboard storms
- realign version line to 2.1.0 (build 110)
Squashes the working tree that had accumulated on cloud-agent. Two
strands of work, kept in one commit at the maintainer's request.
Crash and stability (this session):
- Subscribe to MetricKit so crash, hang, CPU, and disk-write diagnostics
are recorded on device. Keyboard-extension memory kills are the most
likely way a user loses the keyboard mid-sentence and iOS does not
classify them as crashes, so they never reach App Store Connect.
- Upload those reports to POST /v1/analytics/diagnostics from internal
builds only (local Debug and TestFlight), reusing the analytics
installation identifier and skipping entirely when analytics are off.
App Store builds keep everything on device.
- Shed the Chinese engine, English lexicon, and in-flight pipeline work
at the 40 MiB and 48 MiB budget thresholds instead of waiting for a
system memory warning that jetsam often preempts.
- Install audio taps with the input node's live format through one
shared adaptive resampler, so an audio-route change mid-recording no
longer raises an uncatchable format-mismatch exception. Covers the
dictation preview and the Mac recorder, which still used the pattern
the Flow capture path had already moved away from.
- Harden the librime bridge: never pass a nil string across a nonnull
boundary when a user dictionary holds malformed UTF-8, null-check the
engine API vector, and distrust a candidate count without its array.
- Guard the expanded candidate grid against a list that shrinks while
the collection view is still laying out against the previous count.
- Resolve localized text without the App Group so the screen shown when
that container is missing renders its message instead of crashing on
launch, which is the one situation it exists to handle.
- Gate CI on the App Group and Keychain wiring that the keyboard
extension hard-depends on, with a self-test proving the check fails.
Clipboard, reply, and semantics (pre-existing working tree):
- Unify the reply center, refresh keyboard AI features, and add native
event and reminder exporters.
- Expand clipboard semantic routing with authored corpora and a
product-blind holdout, plus the scripts that build and validate them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- Merge invitation, task, blessing, clarification, and empathy actions
into a single Reply flow, with three fixed, clearly labeled stance
choices whenever user intent must not be guessed.
- Refine clipboard semantic routing with bilingual schedule,
confirmation, and follow-up models, conservative language thresholds,
and explicit-assignment guard for complaint-only text.
- Persist Apple account refresh state, harden session recovery, and
surface durable account diagnostics across keyboard and app.
- Derive personal-style prompts through two-stage corpus evidence and
apply real low-confidence ASR tendencies instead of neutral templates.
- Localize the new reply center, clipboard semantics, and personal-style
surfaces in both English and Simplified Chinese.
Resolve the pinned XcodeGen archive layout and let the keyboard derive bounded clipboard recommendations from the full catalog independently of installation state.
Unify PiP recovery across startup and foreground transitions, surface actionable status, and add privacy-safe analytics plus the refreshed onboarding and account experience. Update documentation assets and advance the release build to 84.
Introduce optional Apple account-backed credits with scoped gateway access while preserving local and BYOK paths. Refresh assistant behavior, tests, privacy disclosures, docs, and the website for the 2.0 experience.
Merge dictation and AI controls into one assistant surface, preserve safe insertion and clipboard actions, and align settings and tests with the new flow.
Add save-to-Notes export and direct map handoff while aligning multi-touch typing, period shortcuts, return actions, and navigation visuals with system behavior.
Rotate AI idle suggestions with optional remote packs, move history/dictionary onto self-sizing Home preview cards, harden clipboard capture/prompting, and simplify keyboard chrome by dropping most liquid-glass shadows.
Ship optional keyboard clipboard history with settings, suggestion strip, and paste-permission guidance; let undo roll back clipboard inserts; bump build to 64.
Add the AI keyboard tab, Agent settings, and user-owned LLM key path for 1.7.0, including streaming answers and web-search transports without the built-in DeepSeek fallback.
Adapt typing/voice surfaces for iPad width and height, add the system globe
key and last-input editing flow, harden host-only Rime deployment, and remove
clipboard voice commands. Bump build to 61.
Keep marketing version at 1.6.6 and bump build to 59. Strengthen never-answer
polish safeguards, clipboard reply-intent continuity, voice undo UX, device
UITests harness, eval fixtures, and What's New assets.
Stabilize clipboard long-press prepare/resume across paste alerts and cold start, add an in-app release-notes sheet with remote bilingual HTML, localize typing input settings, and bump build to 55.
Long-press mic runs ASR as an instruction over eligible clipboard text,
with host-confirm recording UI, min-record gate, and light ASR prewarm.
Bump CURRENT_PROJECT_VERSION to 53.
Custom polish styles can opt in to emotion-matched emoji (default off), with
prompt-level opt-in detection so paste-only styles keep model-added emoji.
Also include Volcengine API-Key ASR auth, voice-processing capture, PiP flash
fix, and related keyboard Shift/haptics reliability work.
Fix cold-start overlay recursion that overflowed the main-thread stack when
recording began while the ready overlay was still up; also remove temporary
on-screen Flow DEBUG panels after the orange-mic investigation, and land the
macOS overlay/catalog/layout polish plus related Flow recovery hardening.
Introduce a standalone macOS menu-bar app (OSGKeyboardMac) that reuses the
platform-agnostic OSGKeyboardShared core: record -> cloud/local ASR -> polish
-> insert. Local mode uses Qwen3-ASR via mlx-swift-asr (macOS 15+, Apple
Silicon); iOS targets stay zero-SPM.
Harden iCloud sync for multi-device correctness:
- Per-field settings merge (appSettings.v2) so concurrent edits no longer
clobber each other's unrelated fields.
- Per-device usage statistics (G-Counter) that sum instead of max().
- Tombstoned dictionary/history merge so deletes propagate and entries can't
resurrect.
- API keys replicate via iCloud Keychain, never iCloud KVS JSON; pulling a
legacy blob without key fields no longer wipes local Keychain entries.
- Add a low-risk "Sync Now" action in Settings.
Fix Flow keyboard mic state: stay orange until the host publishes a real ready
contract, share a single MicVoiceAvailability gate, and self-heal stale
cross-process heartbeat jitter instead of getting stuck.
Extract shared storage (SpeechHistoryStore/UsageStatisticsStore,
ConfigurationStore) into OSGKeyboardShared and add tests for the new
sync/merge logic.
- fix(keyboard): drop unusable extensionContext.open; report the real
result of UIApplication.open and cancel the start watchdog immediately
on failure with clear "open OSGKeyboard" guidance instead of a 30s spin
- feat(capture): recover from audio route changes (AirPods/wired headset
plug-unplug) and phone-call interruptions by rebuilding the engine/tap
against the live hardware format
- feat(home): add a "Start" button in the Home footer when a session is
inactive and permissions are granted (no-jump manual restart)
- feat(hostapps): expand the return whitelist to 46 apps (Signal, Zoom,
Instagram, X, Douyin, Zhihu, Bilibili, Things, Firefox, ...) with
en/zh-Hans names and synced LSApplicationQueriesSchemes
- feat(island): use a "ready" checkmark instead of a mic glyph while idle
and set a staleDate so an orphaned Live Activity fades after a force-quit
- feat(coldstart): use the template OSG brand mark instead of a system
waveform glyph in the cold-start overlay
- perf(keyboard): rely on Darwin notifications as the primary session
signal and drop the fallback poll from 1s to 3s
- fix(keyboard): remove the redundant "Start" text button next to the
inactive-session hint on the keyboard
- docs: correct the per-take cap to 210s (3.5 min) and document that
force-quitting no longer resurrects the session (tracker + READMEs)
Reduce perceived latency from key release to final text:
- Adaptive chunking: 2.5s first chunk + 5s follow-ups so short
utterances start on-device recognition while still recording.
- Session-level ASR warmup and audio-format cache reuse to remove
per-utterance cold-start of SpeechAnalyzer.
- Mirror live pipelined partials to the keyboard transcript line via
a new flow.transcriptionPartial App Group key + Darwin ping.
Also commits the accumulated custom language model, Flow session,
keyboard extension restructure, and Xiaomi MiMo provider work in
progress on this branch.
Batch of in-progress app work from the working tree.
- feat(keyboard): CursorNavigation + CursorDragPad for caret movement;
KeyboardSoundFeedback for system key click sounds
- feat(dictionary): DictionaryAliasGenerator + PersonalDictionaryEntrySheet;
TranscriptPostProcessor quality gate; retire DictionaryLearner
- feat(ui): TabBarVisibility handling; drop PageHeaderRow /
PageHeaderConfirmButton; refresh views and localizable strings
- fix(security): move the hardcoded DeepSeek key out of
PreconfiguredKeys.swift into a gitignored PreconfiguredKeys.local.swift
(seeded from .example by generate-xcodeproj.sh)
- docs(agents): add Conventional Commits versioning + bilingual changelog rules
- chore(gitignore): ignore PreconfiguredKeys.local.swift, .cache/, pycache
Custom language model / lexicon work stays on
feature/custom-language-model-asr. Changelog bullets added under
[Unreleased]; no version bump.
- Remove polish scenario and ASR locale chips from keyboard top bar
- Fix polish intensity labels via SharedL10n (Shared.strings table)
- Merge polish intensity and personal dictionary into preferences card
- Personal dictionary: no list icon; detail view uses nav bar + back
Co-authored-by: Rocky <hkgood@users.noreply.github.com>
ExtL10n.text returns Text; use ExtL10n.string where String is
required (computed properties, Text/Button initializers). TypeStyle
has caption, not caption1 — align onboarding and dictionary views.
Co-authored-by: Rocky <hkgood@users.noreply.github.com>
Two long-standing UX papercuts, fixed without leaving the keyboard:
1. **First-launch onboarding inside the keyboard** — iOS keyboard
extensions *cannot* programmatically switch back to the previous
app after jumping out, so the old flow (jump to host app → user
has to manually navigate back) was 5+ taps of friction. The new
`KeyboardOnboardingOverlay` keeps the user inside the keyboard
for steps 1 (welcome), 2 (mic permission), 3 (speech permission),
and 5 (API key hint). The only step that *must* leave is step 4
("Enable Keyboard"), which jumps to `Settings.app` via
`UIApplication.openSettingsURLString` — on return,
`viewWillAppear` calls `autoAdvancePastKeyboardSetupStepIfNeeded`
which silently advances past that step if the keyboard is now
enabled. Net UX: user types in their app, keyboard walks them
through setup, normal UI appears as soon as setup is done.
2. **Per-app context chip on the keyboard top bar** — the v0.3.0
intelligent-prompt pipeline already adapted tone to detected
context (code/email/chat/document), but without a UI cue the user
had no way to know which mode was active or override the heuristic
when it guessed wrong. The new `AppContextChip` surfaces the
detected context; tap-to-override writes back to
`AppGroupStore.setDetectedAppContext` so the next LLM call
picks up the new tone immediately. Wired into `pressBegan` so
the chip updates in real time as the user types into different
fields.
### Files added
- `OSGKeyboardExt/Views/AppContextChip.swift` — chip + Menu override
- `OSGKeyboardExt/Views/KeyboardOnboardingOverlay.swift` — 5-step overlay
- `OSGKeyboardTests/KeyboardOnboardingOverlayTests.swift` — round-trip + enum surface tests
### Files modified
- `OSGKeyboardShared/Services/KeyboardState.swift`
+ `hasCompletedOnboarding`, `onboardingPage`, `appContext`
+ `setAppContext`, `advanceOnboarding`, `completeOnboarding`
+ `requestMicPermission`, `requestSpeechPermission`, `openSystemSettings`
- `OSGKeyboardShared/Services/AppGroupStore.swift`
+ `hasCompletedOnboarding` / `onboardingPage` accessors (mirror of
`ProviderConfig` keys, so the keyboard extension never has to
instantiate the host-app config)
- `OSGKeyboardExt/KeyboardViewController.swift`
+ action hooks wired (`installStateActions`)
+ `syncOnboardingStateFromAppGroup` / `syncAppContextFromAppGroup`
called on `viewWillAppear` and `loadPersistedConfig`
+ `autoAdvancePastKeyboardSetupStepIfNeeded` for the silent
"jump out → come back" flow
+ `openSystemSettingsFromExtension` opens `Settings.app` via
`UIApplication.openSettingsURLString` (the only system URL
the extension is allowed to open)
+ `detectAndStoreAppContext` mirrors to `state.appContext` so
the chip updates without waiting for `viewWillAppear`
- `OSGKeyboardExt/Views/KeyboardRootView.swift`
+ overlay mounted in `ZStack` over normal UI (animated)
+ AppContextChip in top bar (hidden during onboarding)
- `OSGKeyboardExt/{en,zh-Hans}.lproj/Keyboard.strings`
+ onboarding copy + chip labels (35 keys per language)
### iOS sandbox notes (kept here for posterity)
- Keyboard extensions **cannot** present AVAudioSession /
SFSpeechRecognizer permission dialogs directly. The overlay's
step 2/3 buttons optimistically advance; the actual permission
is granted when the user first opens the host app (which the
step-5 "Open OSGKeyboard" button triggers). This is the same
pattern the previous "jump to host app" flow used — just
without the broken return trip.
- `UIApplication.openSettingsURLString` is the only system URL
reachable from `extensionContext.open`. Both step 4 and the
"Open Settings" button route through `HostAppLauncher` so the
responder-chain fallback also kicks in if needed.
Co-authored-by: Mavis <Mavis@hkgood.dev>
Rework the action cluster to a mic-above-bottom-row layout, add left/right
handedness setting that swaps delete and return, and keep the screen awake
during Flow recording sessions.
Add preset-driven polish scenarios (Settings, onboarding, ScenarioChip)
with ScenarioPrompt and style directives; drive keyboard height from
content (240pt) and use viewIsAppearing encapsulated-height offset for
smoother keyboard switches; remove redundant StatusBadge and hide system
dictation via hasDictationKey.
- Add LLMProvider.isUserSelectable (default true) and filter
ProviderPickerSection on it; next pass can hide non-user presets
(e.g. a future DeepSeek key-preset) without changing call sites.
- KeyboardRootView: hide TranslationChip when off (matches user's
mental model of an opt-in feature), drop the 'warming' branch
(Qwen3 download UX was removed with the backend in v0.2.0), unify
chip pill height to minHeight 28 + vertical 6 for visual rhythm
across all topbar chips.
- TranslationChip: drop isLocal warning path — both engines now
run the translate-and-polish step (local routes through DeepSeek
via ProviderConfig.localModeProviderId).
- OnboardingView: cloud engine branch now wraps the translation
row in the same surface card chrome as the local branch.
- Strings: drop keyboard.models.warming (no longer referenced).
DeepSeek key pre-fill deferred to a follow-up.
Adds an opt-in translation pipeline that reuses the existing
PolishingService + LLMClient + AppGroupStore chain. Translation is
implemented as a new PolishMode (.translate(targetLocaleId:)); all
existing call sites are unchanged.
Settings:
- New TranslationPickerRow in the language tab (Toggle + 10-locale
picker: en/zh-Hans/zh-Hant/ja/ko/fr/de/es/ru/pt), persisted to the
App Group so the keyboard extension can read it during live dictation.
- 5 new strings per language (en + zh-Hans).
Keyboard:
- New TranslationChip on the top bar to the right of LocaleChip;
same Menu pattern, lets users toggle or quickly switch target
language without leaving the keyboard.
- PolishingService dispatches .translate with a parameterised prompt
(en/zh variants selected by provider id); PolishingService.error
gains a translationNotAvailable case so local-engine users get a
clear inline warning when the toggle is on but cloud is off.
- 6 new strings per language (en + zh-Hans) for the chip + banner.
Local engine policy:
- Translation is cloud-only by design (local engine stays ASR-only
to honour the no-roundtrip promise). Chip shows a 'cloud required'
state and raw transcript still inserts on failure — no data loss.
Build:
- OSGKeyboardShared adds TranslationLanguage enum (10 locales) and
TranslationPrompt factory.
- 4 new files, 9 modified. xcodebuild scheme=OSGKeyboard
config=Debug destination=iPhone 17 Simulator: BUILD SUCCEEDED
(0 warning, 0 error).
Also pins DEVELOPMENT_TEAM in project.yml for TestFlight uploads
(3 targets; Team X329MZU23S).
Rocky
rocky