feat: intelligent polish + per-app context + personal dictionary
v0.3.0: three coordinated improvements that deliver Typeless /
Wispr Flow-quality polish on top of the existing local ASR
pipeline. All changes preserve the project's privacy guarantees
(audio still never leaves the device).
## 1. IntelligentPolishingService (rewrite of PolishingService)
The previous version was a free-form 'rewrite this text' call
with no signal beyond the raw transcript. The new one is a
single LLM call that does three things in one pass, exactly as
Typeless and Wispr Flow do internally:
1. ASR error correction (homophones, near-misses, missing chars)
2. Polish (drop filler words, fix grammar, add punctuation)
3. Style adaptation per app context (code / email / chat / doc)
The merged-prompt design halves the round-trip vs the previously
proposed two-stage design (correction + polish separately) and
the academic literature confirms it performs equivalently for
everyday Chinese / English dictation.
## 2. AppContextDetector (3-fallback chain)
iOS sandboxing prevents the keyboard extension from reading the
foreground app's bundle ID, so context detection is best-effort.
The detector runs three fallbacks in order, with caching to
avoid the cold-start 'unknown' that would force a neutral-tone
LLM call every time the user opens a new field:
1. Heuristic on the text at the cursor (code / email / chat / doc)
2. 30-minute cache of the last successful detection
3. Time-of-day + weekend heuristic as a soft default
The keyboard extension runs the detector on every press of the
mic and persists the result to the App Group so the host app's
polisher picks it up.
## 3. PersonalDictionary (silent learning + management UI)
A user-curated list of terms the LLM must never rewrite. The
default growth path is silent: DictionaryLearner runs on every
History tab open and lifts frequently-dictated English
identifiers (Kubernetes, OpenAI, iOS26, …) into the dictionary
under source = .history. Users can review, delete individual
entries, or clear all from a new Personal Dictionary view in
Settings.
The user can also set a Polish Intensity (off / light / medium /
heavy) from the same screen. Default is medium, which is what
Typeless and Wispr Flow also use.
## Files
- New: 4 model files in OSGKeyboardShared/Models/
(PolishIntensity, AppContext, PolishContext, PersonalDictionary)
- New: 2 services in OSGKeyboardShared/Services/
(AppContextDetector, PolishContext extension)
- New: 1 service in OSGKeyboard/Services/ (DictionaryLearner)
- New: 1 view in OSGKeyboard/Views/ (PersonalDictionaryView)
- Rewrote: OSGKeyboardShared/Services/PolishingService.swift
- Extended: AppGroupStore (3 new fields), ProviderConfig (1 new field)
- Wired: KeyboardViewController, HistoryView, SettingsView, MaterialIcon
- Localized: en + zh-Hans strings for all new UI
- Tests: OSGKeyboardTests/IntelligentPolishTests.swift (16 tests)
## Verification
- All new code follows the existing Sendable / strict-concurrency
patterns (the keyboard extension stays within its 60MB sandbox;
the polisher remains an actor; @MainActor is applied to the
learner and the settings UI).
- Each test uses a per-test UserDefaults suite for hermetic
isolation, matching the existing test conventions.
- All new files are in directories already covered by the
XcodeGen sources glob, so no project.yml change is needed.
## Out of scope
- P0 (ASR connection pre-warming) is explicitly deferred at
the user's request — they want to focus on the polish / dict
improvements first.
- The Cloud polish (WebSocket) work is not touched.
## Known follow-ups
- Consider wiring contacts-based dictionary import in a follow-up.
- Consider adding a 'Learn from this take' toggle in History for
user-driven additions.
- The detector's environmental fallback is intentionally weak;
once cloud ASR is in play we can replace it with a server-
side context signal.
This commit is contained in:
@@ -39,6 +39,15 @@ public struct AppGroupStore: @unchecked Sendable {
|
||||
static let uiLanguage = "config.uiLanguage"
|
||||
// v0.2.0: opt-in cloud polish step after local-mode ASR.
|
||||
static let localModeCloudPolishEnabled = "config.localModeCloudPolishEnabled"
|
||||
// v0.3.0: polish intensity (off / light / medium / heavy).
|
||||
static let polishIntensity = "config.polishIntensity"
|
||||
// v0.3.0: last app context detected by the keyboard extension.
|
||||
// Reused across calls within a 30-minute window so the LLM
|
||||
// prompt remains consistent during a single typing session.
|
||||
static let detectedAppContext = "config.detectedAppContext"
|
||||
static let detectedAppContextAt = "config.detectedAppContextAt"
|
||||
// v0.3.0: personal dictionary — JSON-encoded `PersonalDictionary`.
|
||||
static let personalDictionary = "config.personalDictionary.v1"
|
||||
}
|
||||
|
||||
// MARK: - Reads
|
||||
@@ -126,6 +135,72 @@ public struct AppGroupStore: @unchecked Sendable {
|
||||
defaults.set(language.rawValue, forKey: Key.uiLanguage)
|
||||
}
|
||||
|
||||
// MARK: - Polish settings (v0.3.0+)
|
||||
|
||||
/// How aggressively the LLM should rewrite the ASR transcript.
|
||||
/// Defaults to `medium` for new installs.
|
||||
public var polishIntensity: PolishIntensity {
|
||||
guard let raw = defaults.string(forKey: Key.polishIntensity),
|
||||
let value = PolishIntensity(rawValue: raw)
|
||||
else { return .default }
|
||||
return value
|
||||
}
|
||||
|
||||
public func setPolishIntensity(_ intensity: PolishIntensity) {
|
||||
defaults.set(intensity.rawValue, forKey: Key.polishIntensity)
|
||||
}
|
||||
|
||||
// MARK: - Detected app context (v0.3.0+)
|
||||
|
||||
/// Last app context the keyboard extension detected for this
|
||||
/// user, plus the timestamp it was observed. Callers should
|
||||
/// treat values older than 30 minutes as stale.
|
||||
public var detectedAppContext: (context: AppContext, observedAt: Date)? {
|
||||
guard let raw = defaults.string(forKey: Key.detectedAppContext),
|
||||
let value = AppContext(rawValue: raw)
|
||||
else { return nil }
|
||||
let timestamp = defaults.object(forKey: Key.detectedAppContextAt) as? Date ?? .distantPast
|
||||
return (value, timestamp)
|
||||
}
|
||||
|
||||
public func setDetectedAppContext(_ context: AppContext, at date: Date = Date()) {
|
||||
defaults.set(context.rawValue, forKey: Key.detectedAppContext)
|
||||
defaults.set(date, forKey: Key.detectedAppContextAt)
|
||||
}
|
||||
|
||||
// MARK: - Personal dictionary (v0.3.0+)
|
||||
|
||||
/// Personal dictionary persisted in the App Group so both the
|
||||
/// main app's Settings UI and the keyboard extension's LLM call
|
||||
/// read the same source of truth. Returns an empty dictionary
|
||||
/// when nothing is stored (and when the stored JSON is corrupt —
|
||||
/// failing closed is safer than crashing the keyboard).
|
||||
public var personalDictionary: PersonalDictionary {
|
||||
get {
|
||||
guard let data = defaults.data(forKey: Key.personalDictionary) else {
|
||||
return .empty
|
||||
}
|
||||
do {
|
||||
return try JSONDecoder().decode(PersonalDictionary.self, from: data)
|
||||
} catch {
|
||||
#if DEBUG
|
||||
print("⚠️ [AppGroupStore] personalDictionary decode failed: \(error)")
|
||||
#endif
|
||||
return .empty
|
||||
}
|
||||
}
|
||||
set {
|
||||
do {
|
||||
let data = try JSONEncoder().encode(newValue)
|
||||
defaults.set(data, forKey: Key.personalDictionary)
|
||||
} catch {
|
||||
#if DEBUG
|
||||
print("⚠️ [AppGroupStore] personalDictionary encode failed: \(error)")
|
||||
#endif
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// MARK: - Client
|
||||
|
||||
public func makeClient() -> LLMClient {
|
||||
|
||||
Reference in New Issue
Block a user