Mavis
c5b2e21edf
v0.3.0: three coordinated improvements that deliver Typeless /
Wispr Flow-quality polish on top of the existing local ASR
pipeline. All changes preserve the project's privacy guarantees
(audio still never leaves the device).
## 1. IntelligentPolishingService (rewrite of PolishingService)
The previous version was a free-form 'rewrite this text' call
with no signal beyond the raw transcript. The new one is a
single LLM call that does three things in one pass, exactly as
Typeless and Wispr Flow do internally:
1. ASR error correction (homophones, near-misses, missing chars)
2. Polish (drop filler words, fix grammar, add punctuation)
3. Style adaptation per app context (code / email / chat / doc)
The merged-prompt design halves the round-trip vs the previously
proposed two-stage design (correction + polish separately) and
the academic literature confirms it performs equivalently for
everyday Chinese / English dictation.
## 2. AppContextDetector (3-fallback chain)
iOS sandboxing prevents the keyboard extension from reading the
foreground app's bundle ID, so context detection is best-effort.
The detector runs three fallbacks in order, with caching to
avoid the cold-start 'unknown' that would force a neutral-tone
LLM call every time the user opens a new field:
1. Heuristic on the text at the cursor (code / email / chat / doc)
2. 30-minute cache of the last successful detection
3. Time-of-day + weekend heuristic as a soft default
The keyboard extension runs the detector on every press of the
mic and persists the result to the App Group so the host app's
polisher picks it up.
## 3. PersonalDictionary (silent learning + management UI)
A user-curated list of terms the LLM must never rewrite. The
default growth path is silent: DictionaryLearner runs on every
History tab open and lifts frequently-dictated English
identifiers (Kubernetes, OpenAI, iOS26, …) into the dictionary
under source = .history. Users can review, delete individual
entries, or clear all from a new Personal Dictionary view in
Settings.
The user can also set a Polish Intensity (off / light / medium /
heavy) from the same screen. Default is medium, which is what
Typeless and Wispr Flow also use.
## Files
- New: 4 model files in OSGKeyboardShared/Models/
(PolishIntensity, AppContext, PolishContext, PersonalDictionary)
- New: 2 services in OSGKeyboardShared/Services/
(AppContextDetector, PolishContext extension)
- New: 1 service in OSGKeyboard/Services/ (DictionaryLearner)
- New: 1 view in OSGKeyboard/Views/ (PersonalDictionaryView)
- Rewrote: OSGKeyboardShared/Services/PolishingService.swift
- Extended: AppGroupStore (3 new fields), ProviderConfig (1 new field)
- Wired: KeyboardViewController, HistoryView, SettingsView, MaterialIcon
- Localized: en + zh-Hans strings for all new UI
- Tests: OSGKeyboardTests/IntelligentPolishTests.swift (16 tests)
## Verification
- All new code follows the existing Sendable / strict-concurrency
patterns (the keyboard extension stays within its 60MB sandbox;
the polisher remains an actor; @MainActor is applied to the
learner and the settings UI).
- Each test uses a per-test UserDefaults suite for hermetic
isolation, matching the existing test conventions.
- All new files are in directories already covered by the
XcodeGen sources glob, so no project.yml change is needed.
## Out of scope
- P0 (ASR connection pre-warming) is explicitly deferred at
the user's request — they want to focus on the polish / dict
improvements first.
- The Cloud polish (WebSocket) work is not touched.
## Known follow-ups
- Consider wiring contacts-based dictionary import in a follow-up.
- Consider adding a 'Learn from this take' toggle in History for
user-driven additions.
- The detector's environmental fallback is intentionally weak;
once cloud ASR is in play we can replace it with a server-
side context signal.
77 lines
3.3 KiB
Swift
77 lines
3.3 KiB
Swift
// PolishIntensity.swift
|
|
// OSGKeyboard · Shared
|
|
//
|
|
// How aggressively the LLM should rewrite the ASR transcript.
|
|
//
|
|
// Persisted in `AppGroupStore` via `ProviderConfig` so the keyboard
|
|
// extension can honour the chosen intensity during live dictation.
|
|
|
|
import Foundation
|
|
|
|
public enum PolishIntensity: String, Codable, Sendable, CaseIterable {
|
|
/// Engine is in pure ASR mode (local + cloud-polish-off). The LLM
|
|
/// is never called; the raw transcript is inserted as-is. This
|
|
/// value is mostly a UI default — the actual behaviour is
|
|
/// determined by `engineMode` + `localModeCloudPolishEnabled`.
|
|
case off
|
|
|
|
/// Drop only isolated filler words (嗯 / 呃 / 那个 / 就是 / 然后)
|
|
/// and obvious duplicated fragments. Everything else stays.
|
|
case light
|
|
|
|
/// Correction + light polish: drop fillers, fix homophone errors,
|
|
/// adjust obviously-broken word order, add punctuation. Preserves
|
|
/// the speaker's voice and intent.
|
|
case medium
|
|
|
|
/// Full structural rewrite: split long sentences, auto-number
|
|
/// enumerated items, format as paragraphs / lists. Use for
|
|
/// meeting notes, weekly reports, blog drafts.
|
|
case heavy
|
|
|
|
/// User-facing label key for the Settings picker. Localized
|
|
/// through `SharedL10n` so the same key works in the main app
|
|
/// and the keyboard extension.
|
|
public var labelKey: String {
|
|
switch self {
|
|
case .off: return "polish.intensity.off"
|
|
case .light: return "polish.intensity.light"
|
|
case .medium: return "polish.intensity.medium"
|
|
case .heavy: return "polish.intensity.heavy"
|
|
}
|
|
}
|
|
|
|
/// Short description shown under the picker. Same localization
|
|
/// story as `labelKey`.
|
|
public var descriptionKey: String {
|
|
switch self {
|
|
case .off: return "polish.intensity.off.desc"
|
|
case .light: return "polish.intensity.light.desc"
|
|
case .medium: return "polish.intensity.medium.desc"
|
|
case .heavy: return "polish.intensity.heavy.desc"
|
|
}
|
|
}
|
|
|
|
/// Inline guideline injected into the LLM prompt. The polish
|
|
/// service appends this verbatim so the LLM has an explicit,
|
|
/// non-ambiguous constraint per call.
|
|
public var promptGuideline: String {
|
|
switch self {
|
|
case .off:
|
|
return "Do not change the input at all. Output the original text verbatim."
|
|
case .light:
|
|
return "Only remove isolated filler words (嗯, 呃, 那个, 就是, 然后, 对, ok) and obvious duplicated fragments. Do not change any other words, word order, or punctuation."
|
|
case .medium:
|
|
return "Correct obvious speech-recognition errors (homophones, missing/extra characters). Remove filler words and duplicated fragments. Adjust obviously-broken word order. Add punctuation. Do not restructure sentences, invent facts, or change the speaker's voice."
|
|
case .heavy:
|
|
return "Apply medium corrections, then optionally restructure: split long sentences, auto-number enumerated items into markdown lists, group related ideas into paragraphs. Preserve every fact, number, and proper noun."
|
|
}
|
|
}
|
|
}
|
|
|
|
extension PolishIntensity {
|
|
/// Default for new installs. `medium` is what Typeless and Wispr
|
|
/// Flow also use as their first-run default.
|
|
public static let `default`: PolishIntensity = .medium
|
|
}
|