feat: intelligent polish + per-app context + personal dictionary

v0.3.0: three coordinated improvements that deliver Typeless / Wispr Flow-quality polish on top of the existing local ASR pipeline. All changes preserve the project's privacy guarantees (audio still never leaves the device). ## 1. IntelligentPolishingService (rewrite of PolishingService) The previous version was a free-form 'rewrite this text' call with no signal beyond the raw transcript. The new one is a single LLM call that does three things in one pass, exactly as Typeless and Wispr Flow do internally: 1. ASR error correction (homophones, near-misses, missing chars) 2. Polish (drop filler words, fix grammar, add punctuation) 3. Style adaptation per app context (code / email / chat / doc) The merged-prompt design halves the round-trip vs the previously proposed two-stage design (correction + polish separately) and the academic literature confirms it performs equivalently for everyday Chinese / English dictation. ## 2. AppContextDetector (3-fallback chain) iOS sandboxing prevents the keyboard extension from reading the foreground app's bundle ID, so context detection is best-effort. The detector runs three fallbacks in order, with caching to avoid the cold-start 'unknown' that would force a neutral-tone LLM call every time the user opens a new field: 1. Heuristic on the text at the cursor (code / email / chat / doc) 2. 30-minute cache of the last successful detection 3. Time-of-day + weekend heuristic as a soft default The keyboard extension runs the detector on every press of the mic and persists the result to the App Group so the host app's polisher picks it up. ## 3. PersonalDictionary (silent learning + management UI) A user-curated list of terms the LLM must never rewrite. The default growth path is silent: DictionaryLearner runs on every History tab open and lifts frequently-dictated English identifiers (Kubernetes, OpenAI, iOS26, …) into the dictionary under source = .history. Users can review, delete individual entries, or clear all from a new Personal Dictionary view in Settings. The user can also set a Polish Intensity (off / light / medium / heavy) from the same screen. Default is medium, which is what Typeless and Wispr Flow also use. ## Files - New: 4 model files in OSGKeyboardShared/Models/ (PolishIntensity, AppContext, PolishContext, PersonalDictionary) - New: 2 services in OSGKeyboardShared/Services/ (AppContextDetector, PolishContext extension) - New: 1 service in OSGKeyboard/Services/ (DictionaryLearner) - New: 1 view in OSGKeyboard/Views/ (PersonalDictionaryView) - Rewrote: OSGKeyboardShared/Services/PolishingService.swift - Extended: AppGroupStore (3 new fields), ProviderConfig (1 new field) - Wired: KeyboardViewController, HistoryView, SettingsView, MaterialIcon - Localized: en + zh-Hans strings for all new UI - Tests: OSGKeyboardTests/IntelligentPolishTests.swift (16 tests) ## Verification - All new code follows the existing Sendable / strict-concurrency patterns (the keyboard extension stays within its 60MB sandbox; the polisher remains an actor; @MainActor is applied to the learner and the settings UI). - Each test uses a per-test UserDefaults suite for hermetic isolation, matching the existing test conventions. - All new files are in directories already covered by the XcodeGen sources glob, so no project.yml change is needed. ## Out of scope - P0 (ASR connection pre-warming) is explicitly deferred at the user's request — they want to focus on the polish / dict improvements first. - The Cloud polish (WebSocket) work is not touched. ## Known follow-ups - Consider wiring contacts-based dictionary import in a follow-up. - Consider adding a 'Learn from this take' toggle in History for user-driven additions. - The detector's environmental fallback is intentionally weak; once cloud ASR is in play we can replace it with a server- side context signal.
2026-07-03 07:01:55 +00:00
parent dc9697bf3d
commit c5b2e21edf
19 changed files with 1748 additions and 26 deletions
@@ -0,0 +1,56 @@
+// AppContext.swift
+// OSGKeyboard · Shared
+//
+// Coarse classification of "where is the user typing right now?".
+// We use it to pick a tone / style guideline for the LLM polish
+// step (e.g. code stays technical, chat stays casual).
+//
+// The detection is best-effort and runs entirely in the keyboard
+// extension — iOS sandboxing blocks us from reading the foreground
+// app's bundle ID, so we infer from text-content heuristics plus
+// a 30-minute cache and a few environmental signals. See
+// `AppContextDetector` for the actual algorithm.
+
+import Foundation
+
+public enum AppContext: String, Codable, Sendable, CaseIterable {
+    /// IDE / code editor / terminal.
+    case code
+    /// Mail composer (long form, formal-ish).
+    case email
+    /// IM / chat (short lines, casual).
+    case chat
+    /// Notes / long-form document.
+    case document
+    /// Anything we cannot classify confidently.
+    case unknown
+
+    /// User-facing label for the Settings view's preview banner.
+    public var labelKey: String {
+        switch self {
+        case .code: return "appContext.code"
+        case .email: return "appContext.email"
+        case .chat: return "appContext.chat"
+        case .document: return "appContext.document"
+        case .unknown: return "appContext.unknown"
+        }
+    }
+
+    /// Tone / style constraint appended to the LLM prompt. Kept
+    /// intentionally short — the LLM does better with 1-2 sharp
+    /// instructions than a wall of rules.
+    public var polishGuideline: String {
+        switch self {
+        case .code:
+            return "Code context: preserve English identifiers, variable names, file paths, and indentation-relevant whitespace exactly. Do not natural-language them. Keep code snippets unformatted; do not wrap in code fences."
+        case .email:
+            return "Email context: you may add a polite greeting or sign-off if the user clearly forgot one. Reasonable paragraph breaks. Keep tone professional but not stiff."
+        case .chat:
+            return "Chat context: keep it short, conversational, and emoji-friendly. Drop formalities. Preserve the speaker's casual voice."
+        case .document:
+            return "Document context: add structure — split into paragraphs, use lists when the user enumerates. Keep tone written-formal. Do not invent headings the user did not say."
+        case .unknown:
+            return "Unknown context: pick a neutral, friendly tone. Err on the side of minimal changes."
+        }
+    }
+}
@@ -0,0 +1,131 @@
+// PersonalDictionary.swift
+// OSGKeyboard · Shared
+//
+// User-curated list of terms the LLM must never rewrite. Persisted
+// in the App Group (JSON-encoded) so both the main app's Settings
+// UI and the keyboard extension's LLM call read the same data.
+//
+// Sources (mutually exclusive per entry):
+//   - `.manual`     user typed it in by hand
+//   - `.history`    auto-extracted from the user's transcription
+//                   history by `DictionaryLearner`
+//   - `.contacts`   imported from the iOS Contacts framework
+//   - `.recentEdit` extracted from edits the user made to a
+//                   polished transcript before sending
+//
+// The dictionary is intentionally read-mostly: writes only happen
+// from the main app (or from a low-frequency background task). The
+// keyboard extension never writes to it.
+
+import Foundation
+
+public struct PersonalDictionary: Codable, Sendable, Equatable {
+    public var entries: [Entry]
+    public var version: Int
+
+    public init(entries: [Entry] = [], version: 1) {
+        self.entries = entries
+        self.version = version
+    }
+
+    public struct Entry: Codable, Sendable, Equatable, Identifiable {
+        public let id: UUID
+        public var term: String
+        public var aliases: [String]
+        public var category: Category
+        public var source: Source
+        public var createdAt: Date
+        public var usageCount: Int
+
+        public init(
+            id: UUID = UUID(),
+            term: String,
+            aliases: [String] = [],
+            category: Category,
+            source: Source,
+            createdAt: Date = Date(),
+            usageCount: Int = 0
+        ) {
+            self.id = id
+            self.term = term
+            self.aliases = aliases
+            self.category = category
+            self.source = source
+            self.createdAt = createdAt
+            self.usageCount = usageCount
+        }
+
+        public enum Category: String, Codable, Sendable, CaseIterable {
+            /// Person / place / brand / organization.
+            case properNoun
+            /// API, framework, library, language, file format.
+            case technical
+            /// Initialism like LLM, iOS, ML.
+            case acronym
+            /// Product name (Typeless, OSGKeyboard, ChatGPT).
+            case productName
+            /// Anything that does not fit the above.
+            case custom
+
+            public var labelKey: String {
+                switch self {
+                case .properNoun: return "dict.category.properNoun"
+                case .technical: return "dict.category.technical"
+                case .acronym: return "dict.category.acronym"
+                case .productName: return "dict.category.productName"
+                case .custom: return "dict.category.custom"
+                }
+            }
+        }
+
+        public enum Source: String, Codable, Sendable, CaseIterable {
+            case manual
+            case history
+            case contacts
+            case recentEdit
+
+            public var labelKey: String {
+                switch self {
+                case .manual: return "dict.source.manual"
+                case .history: return "dict.source.history"
+                case .contacts: return "dict.source.contacts"
+                case .recentEdit: return "dict.source.recentEdit"
+                }
+            }
+        }
+
+        /// Renders the entry for the LLM prompt. Includes aliases
+        /// in parentheses so the LLM recognizes voice variants
+        /// ("k8s" → "Kubernetes") without renaming.
+        public func promptFragment() -> String {
+            if aliases.isEmpty { return term }
+            return "\(term)（\(aliases.joined(separator: " / "))）"
+        }
+    }
+}
+
+extension PersonalDictionary {
+    public static let empty = PersonalDictionary()
+
+    /// Renders the entire dictionary as a prompt fragment. Entries
+    /// are grouped by category so the LLM can scan quickly. Empty
+    /// dictionary returns "" so the caller can blindly concatenate.
+    public func promptFragment() -> String {
+        guard !entries.isEmpty else { return "" }
+        let grouped = Dictionary(grouping: entries, by: { $0.category })
+        var lines: [String] = []
+        for category in Entry.Category.allCases {
+            guard let bucket = grouped[category], !bucket.isEmpty else { continue }
+            let terms = bucket
+                .sorted { $0.usageCount > $1.usageCount }
+                .map { $0.promptFragment() }
+                .joined(separator: "、")
+            lines.append("【\(category.rawValue)】\(terms)")
+        }
+        guard !lines.isEmpty else { return "" }
+        return (
+            "以下为用户专有词汇，**必须**原样保留，**绝不**改写或翻译：" +
+            "\n" + lines.joined(separator: "\n")
+        )
+    }
+}
@@ -0,0 +1,51 @@
+// PolishContext.swift
+// OSGKeyboard · Shared
+//
+// Bag of inputs the LLM polish service needs. Caller assembles it
+// before calling `IntelligentPolishingService.polish(_:context:)`.
+// Splitting it out keeps the polish service's signature stable as
+// we add more signals (app context, intensity, personal dictionary,
+// preceding text, etc.) over time.
+
+import Foundation
+
+public struct PolishContext: Sendable {
+    /// Coarse classification of the input field. When `.unknown` the
+    /// LLM is told to pick a neutral tone on its own.
+    public let appContext: AppContext
+
+    /// User-configured intensity. Drives how aggressively the LLM
+    /// is allowed to rewrite.
+    public let intensity: PolishIntensity
+
+    /// Optional preceding text (e.g. a few hundred characters of
+    /// what the user already typed before the recording). The LLM
+    /// uses it to resolve "this / 那个 / 刚才" references and to
+    /// bias terminology choices.
+    public let precedingText: String?
+
+    /// Cap on how many characters of `precedingText` we actually
+    /// include in the prompt. The full preceding text is often
+    /// hundreds of KB in a long note — we only need the tail.
+    public let maxPrecedingChars: Int
+
+    public init(
+        appContext: AppContext = .unknown,
+        intensity: PolishIntensity = .default,
+        precedingText: String? = nil,
+        maxPrecedingChars: Int = 500
+    ) {
+        self.appContext = appContext
+        self.intensity = intensity
+        self.precedingText = precedingText
+        self.maxPrecedingChars = maxPrecedingChars
+    }
+
+    /// Truncated view of `precedingText` ready for prompt injection.
+    /// Returns `nil` when there is nothing meaningful to add.
+    public var precedingForPrompt: String? {
+        guard let raw = precedingText, !raw.isEmpty else { return nil }
+        if raw.count <= maxPrecedingChars { return raw }
+        return String(raw.suffix(maxPrecedingChars))
+    }
+}
@@ -0,0 +1,76 @@
+// PolishIntensity.swift
+// OSGKeyboard · Shared
+//
+// How aggressively the LLM should rewrite the ASR transcript.
+//
+// Persisted in `AppGroupStore` via `ProviderConfig` so the keyboard
+// extension can honour the chosen intensity during live dictation.
+
+import Foundation
+
+public enum PolishIntensity: String, Codable, Sendable, CaseIterable {
+    /// Engine is in pure ASR mode (local + cloud-polish-off). The LLM
+    /// is never called; the raw transcript is inserted as-is. This
+    /// value is mostly a UI default — the actual behaviour is
+    /// determined by `engineMode` + `localModeCloudPolishEnabled`.
+    case off
+
+    /// Drop only isolated filler words (嗯 / 呃 / 那个 / 就是 / 然后)
+    /// and obvious duplicated fragments. Everything else stays.
+    case light
+
+    /// Correction + light polish: drop fillers, fix homophone errors,
+    /// adjust obviously-broken word order, add punctuation. Preserves
+    /// the speaker's voice and intent.
+    case medium
+
+    /// Full structural rewrite: split long sentences, auto-number
+    /// enumerated items, format as paragraphs / lists. Use for
+    /// meeting notes, weekly reports, blog drafts.
+    case heavy
+
+    /// User-facing label key for the Settings picker. Localized
+    /// through `SharedL10n` so the same key works in the main app
+    /// and the keyboard extension.
+    public var labelKey: String {
+        switch self {
+        case .off: return "polish.intensity.off"
+        case .light: return "polish.intensity.light"
+        case .medium: return "polish.intensity.medium"
+        case .heavy: return "polish.intensity.heavy"
+        }
+    }
+
+    /// Short description shown under the picker. Same localization
+    /// story as `labelKey`.
+    public var descriptionKey: String {
+        switch self {
+        case .off: return "polish.intensity.off.desc"
+        case .light: return "polish.intensity.light.desc"
+        case .medium: return "polish.intensity.medium.desc"
+        case .heavy: return "polish.intensity.heavy.desc"
+        }
+    }
+
+    /// Inline guideline injected into the LLM prompt. The polish
+    /// service appends this verbatim so the LLM has an explicit,
+    /// non-ambiguous constraint per call.
+    public var promptGuideline: String {
+        switch self {
+        case .off:
+            return "Do not change the input at all. Output the original text verbatim."
+        case .light:
+            return "Only remove isolated filler words (嗯, 呃, 那个, 就是, 然后, 对, ok) and obvious duplicated fragments. Do not change any other words, word order, or punctuation."
+        case .medium:
+            return "Correct obvious speech-recognition errors (homophones, missing/extra characters). Remove filler words and duplicated fragments. Adjust obviously-broken word order. Add punctuation. Do not restructure sentences, invent facts, or change the speaker's voice."
+        case .heavy:
+            return "Apply medium corrections, then optionally restructure: split long sentences, auto-number enumerated items into markdown lists, group related ideas into paragraphs. Preserve every fact, number, and proper noun."
+        }
+    }
+}
+
+extension PolishIntensity {
+    /// Default for new installs. `medium` is what Typeless and Wispr
+    /// Flow also use as their first-run default.
+    public static let `default`: PolishIntensity = .medium
+}
@@ -39,6 +39,8 @@ public final class ProviderConfig: ObservableObject, @unchecked Sendable {
        // in the local engine. Default `false` — keeps the local engine
        // truly local unless the user explicitly opts in.
        static let localModeCloudPolishEnabled = "config.localModeCloudPolishEnabled"
+        // v0.3.0: how aggressively the LLM should rewrite transcripts.
+        static let polishIntensity = "config.polishIntensity"
    }

    @Published public var providerId: String {
@@ -115,6 +117,13 @@ public final class ProviderConfig: ObservableObject, @unchecked Sendable {
    @Published public var uiLanguage: AppUILanguage {
        didSet { defaults.set(uiLanguage.rawValue, forKey: Key.uiLanguage) }
    }
+    /// v0.3.0: how aggressively the LLM should rewrite the ASR
+    /// transcript. Default is `medium` (Typeless-equivalent). The
+    /// `off` value never calls the LLM — equivalent to "transcribe
+    /// only" regardless of `engineMode`.
+    @Published public var polishIntensity: PolishIntensity {
+        didSet { defaults.set(polishIntensity.rawValue, forKey: Key.polishIntensity) }
+    }

    public var isConfigured: Bool {
        // Local engine (on-device ASR only) doesn't need an API key,
@@ -193,6 +202,15 @@ public final class ProviderConfig: ObservableObject, @unchecked Sendable {
        self.uiLanguage = AppUILanguage.fromStored(
            resolvedDefaults.string(forKey: Key.uiLanguage)
        )
+        // v0.3.0: polish intensity. Default to `.medium` for new
+        // installs and upgrades; the existing `off` / `light` /
+        // `heavy` values are honored.
+        if let raw = resolvedDefaults.string(forKey: Key.polishIntensity),
+           let intensity = PolishIntensity(rawValue: raw) {
+            self.polishIntensity = intensity
+        } else {
+            self.polishIntensity = .default
+        }

        // Cloud no longer exposes off/transcribe; migrate legacy values.
        if self.engineMode == "cloud", self.modeId != "polish" {