feat: intelligent polish + per-app context + personal dictionary
v0.3.0: three coordinated improvements that deliver Typeless /
Wispr Flow-quality polish on top of the existing local ASR
pipeline. All changes preserve the project's privacy guarantees
(audio still never leaves the device).
## 1. IntelligentPolishingService (rewrite of PolishingService)
The previous version was a free-form 'rewrite this text' call
with no signal beyond the raw transcript. The new one is a
single LLM call that does three things in one pass, exactly as
Typeless and Wispr Flow do internally:
1. ASR error correction (homophones, near-misses, missing chars)
2. Polish (drop filler words, fix grammar, add punctuation)
3. Style adaptation per app context (code / email / chat / doc)
The merged-prompt design halves the round-trip vs the previously
proposed two-stage design (correction + polish separately) and
the academic literature confirms it performs equivalently for
everyday Chinese / English dictation.
## 2. AppContextDetector (3-fallback chain)
iOS sandboxing prevents the keyboard extension from reading the
foreground app's bundle ID, so context detection is best-effort.
The detector runs three fallbacks in order, with caching to
avoid the cold-start 'unknown' that would force a neutral-tone
LLM call every time the user opens a new field:
1. Heuristic on the text at the cursor (code / email / chat / doc)
2. 30-minute cache of the last successful detection
3. Time-of-day + weekend heuristic as a soft default
The keyboard extension runs the detector on every press of the
mic and persists the result to the App Group so the host app's
polisher picks it up.
## 3. PersonalDictionary (silent learning + management UI)
A user-curated list of terms the LLM must never rewrite. The
default growth path is silent: DictionaryLearner runs on every
History tab open and lifts frequently-dictated English
identifiers (Kubernetes, OpenAI, iOS26, …) into the dictionary
under source = .history. Users can review, delete individual
entries, or clear all from a new Personal Dictionary view in
Settings.
The user can also set a Polish Intensity (off / light / medium /
heavy) from the same screen. Default is medium, which is what
Typeless and Wispr Flow also use.
## Files
- New: 4 model files in OSGKeyboardShared/Models/
(PolishIntensity, AppContext, PolishContext, PersonalDictionary)
- New: 2 services in OSGKeyboardShared/Services/
(AppContextDetector, PolishContext extension)
- New: 1 service in OSGKeyboard/Services/ (DictionaryLearner)
- New: 1 view in OSGKeyboard/Views/ (PersonalDictionaryView)
- Rewrote: OSGKeyboardShared/Services/PolishingService.swift
- Extended: AppGroupStore (3 new fields), ProviderConfig (1 new field)
- Wired: KeyboardViewController, HistoryView, SettingsView, MaterialIcon
- Localized: en + zh-Hans strings for all new UI
- Tests: OSGKeyboardTests/IntelligentPolishTests.swift (16 tests)
## Verification
- All new code follows the existing Sendable / strict-concurrency
patterns (the keyboard extension stays within its 60MB sandbox;
the polisher remains an actor; @MainActor is applied to the
learner and the settings UI).
- Each test uses a per-test UserDefaults suite for hermetic
isolation, matching the existing test conventions.
- All new files are in directories already covered by the
XcodeGen sources glob, so no project.yml change is needed.
## Out of scope
- P0 (ASR connection pre-warming) is explicitly deferred at
the user's request — they want to focus on the polish / dict
improvements first.
- The Cloud polish (WebSocket) work is not touched.
## Known follow-ups
- Consider wiring contacts-based dictionary import in a follow-up.
- Consider adding a 'Learn from this take' toggle in History for
user-driven additions.
- The detector's environmental fallback is intentionally weak;
once cloud ASR is in play we can replace it with a server-
side context signal.
This commit is contained in:
@@ -0,0 +1,131 @@
|
||||
// PersonalDictionary.swift
|
||||
// OSGKeyboard · Shared
|
||||
//
|
||||
// User-curated list of terms the LLM must never rewrite. Persisted
|
||||
// in the App Group (JSON-encoded) so both the main app's Settings
|
||||
// UI and the keyboard extension's LLM call read the same data.
|
||||
//
|
||||
// Sources (mutually exclusive per entry):
|
||||
// - `.manual` user typed it in by hand
|
||||
// - `.history` auto-extracted from the user's transcription
|
||||
// history by `DictionaryLearner`
|
||||
// - `.contacts` imported from the iOS Contacts framework
|
||||
// - `.recentEdit` extracted from edits the user made to a
|
||||
// polished transcript before sending
|
||||
//
|
||||
// The dictionary is intentionally read-mostly: writes only happen
|
||||
// from the main app (or from a low-frequency background task). The
|
||||
// keyboard extension never writes to it.
|
||||
|
||||
import Foundation
|
||||
|
||||
public struct PersonalDictionary: Codable, Sendable, Equatable {
|
||||
public var entries: [Entry]
|
||||
public var version: Int
|
||||
|
||||
public init(entries: [Entry] = [], version: 1) {
|
||||
self.entries = entries
|
||||
self.version = version
|
||||
}
|
||||
|
||||
public struct Entry: Codable, Sendable, Equatable, Identifiable {
|
||||
public let id: UUID
|
||||
public var term: String
|
||||
public var aliases: [String]
|
||||
public var category: Category
|
||||
public var source: Source
|
||||
public var createdAt: Date
|
||||
public var usageCount: Int
|
||||
|
||||
public init(
|
||||
id: UUID = UUID(),
|
||||
term: String,
|
||||
aliases: [String] = [],
|
||||
category: Category,
|
||||
source: Source,
|
||||
createdAt: Date = Date(),
|
||||
usageCount: Int = 0
|
||||
) {
|
||||
self.id = id
|
||||
self.term = term
|
||||
self.aliases = aliases
|
||||
self.category = category
|
||||
self.source = source
|
||||
self.createdAt = createdAt
|
||||
self.usageCount = usageCount
|
||||
}
|
||||
|
||||
public enum Category: String, Codable, Sendable, CaseIterable {
|
||||
/// Person / place / brand / organization.
|
||||
case properNoun
|
||||
/// API, framework, library, language, file format.
|
||||
case technical
|
||||
/// Initialism like LLM, iOS, ML.
|
||||
case acronym
|
||||
/// Product name (Typeless, OSGKeyboard, ChatGPT).
|
||||
case productName
|
||||
/// Anything that does not fit the above.
|
||||
case custom
|
||||
|
||||
public var labelKey: String {
|
||||
switch self {
|
||||
case .properNoun: return "dict.category.properNoun"
|
||||
case .technical: return "dict.category.technical"
|
||||
case .acronym: return "dict.category.acronym"
|
||||
case .productName: return "dict.category.productName"
|
||||
case .custom: return "dict.category.custom"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public enum Source: String, Codable, Sendable, CaseIterable {
|
||||
case manual
|
||||
case history
|
||||
case contacts
|
||||
case recentEdit
|
||||
|
||||
public var labelKey: String {
|
||||
switch self {
|
||||
case .manual: return "dict.source.manual"
|
||||
case .history: return "dict.source.history"
|
||||
case .contacts: return "dict.source.contacts"
|
||||
case .recentEdit: return "dict.source.recentEdit"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Renders the entry for the LLM prompt. Includes aliases
|
||||
/// in parentheses so the LLM recognizes voice variants
|
||||
/// ("k8s" → "Kubernetes") without renaming.
|
||||
public func promptFragment() -> String {
|
||||
if aliases.isEmpty { return term }
|
||||
return "\(term)(\(aliases.joined(separator: " / ")))"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
extension PersonalDictionary {
|
||||
public static let empty = PersonalDictionary()
|
||||
|
||||
/// Renders the entire dictionary as a prompt fragment. Entries
|
||||
/// are grouped by category so the LLM can scan quickly. Empty
|
||||
/// dictionary returns "" so the caller can blindly concatenate.
|
||||
public func promptFragment() -> String {
|
||||
guard !entries.isEmpty else { return "" }
|
||||
let grouped = Dictionary(grouping: entries, by: { $0.category })
|
||||
var lines: [String] = []
|
||||
for category in Entry.Category.allCases {
|
||||
guard let bucket = grouped[category], !bucket.isEmpty else { continue }
|
||||
let terms = bucket
|
||||
.sorted { $0.usageCount > $1.usageCount }
|
||||
.map { $0.promptFragment() }
|
||||
.joined(separator: "、")
|
||||
lines.append("【\(category.rawValue)】\(terms)")
|
||||
}
|
||||
guard !lines.isEmpty else { return "" }
|
||||
return (
|
||||
"以下为用户专有词汇,**必须**原样保留,**绝不**改写或翻译:" +
|
||||
"\n" + lines.joined(separator: "\n")
|
||||
)
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user