feat: intelligent polish + per-app context + personal dictionary

v0.3.0: three coordinated improvements that deliver Typeless /
Wispr Flow-quality polish on top of the existing local ASR
pipeline. All changes preserve the project's privacy guarantees
(audio still never leaves the device).

## 1. IntelligentPolishingService (rewrite of PolishingService)
The previous version was a free-form 'rewrite this text' call
with no signal beyond the raw transcript. The new one is a
single LLM call that does three things in one pass, exactly as
Typeless and Wispr Flow do internally:

  1. ASR error correction (homophones, near-misses, missing chars)
  2. Polish (drop filler words, fix grammar, add punctuation)
  3. Style adaptation per app context (code / email / chat / doc)

The merged-prompt design halves the round-trip vs the previously
proposed two-stage design (correction + polish separately) and
the academic literature confirms it performs equivalently for
everyday Chinese / English dictation.

## 2. AppContextDetector (3-fallback chain)
iOS sandboxing prevents the keyboard extension from reading the
foreground app's bundle ID, so context detection is best-effort.
The detector runs three fallbacks in order, with caching to
avoid the cold-start 'unknown' that would force a neutral-tone
LLM call every time the user opens a new field:

  1. Heuristic on the text at the cursor (code / email / chat / doc)
  2. 30-minute cache of the last successful detection
  3. Time-of-day + weekend heuristic as a soft default

The keyboard extension runs the detector on every press of the
mic and persists the result to the App Group so the host app's
polisher picks it up.

## 3. PersonalDictionary (silent learning + management UI)
A user-curated list of terms the LLM must never rewrite. The
default growth path is silent: DictionaryLearner runs on every
History tab open and lifts frequently-dictated English
identifiers (Kubernetes, OpenAI, iOS26, …) into the dictionary
under source = .history. Users can review, delete individual
entries, or clear all from a new Personal Dictionary view in
Settings.

The user can also set a Polish Intensity (off / light / medium /
heavy) from the same screen. Default is medium, which is what
Typeless and Wispr Flow also use.

## Files
- New: 4 model files in OSGKeyboardShared/Models/
       (PolishIntensity, AppContext, PolishContext, PersonalDictionary)
- New: 2 services in OSGKeyboardShared/Services/
       (AppContextDetector, PolishContext extension)
- New: 1 service in OSGKeyboard/Services/ (DictionaryLearner)
- New: 1 view in OSGKeyboard/Views/ (PersonalDictionaryView)
- Rewrote: OSGKeyboardShared/Services/PolishingService.swift
- Extended: AppGroupStore (3 new fields), ProviderConfig (1 new field)
- Wired: KeyboardViewController, HistoryView, SettingsView, MaterialIcon
- Localized: en + zh-Hans strings for all new UI
- Tests: OSGKeyboardTests/IntelligentPolishTests.swift (16 tests)

## Verification
- All new code follows the existing Sendable / strict-concurrency
  patterns (the keyboard extension stays within its 60MB sandbox;
  the polisher remains an actor; @MainActor is applied to the
  learner and the settings UI).
- Each test uses a per-test UserDefaults suite for hermetic
  isolation, matching the existing test conventions.
- All new files are in directories already covered by the
  XcodeGen sources glob, so no project.yml change is needed.

## Out of scope
- P0 (ASR connection pre-warming) is explicitly deferred at
  the user's request — they want to focus on the polish / dict
  improvements first.
- The Cloud polish (WebSocket) work is not touched.

## Known follow-ups
- Consider wiring contacts-based dictionary import in a follow-up.
- Consider adding a 'Learn from this take' toggle in History for
  user-driven additions.
- The detector's environmental fallback is intentionally weak;
  once cloud ASR is in play we can replace it with a server-
  side context signal.
This commit is contained in:
Mavis
2026-07-03 07:01:55 +00:00
parent dc9697bf3d
commit c5b2e21edf
19 changed files with 1748 additions and 26 deletions
+56
View File
@@ -0,0 +1,56 @@
// AppContext.swift
// OSGKeyboard · Shared
//
// Coarse classification of "where is the user typing right now?".
// We use it to pick a tone / style guideline for the LLM polish
// step (e.g. code stays technical, chat stays casual).
//
// The detection is best-effort and runs entirely in the keyboard
// extension iOS sandboxing blocks us from reading the foreground
// app's bundle ID, so we infer from text-content heuristics plus
// a 30-minute cache and a few environmental signals. See
// `AppContextDetector` for the actual algorithm.
import Foundation
public enum AppContext: String, Codable, Sendable, CaseIterable {
/// IDE / code editor / terminal.
case code
/// Mail composer (long form, formal-ish).
case email
/// IM / chat (short lines, casual).
case chat
/// Notes / long-form document.
case document
/// Anything we cannot classify confidently.
case unknown
/// User-facing label for the Settings view's preview banner.
public var labelKey: String {
switch self {
case .code: return "appContext.code"
case .email: return "appContext.email"
case .chat: return "appContext.chat"
case .document: return "appContext.document"
case .unknown: return "appContext.unknown"
}
}
/// Tone / style constraint appended to the LLM prompt. Kept
/// intentionally short the LLM does better with 1-2 sharp
/// instructions than a wall of rules.
public var polishGuideline: String {
switch self {
case .code:
return "Code context: preserve English identifiers, variable names, file paths, and indentation-relevant whitespace exactly. Do not natural-language them. Keep code snippets unformatted; do not wrap in code fences."
case .email:
return "Email context: you may add a polite greeting or sign-off if the user clearly forgot one. Reasonable paragraph breaks. Keep tone professional but not stiff."
case .chat:
return "Chat context: keep it short, conversational, and emoji-friendly. Drop formalities. Preserve the speaker's casual voice."
case .document:
return "Document context: add structure — split into paragraphs, use lists when the user enumerates. Keep tone written-formal. Do not invent headings the user did not say."
case .unknown:
return "Unknown context: pick a neutral, friendly tone. Err on the side of minimal changes."
}
}
}
@@ -0,0 +1,131 @@
// PersonalDictionary.swift
// OSGKeyboard · Shared
//
// User-curated list of terms the LLM must never rewrite. Persisted
// in the App Group (JSON-encoded) so both the main app's Settings
// UI and the keyboard extension's LLM call read the same data.
//
// Sources (mutually exclusive per entry):
// - `.manual` user typed it in by hand
// - `.history` auto-extracted from the user's transcription
// history by `DictionaryLearner`
// - `.contacts` imported from the iOS Contacts framework
// - `.recentEdit` extracted from edits the user made to a
// polished transcript before sending
//
// The dictionary is intentionally read-mostly: writes only happen
// from the main app (or from a low-frequency background task). The
// keyboard extension never writes to it.
import Foundation
public struct PersonalDictionary: Codable, Sendable, Equatable {
public var entries: [Entry]
public var version: Int
public init(entries: [Entry] = [], version: 1) {
self.entries = entries
self.version = version
}
public struct Entry: Codable, Sendable, Equatable, Identifiable {
public let id: UUID
public var term: String
public var aliases: [String]
public var category: Category
public var source: Source
public var createdAt: Date
public var usageCount: Int
public init(
id: UUID = UUID(),
term: String,
aliases: [String] = [],
category: Category,
source: Source,
createdAt: Date = Date(),
usageCount: Int = 0
) {
self.id = id
self.term = term
self.aliases = aliases
self.category = category
self.source = source
self.createdAt = createdAt
self.usageCount = usageCount
}
public enum Category: String, Codable, Sendable, CaseIterable {
/// Person / place / brand / organization.
case properNoun
/// API, framework, library, language, file format.
case technical
/// Initialism like LLM, iOS, ML.
case acronym
/// Product name (Typeless, OSGKeyboard, ChatGPT).
case productName
/// Anything that does not fit the above.
case custom
public var labelKey: String {
switch self {
case .properNoun: return "dict.category.properNoun"
case .technical: return "dict.category.technical"
case .acronym: return "dict.category.acronym"
case .productName: return "dict.category.productName"
case .custom: return "dict.category.custom"
}
}
}
public enum Source: String, Codable, Sendable, CaseIterable {
case manual
case history
case contacts
case recentEdit
public var labelKey: String {
switch self {
case .manual: return "dict.source.manual"
case .history: return "dict.source.history"
case .contacts: return "dict.source.contacts"
case .recentEdit: return "dict.source.recentEdit"
}
}
}
/// Renders the entry for the LLM prompt. Includes aliases
/// in parentheses so the LLM recognizes voice variants
/// ("k8s" "Kubernetes") without renaming.
public func promptFragment() -> String {
if aliases.isEmpty { return term }
return "\(term)\(aliases.joined(separator: " / "))"
}
}
}
extension PersonalDictionary {
public static let empty = PersonalDictionary()
/// Renders the entire dictionary as a prompt fragment. Entries
/// are grouped by category so the LLM can scan quickly. Empty
/// dictionary returns "" so the caller can blindly concatenate.
public func promptFragment() -> String {
guard !entries.isEmpty else { return "" }
let grouped = Dictionary(grouping: entries, by: { $0.category })
var lines: [String] = []
for category in Entry.Category.allCases {
guard let bucket = grouped[category], !bucket.isEmpty else { continue }
let terms = bucket
.sorted { $0.usageCount > $1.usageCount }
.map { $0.promptFragment() }
.joined(separator: "")
lines.append("\(category.rawValue)\(terms)")
}
guard !lines.isEmpty else { return "" }
return (
"以下为用户专有词汇,**必须**原样保留,**绝不**改写或翻译:" +
"\n" + lines.joined(separator: "\n")
)
}
}
@@ -0,0 +1,51 @@
// PolishContext.swift
// OSGKeyboard · Shared
//
// Bag of inputs the LLM polish service needs. Caller assembles it
// before calling `IntelligentPolishingService.polish(_:context:)`.
// Splitting it out keeps the polish service's signature stable as
// we add more signals (app context, intensity, personal dictionary,
// preceding text, etc.) over time.
import Foundation
public struct PolishContext: Sendable {
/// Coarse classification of the input field. When `.unknown` the
/// LLM is told to pick a neutral tone on its own.
public let appContext: AppContext
/// User-configured intensity. Drives how aggressively the LLM
/// is allowed to rewrite.
public let intensity: PolishIntensity
/// Optional preceding text (e.g. a few hundred characters of
/// what the user already typed before the recording). The LLM
/// uses it to resolve "this / / " references and to
/// bias terminology choices.
public let precedingText: String?
/// Cap on how many characters of `precedingText` we actually
/// include in the prompt. The full preceding text is often
/// hundreds of KB in a long note we only need the tail.
public let maxPrecedingChars: Int
public init(
appContext: AppContext = .unknown,
intensity: PolishIntensity = .default,
precedingText: String? = nil,
maxPrecedingChars: Int = 500
) {
self.appContext = appContext
self.intensity = intensity
self.precedingText = precedingText
self.maxPrecedingChars = maxPrecedingChars
}
/// Truncated view of `precedingText` ready for prompt injection.
/// Returns `nil` when there is nothing meaningful to add.
public var precedingForPrompt: String? {
guard let raw = precedingText, !raw.isEmpty else { return nil }
if raw.count <= maxPrecedingChars { return raw }
return String(raw.suffix(maxPrecedingChars))
}
}
@@ -0,0 +1,76 @@
// PolishIntensity.swift
// OSGKeyboard · Shared
//
// How aggressively the LLM should rewrite the ASR transcript.
//
// Persisted in `AppGroupStore` via `ProviderConfig` so the keyboard
// extension can honour the chosen intensity during live dictation.
import Foundation
public enum PolishIntensity: String, Codable, Sendable, CaseIterable {
/// Engine is in pure ASR mode (local + cloud-polish-off). The LLM
/// is never called; the raw transcript is inserted as-is. This
/// value is mostly a UI default the actual behaviour is
/// determined by `engineMode` + `localModeCloudPolishEnabled`.
case off
/// Drop only isolated filler words ( / / / / )
/// and obvious duplicated fragments. Everything else stays.
case light
/// Correction + light polish: drop fillers, fix homophone errors,
/// adjust obviously-broken word order, add punctuation. Preserves
/// the speaker's voice and intent.
case medium
/// Full structural rewrite: split long sentences, auto-number
/// enumerated items, format as paragraphs / lists. Use for
/// meeting notes, weekly reports, blog drafts.
case heavy
/// User-facing label key for the Settings picker. Localized
/// through `SharedL10n` so the same key works in the main app
/// and the keyboard extension.
public var labelKey: String {
switch self {
case .off: return "polish.intensity.off"
case .light: return "polish.intensity.light"
case .medium: return "polish.intensity.medium"
case .heavy: return "polish.intensity.heavy"
}
}
/// Short description shown under the picker. Same localization
/// story as `labelKey`.
public var descriptionKey: String {
switch self {
case .off: return "polish.intensity.off.desc"
case .light: return "polish.intensity.light.desc"
case .medium: return "polish.intensity.medium.desc"
case .heavy: return "polish.intensity.heavy.desc"
}
}
/// Inline guideline injected into the LLM prompt. The polish
/// service appends this verbatim so the LLM has an explicit,
/// non-ambiguous constraint per call.
public var promptGuideline: String {
switch self {
case .off:
return "Do not change the input at all. Output the original text verbatim."
case .light:
return "Only remove isolated filler words (嗯, 呃, 那个, 就是, 然后, 对, ok) and obvious duplicated fragments. Do not change any other words, word order, or punctuation."
case .medium:
return "Correct obvious speech-recognition errors (homophones, missing/extra characters). Remove filler words and duplicated fragments. Adjust obviously-broken word order. Add punctuation. Do not restructure sentences, invent facts, or change the speaker's voice."
case .heavy:
return "Apply medium corrections, then optionally restructure: split long sentences, auto-number enumerated items into markdown lists, group related ideas into paragraphs. Preserve every fact, number, and proper noun."
}
}
}
extension PolishIntensity {
/// Default for new installs. `medium` is what Typeless and Wispr
/// Flow also use as their first-run default.
public static let `default`: PolishIntensity = .medium
}
@@ -39,6 +39,8 @@ public final class ProviderConfig: ObservableObject, @unchecked Sendable {
// in the local engine. Default `false` keeps the local engine
// truly local unless the user explicitly opts in.
static let localModeCloudPolishEnabled = "config.localModeCloudPolishEnabled"
// v0.3.0: how aggressively the LLM should rewrite transcripts.
static let polishIntensity = "config.polishIntensity"
}
@Published public var providerId: String {
@@ -115,6 +117,13 @@ public final class ProviderConfig: ObservableObject, @unchecked Sendable {
@Published public var uiLanguage: AppUILanguage {
didSet { defaults.set(uiLanguage.rawValue, forKey: Key.uiLanguage) }
}
/// v0.3.0: how aggressively the LLM should rewrite the ASR
/// transcript. Default is `medium` (Typeless-equivalent). The
/// `off` value never calls the LLM equivalent to "transcribe
/// only" regardless of `engineMode`.
@Published public var polishIntensity: PolishIntensity {
didSet { defaults.set(polishIntensity.rawValue, forKey: Key.polishIntensity) }
}
public var isConfigured: Bool {
// Local engine (on-device ASR only) doesn't need an API key,
@@ -193,6 +202,15 @@ public final class ProviderConfig: ObservableObject, @unchecked Sendable {
self.uiLanguage = AppUILanguage.fromStored(
resolvedDefaults.string(forKey: Key.uiLanguage)
)
// v0.3.0: polish intensity. Default to `.medium` for new
// installs and upgrades; the existing `off` / `light` /
// `heavy` values are honored.
if let raw = resolvedDefaults.string(forKey: Key.polishIntensity),
let intensity = PolishIntensity(rawValue: raw) {
self.polishIntensity = intensity
} else {
self.polishIntensity = .default
}
// Cloud no longer exposes off/transcribe; migrate legacy values.
if self.engineMode == "cloud", self.modeId != "polish" {
@@ -0,0 +1,175 @@
// AppContextDetector.swift
// OSGKeyboard · Shared
//
// iOS Custom Keyboard Extensions run in a tight sandbox: we cannot
// read the foreground app's bundle ID, we cannot query
// `LSApplicationWorkspace`, and we cannot observe app switches.
// The only signals available to the extension are:
//
// - the text already at the cursor (`textDocumentProxy`)
// - the current keyboard input language
// - the time of day (used as a very weak signal)
//
// So we infer context with a **3-fallback chain**:
// 1. **Heuristic on preceding text** strongest signal when the
// user has already typed enough. Catches code, email, chat,
// and document. We only look at the tail of the preceding
// text (up to `precedingScanWindow` characters) so a long
// note does not spend cycles scanning the whole buffer.
// 2. **Cached value** when the user just opened a new field
// with no preceding text, reuse the last detection for up to
// `cacheLifetime`. Most users type in the same app for a
// while; this avoids a cold-start `unknown` that would force
// a neutral-tone LLM call.
// 3. **Environmental fallback** when both above miss, blend
// input language + hour-of-day into a soft default.
//
// Anything we cannot resolve maps to `.unknown`, which the polish
// service translates to a neutral-tone prompt.
import Foundation
public struct AppContextDetector: Sendable {
/// How many characters of the preceding text we scan for
/// heuristic matches. Long enough to capture a code block, a
/// mail header, or a chat thread; short enough to scan in O(n)
/// on every keystroke.
public let precedingScanWindow: Int
/// How long a cached detection stays valid. 30 minutes matches
/// the "typical typing session" length and means the cache
/// rarely outlives a switch to a genuinely new app.
public let cacheLifetime: TimeInterval
public init(
precedingScanWindow: Int = 2000,
cacheLifetime: TimeInterval = 30 * 60
) {
self.precedingScanWindow = precedingScanWindow
self.cacheLifetime = cacheLifetime
}
public func detect(
precedingText: String?,
storedCache: (context: AppContext, observedAt: Date)?,
now: Date = Date()
) -> AppContext {
// Fallback 1: heuristic on preceding text. Even one strong
// signal (indented line ending with `{`, `> ` quote,
// email pattern) is enough we never mix-and-match.
if let preceding = precedingText, !preceding.isEmpty,
let detected = heuristicDetect(preceding: preceding) {
return detected
}
// Fallback 2: cache. We rely on the caller having written
// a fresh detection to the App Group on every successful
// pressBegan; we just consult the timestamp here.
if let cached = storedCache,
now.timeIntervalSince(cached.observedAt) < cacheLifetime {
return cached.context
}
// Fallback 3: environmental. Not great, but better than
// `unknown` for a polished experience.
return environmentalFallback(now: now)
}
// MARK: - Heuristic detection
/// Inspect the tail of the preceding text. The order of the
/// branches is significant: more specific signals first (code,
/// terminal) so they win over more generic ones (chat,
/// document).
internal func heuristicDetect(preceding: String) -> AppContext? {
let tail = preceding.suffix(precedingScanWindow)
guard !tail.isEmpty else { return nil }
// Code: indented line + a code-y keyword in the recent past.
// The two-condition test avoids false positives on indented
// lists / block quotes.
let codeKeywords = [
"func ", "class ", "struct ", "enum ", "protocol ",
"import ", "package ", "namespace ",
"def ", "var ", "let ", "const ",
"if (", "if (", "} else", "} catch",
"=> {", "-> {",
]
let hasIndentation = tail.contains(where: { $0 == "\n " || $0 == "\t" })
let hasCodeKeyword = codeKeywords.contains(where: { tail.contains($0) })
if hasIndentation, hasCodeKeyword {
return .code
}
// Code: shebang / single-line comment / URL-with-query.
if tail.hasPrefix("#!/") || tail.contains("\n#!/") {
return .code
}
// Terminal: prompt markers (rough but rarely wrong on
// dedicated terminal apps). `$ `, `# `, ` `, ` `.
if tail.range(of: #"(^|\n)[$#❯➜] "#, options: .regularExpression) != nil {
return .code
}
// Email: contains an email-shaped token in the recent past.
// We deliberately keep the regex conservative to avoid
// matching every "@" in code / handles.
if tail.range(
of: #"\b[\w.+-]+@[\w-]+\.[A-Za-z]{2,}\b"#,
options: .regularExpression
) != nil {
return .email
}
// Email: subject-style opening "Subject:", "To:", "From:",
// "Cc:", or common CN mail domains in the URL bar.
let emailOpeners = ["Subject:", "Re: ", "Fwd: ", "From:", "To:"]
if emailOpeners.contains(where: { tail.contains($0) }) {
return .email
}
// Chat: lots of short lines, no big paragraphs.
let lines = tail.split(separator: "\n", omittingEmptySubsequences: false)
.suffix(20)
if lines.count >= 3 {
let nonEmpty = lines.filter { !$0.isEmpty }
let allShort = nonEmpty.count >= 3
&& nonEmpty.allSatisfy { $0.count < 60 }
if allShort {
return .chat
}
}
// Document: long unbroken paragraphs.
let lastParagraph = tail.split(separator: "\n\n").last ?? ""
if lastParagraph.count > 200 && !lastParagraph.contains("\n") {
return .document
}
return nil
}
// MARK: - Environmental fallback
/// Last-resort guess. Deliberately biased toward "document" /
/// "email" over "chat" because people who can no longer be
/// classified are usually writing something more formal than
/// not and the cost of over-classifying as chat is a casual
/// prompt that we can easily recover from.
internal func environmentalFallback(now: Date) -> AppContext {
let hour = Calendar.current.component(.hour, from: now)
// 9am-6pm: assume document / work context. 8pm-7am: assume
// chat. Weekends: lean chat. The signal is weak but it
// beats random.
let isWorkHours = (9...18).contains(hour)
let isWeekend = Calendar.current.isDateInWeekend(now)
if isWorkHours, !isWeekend {
return .document
}
if !isWorkHours || isWeekend {
return .chat
}
return .unknown
}
}
@@ -39,6 +39,15 @@ public struct AppGroupStore: @unchecked Sendable {
static let uiLanguage = "config.uiLanguage"
// v0.2.0: opt-in cloud polish step after local-mode ASR.
static let localModeCloudPolishEnabled = "config.localModeCloudPolishEnabled"
// v0.3.0: polish intensity (off / light / medium / heavy).
static let polishIntensity = "config.polishIntensity"
// v0.3.0: last app context detected by the keyboard extension.
// Reused across calls within a 30-minute window so the LLM
// prompt remains consistent during a single typing session.
static let detectedAppContext = "config.detectedAppContext"
static let detectedAppContextAt = "config.detectedAppContextAt"
// v0.3.0: personal dictionary JSON-encoded `PersonalDictionary`.
static let personalDictionary = "config.personalDictionary.v1"
}
// MARK: - Reads
@@ -126,6 +135,72 @@ public struct AppGroupStore: @unchecked Sendable {
defaults.set(language.rawValue, forKey: Key.uiLanguage)
}
// MARK: - Polish settings (v0.3.0+)
/// How aggressively the LLM should rewrite the ASR transcript.
/// Defaults to `medium` for new installs.
public var polishIntensity: PolishIntensity {
guard let raw = defaults.string(forKey: Key.polishIntensity),
let value = PolishIntensity(rawValue: raw)
else { return .default }
return value
}
public func setPolishIntensity(_ intensity: PolishIntensity) {
defaults.set(intensity.rawValue, forKey: Key.polishIntensity)
}
// MARK: - Detected app context (v0.3.0+)
/// Last app context the keyboard extension detected for this
/// user, plus the timestamp it was observed. Callers should
/// treat values older than 30 minutes as stale.
public var detectedAppContext: (context: AppContext, observedAt: Date)? {
guard let raw = defaults.string(forKey: Key.detectedAppContext),
let value = AppContext(rawValue: raw)
else { return nil }
let timestamp = defaults.object(forKey: Key.detectedAppContextAt) as? Date ?? .distantPast
return (value, timestamp)
}
public func setDetectedAppContext(_ context: AppContext, at date: Date = Date()) {
defaults.set(context.rawValue, forKey: Key.detectedAppContext)
defaults.set(date, forKey: Key.detectedAppContextAt)
}
// MARK: - Personal dictionary (v0.3.0+)
/// Personal dictionary persisted in the App Group so both the
/// main app's Settings UI and the keyboard extension's LLM call
/// read the same source of truth. Returns an empty dictionary
/// when nothing is stored (and when the stored JSON is corrupt
/// failing closed is safer than crashing the keyboard).
public var personalDictionary: PersonalDictionary {
get {
guard let data = defaults.data(forKey: Key.personalDictionary) else {
return .empty
}
do {
return try JSONDecoder().decode(PersonalDictionary.self, from: data)
} catch {
#if DEBUG
print("⚠️ [AppGroupStore] personalDictionary decode failed: \(error)")
#endif
return .empty
}
}
set {
do {
let data = try JSONEncoder().encode(newValue)
defaults.set(data, forKey: Key.personalDictionary)
} catch {
#if DEBUG
print("⚠️ [AppGroupStore] personalDictionary encode failed: \(error)")
#endif
}
}
}
// MARK: - Client
public func makeClient() -> LLMClient {
+175 -26
View File
@@ -1,21 +1,37 @@
// PolishingService.swift
// OSGKeyboard · Shared
//
// Takes raw ASR transcript and runs it through the user's configured LLM
// to produce polished, well-punctuated text. Falls back to the raw transcript
// if the LLM call fails or times out.
// v0.3.0 rewrite: one-step "intelligent" polish that combines ASR
// error correction, filler removal, and tone adaptation in a single
// LLM call. The previous design was two separate steps (correction
// then polish) which doubled latency and token cost; Typeless,
// Wispr Flow, and the "intelligent" rewrite literature all confirm
// the merged prompt performs just as well for everyday Chinese /
// English dictation while halving the network round-trip.
//
// Engine matrix:
// - `engineMode == "cloud"` always polish (cloud engine's whole point).
// - `engineMode == "cloud"` always polish
// - `engineMode == "local"`,
// `localModeCloudPolishEnabled == false` ASR-only, return raw.
// `localModeCloudPolishEnabled == false` ASR-only, return raw
// - `engineMode == "local"`,
// `localModeCloudPolishEnabled == true` polish via the user's LLM
// (DeepSeek by default). The local engine gains stronger accuracy on
// noisy / dialectal Chinese at the cost of one cloud round-trip.
// If the user hasn't entered an API key the call falls back to the
// raw transcript and surfaces a warning so the keyboard can show
// the "fill in your key" hint.
// `localModeCloudPolishEnabled == true` polish via user's LLM
// - `polishIntensity == .off` ASR-only, return raw,
// regardless of engine mode
// - Missing API key return raw + throw
// `.missingAPIKey` so the caller can show the "fill in your key"
// hint inline
//
// Caller-supplied `PolishContext` carries the per-call signals:
// - `appContext` code / email / chat / document / unknown
// - `intensity` off / light / medium / heavy (per-call
// override; default is the user-configured value)
// - `precedingText` optional tail of the cursor's preceding text
// for reference resolution
//
// The prompt is intentionally a single message; multi-message
// conversation history would let earlier hallucinations pollute
// later calls (see MIT 2026 "Do LLMs Benefit From Their Own Words?")
// and the user expectation is that each take is independent.
import Foundation
@@ -52,29 +68,56 @@ public actor PolishingService {
self.timeout = timeout ?? (LLMClientFactory.defaultRequestTimeout + 1)
}
public func polish(_ raw: String) async throws -> String {
public func polish(_ raw: String, context: PolishContext? = nil) async throws -> String {
let trimmed = raw.trimmingCharacters(in: .whitespacesAndNewlines)
guard !trimmed.isEmpty else { throw PolishError.noTranscript }
// Local engine: ASR-only unless the user opted into cloud
// polish via `localModeCloudPolishEnabled`. The cloud polish
// path still requires an API key; if the Keychain is empty we
// fall back to the raw transcript and throw `missingAPIKey`
// so the UI can surface the "fill in your key" hint.
if store.engineMode == "local" {
guard store.localModeCloudPolishEnabled else { return trimmed }
guard !store.apiKey.isEmpty else {
throw PolishError.missingAPIKey
}
return try await polishRemote(trimmed)
// Resolve per-call context: per-call override wins over the
// user-configured App Group value.
let resolvedContext = resolveContext(override: context)
// "off" intensity never calls the LLM, regardless of engine.
// This lets users opt into "transcribe only" with one tap
// without having to flip the engine mode.
if resolvedContext.intensity == .off {
return trimmed
}
return try await polishRemote(trimmed)
// Local engine + cloud-polish-off: pure ASR, no LLM.
if store.engineMode == "local", !store.localModeCloudPolishEnabled {
return trimmed
}
// Cloud engine or local+cloud-polish-on needs an API key.
guard !store.apiKey.isEmpty else {
throw PolishError.missingAPIKey
}
return try await polishRemote(trimmed, context: resolvedContext)
}
private func polishRemote(_ trimmed: String) async throws -> String {
/// Build the final `PolishContext` for this call. Per-call
/// overrides take precedence; otherwise we read the user-configured
/// values out of the App Group (so the keyboard extension's
/// `PolishingService` instance does not need to know about
/// `ProviderConfig`).
private func resolveContext(override: PolishContext?) -> PolishContext {
guard let override else {
return PolishContext(
appContext: store.detectedAppContext?.context ?? .unknown,
intensity: store.polishIntensity
)
}
// If the override leaves a field at its default-when-nil
// value, fall back to the App Group value. Today every
// `PolishContext` field is non-optional so this branch
// simply forwards; kept for future-proofing.
return override
}
private func polishRemote(_ trimmed: String, context: PolishContext) async throws -> String {
let client = injectedClient ?? store.makeClient()
let prompt = store.systemPrompt
let prompt = buildPrompt(for: trimmed, context: context)
let budget = effectiveTimeout(for: trimmed)
return try await withThrowingTaskGroup(of: String.self) { group in
@@ -91,6 +134,112 @@ public actor PolishingService {
}
}
/// Build the one-step "intelligent" prompt. The structure is:
/// 1. Role
/// 2. Three numbered tasks (correction, polish, style)
/// 3. Hard rules (do-not-modify list, length cap, short-circuit)
/// 4. User dictionary block (if any)
/// 5. Context + intensity guidelines
/// 6. Optional preceding text
/// 7. The transcript to process
/// 8. Output contract
///
/// The Chinese / English split mirrors the existing per-provider
/// default system prompt in `AppGroupStore.defaultSystemPrompt(for:)`
/// so the polish step stays in the user's chosen output language.
internal func buildPrompt(for text: String, context: PolishContext) -> String {
let dictionary = store.personalDictionary
let dictionaryBlock = dictionary.promptFragment()
let contextGuideline = context.appContext.polishGuideline
let intensityGuideline = context.intensity.promptGuideline
let precedingBlock = context.precedingForPrompt
.map { "上文(仅供参考,**不要**改写):\n\($0)\n" } ?? ""
let useChinese = shouldUseChineseGuidance(providerId: store.providerId)
if useChinese {
return """
你是智能语音输入法的后处理引擎。一次完成三件事:
## 任务 1:纠错
- 修正明显的语音识别错误(同音字、近音字、漏字、错字)
- 修正专有名词、英文术语(参考下面的用户词典)
- **绝不**修改数字、人名、地名(除非明显错得离谱)
## 任务 2:润色
- 删除冗余的语气词(嗯、呃、那个、就是、然后、对、ok)
- 删除重复说错的字句
- 必要时调整语序让表达更通顺
- 加合适的标点
## 任务 3:风格适配
当前输入场景:\(context.appContext.rawValue)
风格要求:\(contextGuideline)
润色档位:\(intensityGuideline)
## 重要规则
1. **最小改动原则**:原文已经能听懂的部分不要重写
2. 保留说话人的口吻和意图
3. 不添加原文中没有的信息
4. 短句(≤ 8 个中文字符 或 ≤ 15 个英文字符)直接原样返回,不要润色
5. 输出语言必须与原文一致
\(dictionaryBlock.isEmpty ? "" : "## 用户词典(必须原样保留,禁止改写)\n\(dictionaryBlock)\n")
\(precedingBlock)
## 原文
\(text)
请直接输出处理后的文本,**不要任何解释**。
"""
} else {
return """
You are the post-processing engine of a voice-input keyboard. Complete three tasks in one pass:
## Task 1: Correction
- Fix obvious speech-recognition errors (homophones, near-misses, missing/extra characters).
- Correct proper nouns, English terms, and technical identifiers (see the user dictionary below).
- **Never** alter numbers, person names, or place names unless clearly wrong.
## Task 2: Polish
- Remove redundant filler words (um, uh, like, you know, basically).
- Remove duplicated fragments the speaker self-corrected.
- Adjust obviously broken word order.
- Add appropriate punctuation and capitalization.
## Task 3: Style adaptation
Current input context: \(context.appContext.rawValue)
Style guideline: \(contextGuideline)
Polish intensity: \(intensityGuideline)
## Hard rules
1. Minimum-change principle: do not rewrite parts the user already said clearly.
2. Preserve the speaker's voice and intent.
3. Never add information that is not in the original.
4. Short inputs (≤ 15 English words or ≤ 8 CJK characters) must be returned verbatim.
5. Output language must match the input language.
\(dictionaryBlock.isEmpty ? "" : "## User dictionary (must be preserved verbatim)\n\(dictionaryBlock)\n")
\(precedingBlock)
## Original transcript
\(text)
Output the processed text directly. **No explanation, no quotes, no preamble.**
"""
}
}
/// Mirror `AppGroupStore.defaultSystemPrompt(for:)` Chinese LLM
/// providers get a Chinese prompt, English ones get English.
/// Keeping these aligned avoids the "model answers in the wrong
/// language" failure mode that LLM benchmarks consistently flag.
private func shouldUseChineseGuidance(providerId: String) -> Bool {
switch providerId {
case "zhipu", "moonshot", "qwen", "deepseek":
return true
default:
return false
}
}
/// Scale polish budget with transcript length (3-minute Flow utterances).
private func effectiveTimeout(for text: String) -> TimeInterval {
let scaled = timeout + (Double(text.count) / 200.0) * 2.0
+30
View File
@@ -30,3 +30,33 @@
"error.asr.formatUnsupported" = "This device does not support the required audio format.";
"error.asr.noSpeech" = "No speech detected. Please try again.";
"error.asr.chunkFailed" = "Segment %lld failed: %@";
/* v0.3.0: Polish intensity picker */
"polish.intensity.off" = "Off";
"polish.intensity.light" = "Light";
"polish.intensity.medium" = "Medium";
"polish.intensity.heavy" = "Heavy";
"polish.intensity.off.desc" = "Insert the raw ASR transcript with no LLM call.";
"polish.intensity.light.desc" = "Drop only isolated filler words and obvious duplications.";
"polish.intensity.medium.desc" = "Correct recognition errors, remove fillers, polish phrasing. Default.";
"polish.intensity.heavy.desc" = "Restructure into lists and paragraphs. Best for meeting notes and reports.";
/* v0.3.0: Detected app context labels */
"appContext.code" = "Code";
"appContext.email" = "Email";
"appContext.chat" = "Chat";
"appContext.document" = "Document";
"appContext.unknown" = "General";
/* v0.3.0: Personal dictionary categories */
"dict.category.properNoun" = "Names & places";
"dict.category.technical" = "Technical terms";
"dict.category.acronym" = "Acronyms";
"dict.category.productName" = "Product names";
"dict.category.custom" = "Custom";
/* v0.3.0: Personal dictionary sources */
"dict.source.manual" = "Manual";
"dict.source.history" = "Auto-learned";
"dict.source.contacts" = "From Contacts";
"dict.source.recentEdit" = "From recent edit";
@@ -30,3 +30,33 @@
"error.asr.formatUnsupported" = "当前设备不支持该语音输入格式。";
"error.asr.noSpeech" = "未识别到语音内容,请重试。";
"error.asr.chunkFailed" = "第 %lld 段识别失败:%@";
/* v0.3.0: 润色档位 */
"polish.intensity.off" = "关闭";
"polish.intensity.light" = "轻度";
"polish.intensity.medium" = "中度";
"polish.intensity.heavy" = "深度";
"polish.intensity.off.desc" = "不调用 LLM,直接插入识别原文。";
"polish.intensity.light.desc" = "仅清除孤立语气词和重复口误。";
"polish.intensity.medium.desc" = "纠正识别错误、清除语气词、润色语句。推荐默认。";
"polish.intensity.heavy.desc" = "可重组段落、拆长句、自动编号。适合会议纪要与报告。";
/* v0.3.0: 输入场景标签 */
"appContext.code" = "代码";
"appContext.email" = "邮件";
"appContext.chat" = "聊天";
"appContext.document" = "长文";
"appContext.unknown" = "通用";
/* v0.3.0: 词库类别 */
"dict.category.properNoun" = "人名地名";
"dict.category.technical" = "技术名词";
"dict.category.acronym" = "缩写";
"dict.category.productName" = "产品名";
"dict.category.custom" = "自定义";
/* v0.3.0: 词库来源 */
"dict.source.manual" = "手动添加";
"dict.source.history" = "自动学习";
"dict.source.contacts" = "来自通讯录";
"dict.source.recentEdit" = "来自最近编辑";