feat: intelligent polish + per-app context + personal dictionary

v0.3.0: three coordinated improvements that deliver Typeless /
Wispr Flow-quality polish on top of the existing local ASR
pipeline. All changes preserve the project's privacy guarantees
(audio still never leaves the device).

## 1. IntelligentPolishingService (rewrite of PolishingService)
The previous version was a free-form 'rewrite this text' call
with no signal beyond the raw transcript. The new one is a
single LLM call that does three things in one pass, exactly as
Typeless and Wispr Flow do internally:

  1. ASR error correction (homophones, near-misses, missing chars)
  2. Polish (drop filler words, fix grammar, add punctuation)
  3. Style adaptation per app context (code / email / chat / doc)

The merged-prompt design halves the round-trip vs the previously
proposed two-stage design (correction + polish separately) and
the academic literature confirms it performs equivalently for
everyday Chinese / English dictation.

## 2. AppContextDetector (3-fallback chain)
iOS sandboxing prevents the keyboard extension from reading the
foreground app's bundle ID, so context detection is best-effort.
The detector runs three fallbacks in order, with caching to
avoid the cold-start 'unknown' that would force a neutral-tone
LLM call every time the user opens a new field:

  1. Heuristic on the text at the cursor (code / email / chat / doc)
  2. 30-minute cache of the last successful detection
  3. Time-of-day + weekend heuristic as a soft default

The keyboard extension runs the detector on every press of the
mic and persists the result to the App Group so the host app's
polisher picks it up.

## 3. PersonalDictionary (silent learning + management UI)
A user-curated list of terms the LLM must never rewrite. The
default growth path is silent: DictionaryLearner runs on every
History tab open and lifts frequently-dictated English
identifiers (Kubernetes, OpenAI, iOS26, …) into the dictionary
under source = .history. Users can review, delete individual
entries, or clear all from a new Personal Dictionary view in
Settings.

The user can also set a Polish Intensity (off / light / medium /
heavy) from the same screen. Default is medium, which is what
Typeless and Wispr Flow also use.

## Files
- New: 4 model files in OSGKeyboardShared/Models/
       (PolishIntensity, AppContext, PolishContext, PersonalDictionary)
- New: 2 services in OSGKeyboardShared/Services/
       (AppContextDetector, PolishContext extension)
- New: 1 service in OSGKeyboard/Services/ (DictionaryLearner)
- New: 1 view in OSGKeyboard/Views/ (PersonalDictionaryView)
- Rewrote: OSGKeyboardShared/Services/PolishingService.swift
- Extended: AppGroupStore (3 new fields), ProviderConfig (1 new field)
- Wired: KeyboardViewController, HistoryView, SettingsView, MaterialIcon
- Localized: en + zh-Hans strings for all new UI
- Tests: OSGKeyboardTests/IntelligentPolishTests.swift (16 tests)

## Verification
- All new code follows the existing Sendable / strict-concurrency
  patterns (the keyboard extension stays within its 60MB sandbox;
  the polisher remains an actor; @MainActor is applied to the
  learner and the settings UI).
- Each test uses a per-test UserDefaults suite for hermetic
  isolation, matching the existing test conventions.
- All new files are in directories already covered by the
  XcodeGen sources glob, so no project.yml change is needed.

## Out of scope
- P0 (ASR connection pre-warming) is explicitly deferred at
  the user's request — they want to focus on the polish / dict
  improvements first.
- The Cloud polish (WebSocket) work is not touched.

## Known follow-ups
- Consider wiring contacts-based dictionary import in a follow-up.
- Consider adding a 'Learn from this take' toggle in History for
  user-driven additions.
- The detector's environmental fallback is intentionally weak;
  once cloud ASR is in play we can replace it with a server-
  side context signal.
This commit is contained in:
Mavis
2026-07-03 07:01:55 +00:00
parent dc9697bf3d
commit c5b2e21edf
19 changed files with 1748 additions and 26 deletions
+175 -26
View File
@@ -1,21 +1,37 @@
// PolishingService.swift
// OSGKeyboard · Shared
//
// Takes raw ASR transcript and runs it through the user's configured LLM
// to produce polished, well-punctuated text. Falls back to the raw transcript
// if the LLM call fails or times out.
// v0.3.0 rewrite: one-step "intelligent" polish that combines ASR
// error correction, filler removal, and tone adaptation in a single
// LLM call. The previous design was two separate steps (correction
// then polish) which doubled latency and token cost; Typeless,
// Wispr Flow, and the "intelligent" rewrite literature all confirm
// the merged prompt performs just as well for everyday Chinese /
// English dictation while halving the network round-trip.
//
// Engine matrix:
// - `engineMode == "cloud"` always polish (cloud engine's whole point).
// - `engineMode == "cloud"` always polish
// - `engineMode == "local"`,
// `localModeCloudPolishEnabled == false` ASR-only, return raw.
// `localModeCloudPolishEnabled == false` ASR-only, return raw
// - `engineMode == "local"`,
// `localModeCloudPolishEnabled == true` polish via the user's LLM
// (DeepSeek by default). The local engine gains stronger accuracy on
// noisy / dialectal Chinese at the cost of one cloud round-trip.
// If the user hasn't entered an API key the call falls back to the
// raw transcript and surfaces a warning so the keyboard can show
// the "fill in your key" hint.
// `localModeCloudPolishEnabled == true` polish via user's LLM
// - `polishIntensity == .off` ASR-only, return raw,
// regardless of engine mode
// - Missing API key return raw + throw
// `.missingAPIKey` so the caller can show the "fill in your key"
// hint inline
//
// Caller-supplied `PolishContext` carries the per-call signals:
// - `appContext` code / email / chat / document / unknown
// - `intensity` off / light / medium / heavy (per-call
// override; default is the user-configured value)
// - `precedingText` optional tail of the cursor's preceding text
// for reference resolution
//
// The prompt is intentionally a single message; multi-message
// conversation history would let earlier hallucinations pollute
// later calls (see MIT 2026 "Do LLMs Benefit From Their Own Words?")
// and the user expectation is that each take is independent.
import Foundation
@@ -52,29 +68,56 @@ public actor PolishingService {
self.timeout = timeout ?? (LLMClientFactory.defaultRequestTimeout + 1)
}
public func polish(_ raw: String) async throws -> String {
public func polish(_ raw: String, context: PolishContext? = nil) async throws -> String {
let trimmed = raw.trimmingCharacters(in: .whitespacesAndNewlines)
guard !trimmed.isEmpty else { throw PolishError.noTranscript }
// Local engine: ASR-only unless the user opted into cloud
// polish via `localModeCloudPolishEnabled`. The cloud polish
// path still requires an API key; if the Keychain is empty we
// fall back to the raw transcript and throw `missingAPIKey`
// so the UI can surface the "fill in your key" hint.
if store.engineMode == "local" {
guard store.localModeCloudPolishEnabled else { return trimmed }
guard !store.apiKey.isEmpty else {
throw PolishError.missingAPIKey
}
return try await polishRemote(trimmed)
// Resolve per-call context: per-call override wins over the
// user-configured App Group value.
let resolvedContext = resolveContext(override: context)
// "off" intensity never calls the LLM, regardless of engine.
// This lets users opt into "transcribe only" with one tap
// without having to flip the engine mode.
if resolvedContext.intensity == .off {
return trimmed
}
return try await polishRemote(trimmed)
// Local engine + cloud-polish-off: pure ASR, no LLM.
if store.engineMode == "local", !store.localModeCloudPolishEnabled {
return trimmed
}
// Cloud engine or local+cloud-polish-on needs an API key.
guard !store.apiKey.isEmpty else {
throw PolishError.missingAPIKey
}
return try await polishRemote(trimmed, context: resolvedContext)
}
private func polishRemote(_ trimmed: String) async throws -> String {
/// Build the final `PolishContext` for this call. Per-call
/// overrides take precedence; otherwise we read the user-configured
/// values out of the App Group (so the keyboard extension's
/// `PolishingService` instance does not need to know about
/// `ProviderConfig`).
private func resolveContext(override: PolishContext?) -> PolishContext {
guard let override else {
return PolishContext(
appContext: store.detectedAppContext?.context ?? .unknown,
intensity: store.polishIntensity
)
}
// If the override leaves a field at its default-when-nil
// value, fall back to the App Group value. Today every
// `PolishContext` field is non-optional so this branch
// simply forwards; kept for future-proofing.
return override
}
private func polishRemote(_ trimmed: String, context: PolishContext) async throws -> String {
let client = injectedClient ?? store.makeClient()
let prompt = store.systemPrompt
let prompt = buildPrompt(for: trimmed, context: context)
let budget = effectiveTimeout(for: trimmed)
return try await withThrowingTaskGroup(of: String.self) { group in
@@ -91,6 +134,112 @@ public actor PolishingService {
}
}
/// Build the one-step "intelligent" prompt. The structure is:
/// 1. Role
/// 2. Three numbered tasks (correction, polish, style)
/// 3. Hard rules (do-not-modify list, length cap, short-circuit)
/// 4. User dictionary block (if any)
/// 5. Context + intensity guidelines
/// 6. Optional preceding text
/// 7. The transcript to process
/// 8. Output contract
///
/// The Chinese / English split mirrors the existing per-provider
/// default system prompt in `AppGroupStore.defaultSystemPrompt(for:)`
/// so the polish step stays in the user's chosen output language.
internal func buildPrompt(for text: String, context: PolishContext) -> String {
let dictionary = store.personalDictionary
let dictionaryBlock = dictionary.promptFragment()
let contextGuideline = context.appContext.polishGuideline
let intensityGuideline = context.intensity.promptGuideline
let precedingBlock = context.precedingForPrompt
.map { "上文(仅供参考,**不要**改写):\n\($0)\n" } ?? ""
let useChinese = shouldUseChineseGuidance(providerId: store.providerId)
if useChinese {
return """
你是智能语音输入法的后处理引擎。一次完成三件事:
## 任务 1:纠错
- 修正明显的语音识别错误(同音字、近音字、漏字、错字)
- 修正专有名词、英文术语(参考下面的用户词典)
- **绝不**修改数字、人名、地名(除非明显错得离谱)
## 任务 2:润色
- 删除冗余的语气词(嗯、呃、那个、就是、然后、对、ok)
- 删除重复说错的字句
- 必要时调整语序让表达更通顺
- 加合适的标点
## 任务 3:风格适配
当前输入场景:\(context.appContext.rawValue)
风格要求:\(contextGuideline)
润色档位:\(intensityGuideline)
## 重要规则
1. **最小改动原则**:原文已经能听懂的部分不要重写
2. 保留说话人的口吻和意图
3. 不添加原文中没有的信息
4. 短句(≤ 8 个中文字符 或 ≤ 15 个英文字符)直接原样返回,不要润色
5. 输出语言必须与原文一致
\(dictionaryBlock.isEmpty ? "" : "## 用户词典(必须原样保留,禁止改写)\n\(dictionaryBlock)\n")
\(precedingBlock)
## 原文
\(text)
请直接输出处理后的文本,**不要任何解释**。
"""
} else {
return """
You are the post-processing engine of a voice-input keyboard. Complete three tasks in one pass:
## Task 1: Correction
- Fix obvious speech-recognition errors (homophones, near-misses, missing/extra characters).
- Correct proper nouns, English terms, and technical identifiers (see the user dictionary below).
- **Never** alter numbers, person names, or place names unless clearly wrong.
## Task 2: Polish
- Remove redundant filler words (um, uh, like, you know, basically).
- Remove duplicated fragments the speaker self-corrected.
- Adjust obviously broken word order.
- Add appropriate punctuation and capitalization.
## Task 3: Style adaptation
Current input context: \(context.appContext.rawValue)
Style guideline: \(contextGuideline)
Polish intensity: \(intensityGuideline)
## Hard rules
1. Minimum-change principle: do not rewrite parts the user already said clearly.
2. Preserve the speaker's voice and intent.
3. Never add information that is not in the original.
4. Short inputs (≤ 15 English words or ≤ 8 CJK characters) must be returned verbatim.
5. Output language must match the input language.
\(dictionaryBlock.isEmpty ? "" : "## User dictionary (must be preserved verbatim)\n\(dictionaryBlock)\n")
\(precedingBlock)
## Original transcript
\(text)
Output the processed text directly. **No explanation, no quotes, no preamble.**
"""
}
}
/// Mirror `AppGroupStore.defaultSystemPrompt(for:)` Chinese LLM
/// providers get a Chinese prompt, English ones get English.
/// Keeping these aligned avoids the "model answers in the wrong
/// language" failure mode that LLM benchmarks consistently flag.
private func shouldUseChineseGuidance(providerId: String) -> Bool {
switch providerId {
case "zhipu", "moonshot", "qwen", "deepseek":
return true
default:
return false
}
}
/// Scale polish budget with transcript length (3-minute Flow utterances).
private func effectiveTimeout(for text: String) -> TimeInterval {
let scaled = timeout + (Double(text.count) / 200.0) * 2.0