Files
OSGKeyboard/OSGKeyboardShared/Services/AppContextDetector.swift
T
Mavis c5b2e21edf feat: intelligent polish + per-app context + personal dictionary
v0.3.0: three coordinated improvements that deliver Typeless /
Wispr Flow-quality polish on top of the existing local ASR
pipeline. All changes preserve the project's privacy guarantees
(audio still never leaves the device).

## 1. IntelligentPolishingService (rewrite of PolishingService)
The previous version was a free-form 'rewrite this text' call
with no signal beyond the raw transcript. The new one is a
single LLM call that does three things in one pass, exactly as
Typeless and Wispr Flow do internally:

  1. ASR error correction (homophones, near-misses, missing chars)
  2. Polish (drop filler words, fix grammar, add punctuation)
  3. Style adaptation per app context (code / email / chat / doc)

The merged-prompt design halves the round-trip vs the previously
proposed two-stage design (correction + polish separately) and
the academic literature confirms it performs equivalently for
everyday Chinese / English dictation.

## 2. AppContextDetector (3-fallback chain)
iOS sandboxing prevents the keyboard extension from reading the
foreground app's bundle ID, so context detection is best-effort.
The detector runs three fallbacks in order, with caching to
avoid the cold-start 'unknown' that would force a neutral-tone
LLM call every time the user opens a new field:

  1. Heuristic on the text at the cursor (code / email / chat / doc)
  2. 30-minute cache of the last successful detection
  3. Time-of-day + weekend heuristic as a soft default

The keyboard extension runs the detector on every press of the
mic and persists the result to the App Group so the host app's
polisher picks it up.

## 3. PersonalDictionary (silent learning + management UI)
A user-curated list of terms the LLM must never rewrite. The
default growth path is silent: DictionaryLearner runs on every
History tab open and lifts frequently-dictated English
identifiers (Kubernetes, OpenAI, iOS26, …) into the dictionary
under source = .history. Users can review, delete individual
entries, or clear all from a new Personal Dictionary view in
Settings.

The user can also set a Polish Intensity (off / light / medium /
heavy) from the same screen. Default is medium, which is what
Typeless and Wispr Flow also use.

## Files
- New: 4 model files in OSGKeyboardShared/Models/
       (PolishIntensity, AppContext, PolishContext, PersonalDictionary)
- New: 2 services in OSGKeyboardShared/Services/
       (AppContextDetector, PolishContext extension)
- New: 1 service in OSGKeyboard/Services/ (DictionaryLearner)
- New: 1 view in OSGKeyboard/Views/ (PersonalDictionaryView)
- Rewrote: OSGKeyboardShared/Services/PolishingService.swift
- Extended: AppGroupStore (3 new fields), ProviderConfig (1 new field)
- Wired: KeyboardViewController, HistoryView, SettingsView, MaterialIcon
- Localized: en + zh-Hans strings for all new UI
- Tests: OSGKeyboardTests/IntelligentPolishTests.swift (16 tests)

## Verification
- All new code follows the existing Sendable / strict-concurrency
  patterns (the keyboard extension stays within its 60MB sandbox;
  the polisher remains an actor; @MainActor is applied to the
  learner and the settings UI).
- Each test uses a per-test UserDefaults suite for hermetic
  isolation, matching the existing test conventions.
- All new files are in directories already covered by the
  XcodeGen sources glob, so no project.yml change is needed.

## Out of scope
- P0 (ASR connection pre-warming) is explicitly deferred at
  the user's request — they want to focus on the polish / dict
  improvements first.
- The Cloud polish (WebSocket) work is not touched.

## Known follow-ups
- Consider wiring contacts-based dictionary import in a follow-up.
- Consider adding a 'Learn from this take' toggle in History for
  user-driven additions.
- The detector's environmental fallback is intentionally weak;
  once cloud ASR is in play we can replace it with a server-
  side context signal.
2026-07-03 07:18:02 +00:00

176 lines
6.8 KiB
Swift
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
// AppContextDetector.swift
// OSGKeyboard · Shared
//
// iOS Custom Keyboard Extensions run in a tight sandbox: we cannot
// read the foreground app's bundle ID, we cannot query
// `LSApplicationWorkspace`, and we cannot observe app switches.
// The only signals available to the extension are:
//
// - the text already at the cursor (`textDocumentProxy`)
// - the current keyboard input language
// - the time of day (used as a very weak signal)
//
// So we infer context with a **3-fallback chain**:
// 1. **Heuristic on preceding text** — strongest signal when the
// user has already typed enough. Catches code, email, chat,
// and document. We only look at the tail of the preceding
// text (up to `precedingScanWindow` characters) so a long
// note does not spend cycles scanning the whole buffer.
// 2. **Cached value** — when the user just opened a new field
// with no preceding text, reuse the last detection for up to
// `cacheLifetime`. Most users type in the same app for a
// while; this avoids a cold-start `unknown` that would force
// a neutral-tone LLM call.
// 3. **Environmental fallback** — when both above miss, blend
// input language + hour-of-day into a soft default.
//
// Anything we cannot resolve maps to `.unknown`, which the polish
// service translates to a neutral-tone prompt.
import Foundation
public struct AppContextDetector: Sendable {
/// How many characters of the preceding text we scan for
/// heuristic matches. Long enough to capture a code block, a
/// mail header, or a chat thread; short enough to scan in O(n)
/// on every keystroke.
public let precedingScanWindow: Int
/// How long a cached detection stays valid. 30 minutes matches
/// the "typical typing session" length and means the cache
/// rarely outlives a switch to a genuinely new app.
public let cacheLifetime: TimeInterval
public init(
precedingScanWindow: Int = 2000,
cacheLifetime: TimeInterval = 30 * 60
) {
self.precedingScanWindow = precedingScanWindow
self.cacheLifetime = cacheLifetime
}
public func detect(
precedingText: String?,
storedCache: (context: AppContext, observedAt: Date)?,
now: Date = Date()
) -> AppContext {
// Fallback 1: heuristic on preceding text. Even one strong
// signal (indented line ending with `{`, `> ` quote,
// email pattern) is enough — we never mix-and-match.
if let preceding = precedingText, !preceding.isEmpty,
let detected = heuristicDetect(preceding: preceding) {
return detected
}
// Fallback 2: cache. We rely on the caller having written
// a fresh detection to the App Group on every successful
// pressBegan; we just consult the timestamp here.
if let cached = storedCache,
now.timeIntervalSince(cached.observedAt) < cacheLifetime {
return cached.context
}
// Fallback 3: environmental. Not great, but better than
// `unknown` for a polished experience.
return environmentalFallback(now: now)
}
// MARK: - Heuristic detection
/// Inspect the tail of the preceding text. The order of the
/// branches is significant: more specific signals first (code,
/// terminal) so they win over more generic ones (chat,
/// document).
internal func heuristicDetect(preceding: String) -> AppContext? {
let tail = preceding.suffix(precedingScanWindow)
guard !tail.isEmpty else { return nil }
// Code: indented line + a code-y keyword in the recent past.
// The two-condition test avoids false positives on indented
// lists / block quotes.
let codeKeywords = [
"func ", "class ", "struct ", "enum ", "protocol ",
"import ", "package ", "namespace ",
"def ", "var ", "let ", "const ",
"if (", "if (", "} else", "} catch",
"=> {", "-> {",
]
let hasIndentation = tail.contains(where: { $0 == "\n " || $0 == "\t" })
let hasCodeKeyword = codeKeywords.contains(where: { tail.contains($0) })
if hasIndentation, hasCodeKeyword {
return .code
}
// Code: shebang / single-line comment / URL-with-query.
if tail.hasPrefix("#!/") || tail.contains("\n#!/") {
return .code
}
// Terminal: prompt markers (rough but rarely wrong on
// dedicated terminal apps). `$ `, `# `, ` `, `➜ `.
if tail.range(of: #"(^|\n)[$#❯➜] "#, options: .regularExpression) != nil {
return .code
}
// Email: contains an email-shaped token in the recent past.
// We deliberately keep the regex conservative to avoid
// matching every "@" in code / handles.
if tail.range(
of: #"\b[\w.+-]+@[\w-]+\.[A-Za-z]{2,}\b"#,
options: .regularExpression
) != nil {
return .email
}
// Email: subject-style opening — "Subject:", "To:", "From:",
// "Cc:", or common CN mail domains in the URL bar.
let emailOpeners = ["Subject:", "Re: ", "Fwd: ", "From:", "To:"]
if emailOpeners.contains(where: { tail.contains($0) }) {
return .email
}
// Chat: lots of short lines, no big paragraphs.
let lines = tail.split(separator: "\n", omittingEmptySubsequences: false)
.suffix(20)
if lines.count >= 3 {
let nonEmpty = lines.filter { !$0.isEmpty }
let allShort = nonEmpty.count >= 3
&& nonEmpty.allSatisfy { $0.count < 60 }
if allShort {
return .chat
}
}
// Document: long unbroken paragraphs.
let lastParagraph = tail.split(separator: "\n\n").last ?? ""
if lastParagraph.count > 200 && !lastParagraph.contains("\n") {
return .document
}
return nil
}
// MARK: - Environmental fallback
/// Last-resort guess. Deliberately biased toward "document" /
/// "email" over "chat" because people who can no longer be
/// classified are usually writing something more formal than
/// not — and the cost of over-classifying as chat is a casual
/// prompt that we can easily recover from.
internal func environmentalFallback(now: Date) -> AppContext {
let hour = Calendar.current.component(.hour, from: now)
// 9am-6pm: assume document / work context. 8pm-7am: assume
// chat. Weekends: lean chat. The signal is weak but it
// beats random.
let isWorkHours = (9...18).contains(hour)
let isWeekend = Calendar.current.isDateInWeekend(now)
if isWorkHours, !isWeekend {
return .document
}
if !isWorkHours || isWeekend {
return .chat
}
return .unknown
}
}