feat(macos): local ASR model manager, menu-bar polish, and release 0.5.2

Adds a bundled local ASR model catalog for the macOS app with one-click
Sherpa Qwen3 / SenseVoice downloads (pause/resume, inline actions) and a
shared model storage directory used by MLX Qwen3. Fixes the light-mode
sidebar material and makes the menu-bar icon follow the system appearance
with a refreshed status mark. Renames the built product to OSGKeyboard.app.

Bumps version to 0.5.2 (build 19).
This commit is contained in:
Rocky
2026-07-09 08:55:37 +08:00
parent c2f07bd8d2
commit 200265fbd6
50 changed files with 4666 additions and 266 deletions
+9
View File
@@ -7,20 +7,29 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## [0.5.2] - 2026-07-09
### Added
- **macOS local ASR models**: the desktop app ships a bundled model catalog with one-click download of Sherpa Qwen3 (hotwords) and SenseVoice models, plus a shared model storage directory; downloads show a circular progress ring with pause / resume, and each row has inline Download / Delete actions. / **macOS 本地 ASR 模型**:桌面 App 内置模型目录,可一键下载 Sherpa Qwen3(热词)与 SenseVoice 模型,并共用同一模型存储目录;下载显示带暂停 / 继续的环形进度,每行提供内联的下载 / 删除操作。
- **Shared model directory for MLX**: Qwen3-ASR MLX now uses a fixed subfolder inside the shared model storage — drop converted weights into the folder opened by "Open folder"; no per-model directory picker. / **MLX 共用模型目录**Qwen3-ASR MLX 改用共享模型存储中的固定子目录——把转换好的权重放入「打开目录」指向的文件夹即可,不再逐模型选目录。
- **iCloud sync hardening**: per-field settings merge (`appSettings.v2`), per-device usage statistics (G-Counter), tombstoned dictionary/history merge, and a low-risk **Sync Now** action in Settings. / **iCloud 同步加固**:设置按字段合并(`appSettings.v2`)、统计按设备 G-Counter 累计、词库/历史带墓碑合并,并在设置中新增低风险的**立即同步**操作。
### Changed
- **API key sync**: cloud provider API keys now replicate through **iCloud Keychain** when settings sync is on — never through iCloud KVS JSON. / **API 密钥同步**:开启设置同步后,云端服务商 API 密钥改由 **iCloud 钥匙串**复制,不再写入 iCloud KVS JSON。
- **Speech history cap**: synced history limit is **300** entries (aligned with the sync payload). / **语音历史上限**:可同步历史上限为 **300** 条(与同步载荷一致)。
- **macOS app name**: the built product is now `OSGKeyboard.app` (was `OSGKeyboardMac.app`); Dock, About, and Finder all read **OSGKeyboard**. / **macOS 应用名称**:编译产物改为 `OSGKeyboard.app`(原 `OSGKeyboardMac.app`);Dock、关于窗口与 Finder 均显示 **OSGKeyboard**
- **macOS local recognition label**: the Settings entry is now simply "Local Recognition" and no longer names a specific model. / **macOS 本地识别标签**:设置项改为「本地识别」,不再绑定具体模型名称。
### Fixed
- **macOS menu-bar icon in light mode**: the status-bar icon now follows the *system* menu-bar appearance, so forcing the app into Light while the system is Dark no longer renders an unreadable dark icon; a refreshed status mark is used. / **macOS 菜单栏图标(浅色模式)**:状态栏图标改为跟随*系统*菜单栏外观,App 强制浅色而系统为深色时不再出现看不清的深色图标;并更新了状态栏图标。
- **macOS light-mode sidebar**: restored the native translucent sidebar material so the light appearance matches system apps (e.g. System Settings, Notes) instead of a flat grey fill. / **macOS 浅色侧边栏**:恢复原生半透明侧栏材质,浅色外观与系统应用(如系统设置、备忘录)一致,不再是扁平灰底。
- **Settings sync wiping API keys**: pulling a legacy settings blob without API key fields no longer deletes local Keychain entries. / **设置同步清空 API 密钥**:拉取不含 API 密钥字段的旧版设置包时,不再删除本地 Keychain 项。
- **Cross-device settings conflicts**: changing different settings on two devices no longer lets one device's full blob overwrite the other's unrelated fields. / **跨设备设置冲突**:两台设备分别修改不同设置项时,不再因整包覆盖而冲掉对方未改动的字段。
- **Usage statistics under-counting**: offline usage on multiple devices now sums correctly instead of taking per-field `max()`. / **使用统计少计**:多设备离线各自累计后合并为求和,不再对总量取 `max()`
- **Dictionary/history resurrection**: deletes and "clear all" on one device propagate via tombstones so older remote entries cannot come back. / **词库/历史复活**:单设备删除或清空会通过墓碑传播,远端旧条目无法复活。
- **Flow false-ready mic state**: the keyboard mic now stays orange until the host app publishes a real ready contract (capture engine live + polling idle), not merely a fresh heartbeat; green tap-to-talk and jump-to-host behavior share the same `MicVoiceAvailability` gate, and orphaned `stopped` signals self-heal instead of hanging until timeout. / **Flow 伪就绪麦克风状态**:键盘麦克风在主 App 发布真实就绪合约(音频引擎在跑且轮询空闲)之前保持橙色,不再仅凭心跳误判;绿色「点按说话」与跳转主 App 共用同一 `MicVoiceAvailability` 闸门,孤立的 `stopped` 信号会自愈而不再长时间卡住。
- **Flow mic stuck orange after ready**: a single stale cross-process heartbeat read no longer flips a healthy session into a sticky "session ended" error that forced the mic orange. The "session ended" hint now fires only when the (heartbeat-independent) session contract truly drops; a brief read jitter is smoothed by a ready grace window, and a lingering expired hint auto-recovers to green once the host is ready again. / **就绪后麦克风卡橙色**:单次跨进程心跳读数抖动不再把健康会话打成粘滞的「会话已结束」错误、强制麦克风变橙。「会话已结束」提示现仅在(不依赖心跳的)会话合约真正失效时触发;短暂读数抖动由就绪宽限期平滑,遗留的过期提示会在宿主重新就绪后自动恢复为绿色。
- **Orphaned Live Activity after force-quit**: force-quitting the app no longer leaves a stale OSGKeyboard status stuck on the Lock Screen / Dynamic Island. The `staleDate` is now ~45s (refreshed by the heartbeat while the session is alive) so the system reclaims a dead session's island on its own, and every app foreground now sweeps leftover Live Activities *before* trying to (re)start a session — so even a start that later fails (e.g. mic proof timeout) still clears the zombie island. / **强杀后遗留 Live Activity**:强制退出 App 不再在锁屏 / 灵动岛留下无法消失的 OSGKeyboard 状态。`staleDate` 缩短为约 45 秒(会话存活期间由心跳持续刷新),系统会自动回收已死会话的灵动岛;且每次 App 回到前台都会**先**清扫遗留的 Live Activity 再尝试(重新)启动会话——即便本次启动随后失败(如麦克风就绪超时),也不会留下僵尸灵动岛。
## [0.5.0] - 2026-07-07
@@ -0,0 +1,25 @@
{
"images" : [
{
"filename" : "OSGStatusMark.png",
"idiom" : "universal",
"scale" : "1x"
},
{
"idiom" : "universal",
"scale" : "2x"
},
{
"idiom" : "universal",
"scale" : "3x"
}
],
"info" : {
"author" : "xcode",
"version" : 1
},
"properties" : {
"preserves-vector-representation" : true,
"template-rendering-intent" : "template"
}
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 3.9 KiB

@@ -10,12 +10,16 @@ import OSGKeyboardShared
enum FlowLiveActivityController {
nonisolated(unsafe) private static var currentActivity: Activity<FlowActivityAttributes>?
/// Last phase pushed to the Live Activity so `keepAlive()` can refresh the
/// `staleDate` without changing what the user sees.
nonisolated(unsafe) private static var currentPhase: FlowActivityAttributes.ContentState.Phase = .idle
/// If the host app is force-quit its `endSession()` never runs, orphaning
/// the Live Activity. A `staleDate` lets the system grey it out and become
/// willing to reclaim it without our process refreshed on every update
/// so a genuinely active, in-use session never looks stale.
private static let staleWindow: TimeInterval = 60 * 60
/// the Live Activity. A short `staleDate` lets the system grey it out and
/// reclaim it on its own within ~45s of the process dying. While the host is
/// alive the heartbeat calls `keepAlive()` well inside this window, so a
/// genuinely active session never looks stale.
private static let staleWindow: TimeInterval = 45
private static func freshContent(
phase: FlowActivityAttributes.ContentState.Phase
@@ -41,6 +45,7 @@ enum FlowLiveActivityController {
}
do {
currentPhase = .idle
currentActivity = try Activity.request(
attributes: FlowActivityAttributes(),
content: freshContent(phase: .idle),
@@ -54,14 +59,28 @@ enum FlowLiveActivityController {
static func update(phase: FlowActivityAttributes.ContentState.Phase) {
guard let activity = currentActivity else { return }
currentPhase = phase
let content = freshContent(phase: phase)
Task {
await activity.update(content)
}
}
/// Push a fresh `staleDate` without changing the visible phase. The host
/// heartbeat calls this well inside `staleWindow` so an in-use session
/// never looks stale; once the process dies the refreshes stop and the
/// system reclaims the orphaned Live Activity on its own.
static func keepAlive() {
guard let activity = currentActivity else { return }
let content = freshContent(phase: currentPhase)
Task {
await activity.update(content)
}
}
/// Dismiss the island presentation when the Flow session ends.
static func endSession() {
currentPhase = .idle
guard let activity = currentActivity else {
endStaleActivities()
return
@@ -74,6 +93,18 @@ enum FlowLiveActivityController {
}
}
/// Clear Live Activities orphaned by a previous (force-quit) host process.
///
/// Safe to call on every app foreground: when this process already owns a
/// Live Activity (`currentActivity != nil`) we leave it alone so a healthy
/// running session is never torn down; we only sweep leftovers that belong
/// to a dead process. Call this *before* attempting to (re)start a session
/// so a failed start (e.g. mic timeout) still clears the stale island.
static func clearOrphanedActivities() {
guard currentActivity == nil else { return }
endStaleActivities()
}
/// Host relaunch can leave orphan activities; clear them before starting anew.
private static func endStaleActivities() {
let staleActivities = Activity<FlowActivityAttributes>.activities
+397 -65
View File
@@ -25,9 +25,7 @@ final class FlowSessionManager: ObservableObject {
private let capture = FlowContinuousCapture()
private let store = AppGroupStore()
/// Cloud-engine polish; local engine runs through built-in DeepSeek polish.
private var polisher: PolishingService {
PolishingService()
}
private let polisher = PolishingService()
/// Cached ASR instance. v0.2.0: the only on-device backend is iOS
/// `SpeechAnalyzer`, which has no warm-up step we can hand the
/// factory-built service straight back without going through the
@@ -49,17 +47,26 @@ final class FlowSessionManager: ObservableObject {
private var expiryTask: Task<Void, Never>?
private var levelTask: Task<Void, Never>?
private var startTask: Task<Void, Never>?
private var commandObserver: FlowSessionDarwinObserver?
/// Last recording state the poll loop observed logs only on transition.
private var lastObservedRecordingState: FlowSessionKeys.RecordingState = .idle
private var activeSessionId: UUID?
private var currentUtteranceId: UUID?
private var currentCommandSeq: Int64 = 0
private var lastHandledCommandSeq: Int64 = 0
private var isUtteranceRecording = false
/// True from `stopped` until the result/error is written back to App Group.
private var isUtteranceProcessing = false
private var finalizeTask: Task<Void, Never>?
private var asrTask: Task<Void, Never>?
private var utteranceSafetyTask: Task<Void, Never>?
private var chunkedPipeline: ChunkedUtterancePipeline?
private var currentPartial = ""
private var lastFinal = ""
private var chunkWarnings: [String] = []
private var lastReadyTraceSignature = ""
private var lastCommandFingerprint = ""
private var lastIgnoredCommandSignature = ""
/// Wall-clock span of the current mic-open utterance (excludes LLM polish).
private var utteranceRecordingStartedAt: Date?
/// True while the host app scene is `.active` drives foreground renewal.
@@ -67,7 +74,11 @@ final class FlowSessionManager: ObservableObject {
private var backgroundTaskID: UIBackgroundTaskIdentifier = .invalid
/// True while handling a keyboard-initiated `startflow` cold start.
private var isColdStartHandoff = false
private static let coldStartAudioProofTimeout: TimeInterval = 2.5
private var coldStartRecoveryTask: Task<Void, Never>?
/// Initial proof window cold mic sessions often need >2.5s after app switch.
private static let coldStartAudioProofTimeout: TimeInterval = 6
/// Extra window after the first timeout while the overlay shows a failure hint.
private static let coldStartRecoveryProofTimeout: TimeInterval = 12
init() {
// Sessions are (re)started explicitly on app foreground via
@@ -83,6 +94,10 @@ final class FlowSessionManager: ObservableObject {
/// Starts a Flow session: permissions continuous capture App Group active.
func startSession(duration: TimeInterval? = nil, coldStart: Bool = false) {
traceState(
"startSession.request",
extra: "coldStart=\(coldStart) duration=\(Int(duration ?? FlowSessionPolicy.sessionDuration()))"
)
guard AppGroup.isAvailable else {
debug("cannot start flow session: App Group unavailable")
return
@@ -106,7 +121,10 @@ final class FlowSessionManager: ObservableObject {
return
}
guard !isStarting else { return }
guard !isStarting else {
traceState("startSession.ignored", extra: "reason=alreadyStarting")
return
}
startTask?.cancel()
startTask = Task { @MainActor [weak self] in
@@ -148,6 +166,14 @@ final class FlowSessionManager: ObservableObject {
/// behind (its `endSession()` could not run at kill time).
func activateOnForeground() {
guard AppGroup.isAvailable else { return }
// Sweep any Live Activity a previous (force-quit) process left behind
// *before* we try to (re)start a session. Doing it here rather than
// only inside `startSession()`'s success path means a start that
// later fails (e.g. mic proof timeout) still clears the stale island
// instead of leaving a zombie on the lock screen / Dynamic Island.
// No-op when this process already owns a healthy Live Activity.
FlowLiveActivityController.clearOrphanedActivities()
guard AppPermissions.flowRequirementsMet else {
sessionWarning = permissionWarningMessage()
FlowSessionBridge.setHostReady(false)
@@ -161,7 +187,8 @@ final class FlowSessionManager: ObservableObject {
}
func dismissColdStartOverlay() {
guard coldStartContext?.state != .preparing else { return }
coldStartRecoveryTask?.cancel()
coldStartRecoveryTask = nil
if isActive {
refreshHostReady()
}
@@ -189,8 +216,11 @@ final class FlowSessionManager: ObservableObject {
coldStartContext = nil
isColdStartHandoff = false
coldStartRecoveryTask?.cancel()
coldStartRecoveryTask = nil
startTask?.cancel()
startTask = nil
commandObserver = nil
pollingTask?.cancel()
pollingTask = nil
heartbeatTask?.cancel()
@@ -201,6 +231,8 @@ final class FlowSessionManager: ObservableObject {
levelTask = nil
finalizeTask?.cancel()
finalizeTask = nil
utteranceSafetyTask?.cancel()
utteranceSafetyTask = nil
if isUtteranceRecording || isUtteranceProcessing {
capture.cancelUtterance()
@@ -210,6 +242,10 @@ final class FlowSessionManager: ObservableObject {
}
asrTask = nil
chunkedPipeline = nil
activeSessionId = nil
currentUtteranceId = nil
currentCommandSeq = 0
lastHandledCommandSeq = 0
isUtteranceRecording = false
isUtteranceProcessing = false
@@ -305,9 +341,11 @@ final class FlowSessionManager: ObservableObject {
guard isActive else { return }
if capture.running {
capture.reassertIfRunning()
if capture.engineHasRecentAudio() {
let reasserted = capture.reassertIfRunning()
if reasserted, capture.engineHasRecentAudio() {
sessionWarning = nil
} else if !reasserted {
sessionWarning = AppL10n.string("flow.error.audioUnavailable")
}
refreshHostReady()
return
@@ -329,22 +367,94 @@ final class FlowSessionManager: ObservableObject {
/// Publish whether the keyboard can start a new utterance without jumping to the host app.
private func refreshHostReady() {
guard isActive else {
FlowSessionBridge.setHostReady(false)
FlowSessionBridge.writeReadySnapshot(
FlowReadySnapshot(
sessionId: activeSessionId,
ready: false,
reason: .noSession,
engineMode: store.engineMode,
localeId: store.localeId,
sessionExpiresAt: FlowSessionBridge.sessionExpiresAt()
)
)
return
}
let pollingAlive = pollingTask != nil && pollingTask?.isCancelled != true
// Steady-state ready uses structural engine liveness. The stricter
// "recent audio frame" proof is reserved for cold-start handoff only
// (`waitForAudioProof`) so brief UI-driven session hiccups do not
// drop the keyboard back to orange while the host app is foreground.
let hasRecentAudio = capture.engineHasRecentAudio(maxAge: 2)
let canAcceptUtterance = capture.engineIsLive
&& pollingAlive
&& hasRecentAudio
&& !isUtteranceRecording
&& !isUtteranceProcessing
&& sessionWarning == nil
FlowSessionBridge.setHostReady(canAcceptUtterance)
let reason: FlowReadySnapshot.Reason
if canAcceptUtterance {
reason = .ready
} else if sessionWarning != nil {
reason = .error
} else if isUtteranceRecording {
reason = .recording
} else if isUtteranceProcessing {
reason = .processing
} else if !capture.engineIsLive {
reason = .audioEngineNotLive
} else if !hasRecentAudio {
reason = .waitingForAudioProof
} else {
reason = .starting
}
let now = Date().timeIntervalSince1970
FlowSessionBridge.writeReadySnapshot(
FlowReadySnapshot(
sessionId: activeSessionId,
ready: canAcceptUtterance,
reason: reason,
heartbeatAt: now,
readyAt: canAcceptUtterance ? now : nil,
audioProofAt: hasRecentAudio ? now : nil,
engineMode: store.engineMode,
localeId: store.localeId,
busyUtteranceId: isUtteranceRecording || isUtteranceProcessing ? currentUtteranceId : nil,
sessionExpiresAt: FlowSessionBridge.sessionExpiresAt()
)
)
let signature = [
canAcceptUtterance ? "ready=1" : "ready=0",
"reason=\(reason.rawValue)",
capture.engineIsLive ? "engine=live" : "engine=dead",
hasRecentAudio ? "audio=fresh" : "audio=stale",
isUtteranceRecording ? "recording=1" : "recording=0",
isUtteranceProcessing ? "processing=1" : "processing=0",
sessionWarning == nil ? "warning=0" : "warning=1"
].joined(separator: "|")
if signature != lastReadyTraceSignature {
lastReadyTraceSignature = signature
traceState("hostReady.update", extra: signature)
}
reconcileColdStartOverlayIfRecovered()
}
/// When the host contract turns green while the cold-start overlay still
/// shows a stale preparing/failed snapshot, heal automatically.
private func reconcileColdStartOverlayIfRecovered() {
guard isColdStartHandoff, isActive else { return }
guard FlowSessionBridge.isHostReady() else { return }
guard let context = coldStartContext else { return }
switch context.state {
case .preparing:
presentColdStartReadyOverlay()
case .failed:
sessionWarning = nil
coldStartRecoveryTask?.cancel()
coldStartRecoveryTask = nil
dismissColdStartOverlay()
case .ready:
break
}
}
/// Home preview field gained focus while this app is the Flow host.
@@ -378,12 +488,14 @@ final class FlowSessionManager: ObservableObject {
// MARK: - Session start
private func startSessionAsync(duration: TimeInterval?) async {
traceState("startSessionAsync.begin")
isStarting = true
sessionWarning = nil
defer { isStarting = false }
guard AppPermissions.flowRequirementsMet else {
sessionWarning = permissionWarningMessage()
traceState("startSessionAsync.blocked", extra: "reason=permissions")
FlowSessionBridge.setHostReady(false)
if isColdStartHandoff {
showColdStartPermissionFailure()
@@ -397,6 +509,7 @@ final class FlowSessionManager: ObservableObject {
} catch {
let message = (error as? LocalizedError)?.errorDescription ?? error.localizedDescription
sessionWarning = message
traceState("startSessionAsync.failed", extra: "reason=captureStart error=\(message)")
FlowSessionBridge.setHostReady(false)
if isColdStartHandoff {
showColdStartAudioFailure(message: message)
@@ -408,23 +521,36 @@ final class FlowSessionManager: ObservableObject {
guard await waitForAudioProof() else {
let message = AppL10n.string("flow.coldStart.error.audioTimeout")
sessionWarning = message
capture.stop()
traceState("startSessionAsync.failed", extra: "reason=audioProofTimeout")
FlowSessionBridge.setHostReady(false)
if isColdStartHandoff {
showColdStartAudioFailure(message: message)
scheduleColdStartRecovery(duration: duration)
} else {
capture.stop()
}
debug("continuous capture did not produce audio frames before timeout")
return
}
activateFlowSessionAfterAudioProof(duration: duration)
traceState("startSessionAsync.ready")
debug("Flow session started (\(Int(duration ?? FlowSessionPolicy.sessionDuration()))s inactivity window), continuous capture running")
}
private func activateFlowSessionAfterAudioProof(duration: TimeInterval?) {
let resolvedDuration = duration ?? FlowSessionPolicy.sessionDuration()
FlowSessionBridge.markSessionActive(duration: resolvedDuration)
let sessionId = activeSessionId ?? UUID()
activeSessionId = sessionId
lastHandledCommandSeq = 0
FlowSessionBridge.markSessionActive(duration: resolvedDuration, sessionId: sessionId)
FlowSessionDarwin.postSessionChanged()
isActive = true
ScreenWakeLock.acquire()
sessionExpiresAt = Date().addingTimeInterval(resolvedDuration)
startHeartbeat()
startCommandObserver()
startPolling()
startLevelPublishing()
scheduleExpiry(after: resolvedDuration)
@@ -434,7 +560,7 @@ final class FlowSessionManager: ObservableObject {
FlowLiveActivityController.startSession()
refreshHostReady()
debug("Flow session started (\(Int(resolvedDuration))s inactivity window), continuous capture running")
traceState("activateFlowSessionAfterAudioProof.done")
}
private func prepareExistingSessionForColdStartReturn() async {
@@ -445,6 +571,7 @@ final class FlowSessionManager: ObservableObject {
sessionWarning = message
FlowSessionBridge.setHostReady(false)
showColdStartAudioFailure(message: message)
scheduleColdStartRecovery(duration: nil)
debug("existing session failed cold-start audio proof")
return
}
@@ -466,15 +593,23 @@ final class FlowSessionManager: ObservableObject {
let message = AppL10n.string("flow.coldStart.error.audioTimeout")
sessionWarning = message
showColdStartAudioFailure(message: message)
scheduleColdStartRecovery(duration: nil)
debug("cold-start blocked: host ready contract not published")
return
}
let hostEntry = HostReturnService.pendingHostEntry()
let skipSwitch = FlowSessionPolicy.skipAppSwitch()
coldStartContext = FlowColdStartContext(hostEntry: hostEntry, state: .ready)
presentColdStartReadyOverlay()
}
if skipSwitch, hostEntry != nil {
private func presentColdStartReadyOverlay() {
let hostEntry = HostReturnService.pendingHostEntry()
coldStartContext = FlowColdStartContext(hostEntry: hostEntry, state: .ready)
scheduleAutoReturnToHostIfNeeded(hostEntry: hostEntry)
}
private func scheduleAutoReturnToHostIfNeeded(hostEntry: HostAppEntry?) {
let skipSwitch = FlowSessionPolicy.skipAppSwitch()
guard skipSwitch, hostEntry != nil else { return }
Task { @MainActor [weak self] in
try? await Task.sleep(nanoseconds: 450_000_000)
guard let self, self.coldStartContext?.state == .ready else { return }
@@ -483,6 +618,27 @@ final class FlowSessionManager: ObservableObject {
}
}
}
/// Keeps proving mic readiness after the first timeout instead of tearing
/// capture down many handoffs become ready a few seconds later.
private func scheduleColdStartRecovery(duration: TimeInterval?) {
coldStartRecoveryTask?.cancel()
coldStartRecoveryTask = Task { @MainActor [weak self] in
guard let self else { return }
let recovered = await self.capture.awaitAudioFlowing(
timeout: Self.coldStartRecoveryProofTimeout
)
guard !Task.isCancelled else { return }
guard self.isColdStartHandoff else { return }
guard recovered else { return }
self.sessionWarning = nil
self.traceState("coldStartRecovery.recovered")
if !self.isActive {
self.activateFlowSessionAfterAudioProof(duration: duration)
}
self.refreshHostReady()
}
}
private func showColdStartPreparing() {
@@ -546,6 +702,14 @@ final class FlowSessionManager: ObservableObject {
// MARK: - Polling
private func startCommandObserver() {
commandObserver = FlowSessionDarwinObserver(
notificationName: FlowSessionDarwin.commandNotificationName
) { [weak self] in
self?.handleKeyboardSignal()
}
}
private func startPolling() {
pollingTask?.cancel()
lastObservedRecordingState = FlowSessionBridge.recordingState()
@@ -556,46 +720,122 @@ final class FlowSessionManager: ObservableObject {
pollingTask = Task { @MainActor [weak self] in
while !Task.isCancelled {
self?.handleKeyboardSignal()
try? await Task.sleep(nanoseconds: 50_000_000)
try? await Task.sleep(nanoseconds: 500_000_000)
}
}
}
private func handleKeyboardSignal() {
let signal = FlowSessionBridge.recordingState()
if signal != lastObservedRecordingState {
// The single most important cross-process signal: proves whether the
// host actually SEES the keyboard's recording state writes.
FlowDiagnostics.log(
"poll observed recordingState \(lastObservedRecordingState.rawValue)\(signal.rawValue) " +
"[rec=\(isUtteranceRecording) proc=\(isUtteranceProcessing) fg=\(isAppForeground)]"
)
lastObservedRecordingState = signal
guard let command = FlowSessionBridge.latestCommand() else {
lastCommandFingerprint = ""
return
}
switch signal {
case .recording:
let fingerprint = "\(command.sessionId.uuidString)|\(command.utteranceId.uuidString)|\(command.action.rawValue)|\(command.commandSeq)"
guard fingerprint != lastCommandFingerprint else { return }
lastCommandFingerprint = fingerprint
handleFlowCommand(command)
}
private func handleFlowCommand(_ command: FlowCommand) {
guard let activeSessionId, command.sessionId == activeSessionId else {
traceIgnoredCommand(
reason: "staleSession",
command: command,
detail: "commandSession=\(command.sessionId)"
)
return
}
guard command.commandSeq > lastHandledCommandSeq else {
traceIgnoredCommand(
reason: "seqNotIncreasing",
command: command,
detail: "last=\(lastHandledCommandSeq)"
)
return
}
lastHandledCommandSeq = command.commandSeq
lastIgnoredCommandSignature = ""
FlowDiagnostics.log(
"command \(command.action.rawValue) seq=\(command.commandSeq) utterance=\(command.utteranceId)"
)
switch command.action {
case .startRecording:
guard !isUtteranceRecording, !isUtteranceProcessing else { return }
beginUtterance()
case .stopped:
beginUtterance(utteranceId: command.utteranceId, commandSeq: command.commandSeq)
case .stopRecording:
guard currentUtteranceId == command.utteranceId else { return }
if isUtteranceRecording {
endUtterance()
} else if !isUtteranceProcessing {
FlowSessionBridge.setRecordingState(.idle)
FlowSessionBridge.storeTranscriptionError(
storeCurrentError(
AppL10n.string("flow.error.recognitionInterrupted"),
kind: .recognitionInterrupted
)
debug("stopped without active utterance — notified keyboard")
debug("stop command without active utterance — notified keyboard")
}
case .aborted:
case .abort:
guard currentUtteranceId == command.utteranceId else { return }
abortUtterance()
case .idle, .processing:
break
}
}
private func beginUtterance() {
guard capture.engineIsLive else {
private func storeCurrentPartial(_ text: String) {
guard let activeSessionId, let currentUtteranceId else { return }
let trimmed = text.trimmingCharacters(in: .whitespacesAndNewlines)
guard !trimmed.isEmpty else { return }
FlowSessionBridge.writeResult(
FlowResult(
sessionId: activeSessionId,
utteranceId: currentUtteranceId,
commandSeq: currentCommandSeq,
status: .partial,
text: trimmed
)
)
}
private func storeCurrentFinal(_ text: String, warning: String? = nil) {
let trimmed = text.trimmingCharacters(in: .whitespacesAndNewlines)
guard !trimmed.isEmpty else {
storeCurrentError(AppL10n.string("flow.error.noSpeech"), kind: .noSpeech)
return
}
guard let activeSessionId, let currentUtteranceId else { return }
FlowSessionBridge.writeResult(
FlowResult(
sessionId: activeSessionId,
utteranceId: currentUtteranceId,
commandSeq: currentCommandSeq,
status: .final,
text: trimmed,
warning: warning
)
)
}
private func storeCurrentError(
_ message: String,
kind: FlowSessionKeys.TranscriptionErrorKind = .generic,
status: FlowResult.Status = .error
) {
guard let activeSessionId, let currentUtteranceId else { return }
FlowSessionBridge.writeResult(
FlowResult(
sessionId: activeSessionId,
utteranceId: currentUtteranceId,
commandSeq: currentCommandSeq,
status: status,
text: message,
errorKind: kind
)
)
}
private func beginUtterance(utteranceId: UUID? = nil, commandSeq: Int64 = 0) {
guard capture.engineHasRecentAudio(maxAge: 2) else {
traceState("beginUtterance.blocked", extra: "reason=audioNotRecent")
failUtterance(
message: AppL10n.string("flow.error.audioUnavailable"),
kind: .audioUnavailable
@@ -603,13 +843,25 @@ final class FlowSessionManager: ObservableObject {
return
}
guard !isUtteranceProcessing else {
traceState("beginUtterance.ignored", extra: "reason=processing")
debug("beginUtterance ignored — previous utterance still processing")
return
}
bindSessionASRIfNeeded()
let expectedEngine = store.engineMode
if sessionASREngineMode != expectedEngine {
traceState(
"beginUtterance.rebindMismatch",
extra: "expectedEngine=\(expectedEngine) boundEngine=\(sessionASREngineMode ?? "nil")"
)
bindSessionASRIfNeeded(force: true)
}
// Usually already warm from session start; refresh without blocking the mic gate.
scheduleASRWarmup()
currentUtteranceId = utteranceId ?? UUID()
currentCommandSeq = commandSeq
currentPartial = ""
lastFinal = ""
chunkWarnings = []
@@ -617,6 +869,7 @@ final class FlowSessionManager: ObservableObject {
let localeId = store.localeId
FlowSessionBridge.setTranscriptionLanguage(localeId)
FlowSessionBridge.clearPendingTranscription()
FlowSessionBridge.clearResult()
let locale = SpeechLocaleResolver.resolve(localeId)
let stream = capture.beginUtterance()
@@ -625,6 +878,7 @@ final class FlowSessionManager: ObservableObject {
isUtteranceRecording = true
utteranceRecordingStartedAt = Date()
startUtteranceSafetyTimer()
refreshHostReady()
FlowLiveActivityController.update(phase: .recording)
FlowDiagnostics.log(
@@ -637,8 +891,9 @@ final class FlowSessionManager: ObservableObject {
asrTask = Task.detached(priority: .userInitiated) { [weak manager = self] in
let outcome = await pipeline.transcribe(stream: stream) { partial in
Task { @MainActor in
manager?.currentPartial = partial
FlowSessionBridge.storeTranscriptionPartial(partial)
guard let manager else { return }
manager.currentPartial = partial
manager.storeCurrentPartial(partial)
}
}
// Re-bind `manager` inside the `@MainActor` block so the
@@ -672,14 +927,33 @@ final class FlowSessionManager: ObservableObject {
debug("utterance recording started")
}
private func startUtteranceSafetyTimer() {
utteranceSafetyTask?.cancel()
let utteranceId = currentUtteranceId
utteranceSafetyTask = Task { @MainActor [weak self] in
let timeout = FlowSessionKeys.maxUtteranceDuration + 10
try? await Task.sleep(nanoseconds: UInt64(timeout * 1_000_000_000))
guard let self, !Task.isCancelled else { return }
guard self.isUtteranceRecording, self.currentUtteranceId == utteranceId else { return }
self.storeCurrentError(
AppL10n.string("flow.error.recognitionInterrupted"),
kind: .recognitionInterrupted,
status: .timeout
)
self.abortUtterance()
self.debug("utterance safety timer aborted stale recording")
}
}
private func endUtterance() {
guard isUtteranceRecording else { return }
// Close the mic gate first, then mark processing before dropping the
// recording flag so the poll loop cannot start a second utterance.
FlowSessionBridge.setRecordingState(.processing)
isUtteranceRecording = false
isUtteranceProcessing = true
utteranceSafetyTask?.cancel()
utteranceSafetyTask = nil
refreshHostReady()
FlowLiveActivityController.update(phase: .processing)
@@ -699,6 +973,8 @@ final class FlowSessionManager: ObservableObject {
isUtteranceRecording = false
isUtteranceProcessing = false
utteranceRecordingStartedAt = nil
utteranceSafetyTask?.cancel()
utteranceSafetyTask = nil
finalizeTask?.cancel()
finalizeTask = nil
asrTask?.cancel()
@@ -709,8 +985,8 @@ final class FlowSessionManager: ObservableObject {
currentPartial = ""
lastFinal = ""
chunkWarnings = []
FlowSessionBridge.storeTranscriptionPartial("")
FlowSessionBridge.setRecordingState(.idle)
currentUtteranceId = nil
currentCommandSeq = 0
FlowLiveActivityController.update(phase: .idle)
refreshHostReady()
debug("utterance aborted")
@@ -723,6 +999,8 @@ final class FlowSessionManager: ObservableObject {
isUtteranceRecording = false
isUtteranceProcessing = false
utteranceRecordingStartedAt = nil
utteranceSafetyTask?.cancel()
utteranceSafetyTask = nil
finalizeTask?.cancel()
finalizeTask = nil
asrTask?.cancel()
@@ -733,9 +1011,9 @@ final class FlowSessionManager: ObservableObject {
currentPartial = ""
lastFinal = ""
chunkWarnings = []
FlowSessionBridge.storeTranscriptionPartial("")
FlowSessionBridge.storeTranscriptionError(message, kind: kind)
FlowSessionBridge.setRecordingState(.idle)
storeCurrentError(message, kind: kind)
currentUtteranceId = nil
currentCommandSeq = 0
FlowLiveActivityController.update(phase: .idle)
refreshHostReady()
debug("utterance failed: \(message)")
@@ -747,29 +1025,37 @@ final class FlowSessionManager: ObservableObject {
) {
isUtteranceProcessing = false
utteranceRecordingStartedAt = nil
utteranceSafetyTask?.cancel()
utteranceSafetyTask = nil
finalizeTask?.cancel()
finalizeTask = nil
chunkedPipeline = nil
currentPartial = ""
lastFinal = ""
chunkWarnings = []
FlowSessionBridge.storeTranscriptionPartial("")
FlowSessionBridge.storeTranscriptionError(message, kind: kind)
FlowSessionBridge.setRecordingState(.idle)
storeCurrentError(message, kind: kind)
currentUtteranceId = nil
currentCommandSeq = 0
FlowLiveActivityController.update(phase: .idle)
refreshHostReady()
debug("utterance processing failed: \(message)")
}
private func finalizeUtterance() async {
let finalizeSessionId = activeSessionId
let finalizeUtteranceId = currentUtteranceId
let pipelineStarted = Date()
defer {
if activeSessionId == finalizeSessionId,
currentUtteranceId == finalizeUtteranceId {
isUtteranceProcessing = false
FlowSessionBridge.setRecordingState(.idle)
FlowLiveActivityController.update(phase: .idle)
touchSessionActivity()
currentUtteranceId = nil
currentCommandSeq = 0
refreshHostReady()
}
}
let asrWait = asrWaitTimeout()
FlowDiagnostics.log(
@@ -783,9 +1069,10 @@ final class FlowSessionManager: ObservableObject {
try? await Task.sleep(nanoseconds: 100_000_000)
}
if lastFinal.isEmpty, let asrTask {
FlowDiagnostics.log("ASR wait elapsed — awaiting asrTask completion")
_ = await asrTask.value
if lastFinal.isEmpty {
FlowDiagnostics.log("ASR wait elapsed — cancelling ASR task and using best available transcript")
asrTask?.cancel()
Task { await chunkedPipeline?.cancel() }
}
let asrElapsed = Date().timeIntervalSince(pipelineStarted)
@@ -803,10 +1090,7 @@ final class FlowSessionManager: ObservableObject {
(asrTask?.isCancelled == true) ? .recognitionInterrupted : .noSpeech
FlowDiagnostics.log("finalize failed: empty transcript after \(String(format: "%.1f", asrElapsed))s")
utteranceRecordingStartedAt = nil
FlowSessionBridge.storeTranscriptionError(
AppL10n.string(key),
kind: kind
)
storeCurrentError(AppL10n.string(key), kind: kind)
return
}
@@ -832,7 +1116,7 @@ final class FlowSessionManager: ObservableObject {
providerIdOverride: pipelineStore.polishProviderIdOverride
)
delivered = polished
FlowSessionBridge.storeTranscriptionResult(polished, polishWarning: chunkNote)
storeCurrentFinal(polished, warning: chunkNote)
FlowDiagnostics.log(
"polish done in \(String(format: "%.1f", Date().timeIntervalSince(polishStarted)))s " +
"total=\(String(format: "%.1f", Date().timeIntervalSince(pipelineStarted)))s"
@@ -854,7 +1138,7 @@ final class FlowSessionManager: ObservableObject {
"\(error.localizedDescription)"
)
delivered = fallback.text
FlowSessionBridge.storeTranscriptionResult(fallback.text, polishWarning: fallback.polishWarning)
storeCurrentFinal(fallback.text, warning: fallback.polishWarning)
}
SpeechHistoryStore.shared.recordUtterance(
@@ -867,7 +1151,6 @@ final class FlowSessionManager: ObservableObject {
currentPartial = ""
lastFinal = ""
chunkWarnings = []
FlowSessionBridge.storeTranscriptionPartial("")
chunkedPipeline = nil
debug("utterance finalized length=\(text.count)")
}
@@ -964,6 +1247,12 @@ final class FlowSessionManager: ObservableObject {
heartbeatTask?.cancel()
FlowSessionBridge.writeHeartbeat()
heartbeatTask = Task { @MainActor [weak self] in
// Refresh the Live Activity `staleDate` every N heartbeat ticks
// (1 Hz) well inside `FlowLiveActivityController.staleWindow` so a
// live session never looks stale, while a force-quit stops these
// refreshes and lets the system reclaim the orphaned island.
let liveActivityKeepAliveEveryTicks = 15
var tick = 0
while !Task.isCancelled {
guard let self else { break }
if self.isActive, !self.capture.engineIsLive {
@@ -971,6 +1260,10 @@ final class FlowSessionManager: ObservableObject {
}
FlowSessionBridge.writeHeartbeat()
self.refreshHostReady()
tick += 1
if tick % liveActivityKeepAliveEveryTicks == 0 {
FlowLiveActivityController.keepAlive()
}
try? await Task.sleep(nanoseconds: 1_000_000_000)
guard self.isActive else { break }
}
@@ -989,4 +1282,43 @@ final class FlowSessionManager: ObservableObject {
private func debug(_ message: String) {
FlowDiagnostics.log(message)
}
private func traceIgnoredCommand(reason: String, command: FlowCommand, detail: String) {
let signature = "\(reason)|\(command.action.rawValue)|\(command.commandSeq)|\(command.sessionId.uuidString)|\(command.utteranceId.uuidString)|\(detail)"
guard signature != lastIgnoredCommandSignature else { return }
lastIgnoredCommandSignature = signature
traceState(
"command.ignored",
extra: "reason=\(reason) action=\(command.action.rawValue) seq=\(command.commandSeq) \(detail)"
)
}
private func traceState(_ event: String, extra: String? = nil) {
let staleness = FlowSessionBridge.heartbeatStaleness().map { String(format: "%.1f", $0) } ?? "nil"
let sessionId = activeSessionId?.uuidString ?? "nil"
let utteranceId = currentUtteranceId?.uuidString ?? "nil"
let summary = [
"event=\(event)",
"active=\(isActive)",
"starting=\(isStarting)",
"coldStart=\(isColdStartHandoff)",
"sessionId=\(sessionId)",
"utteranceId=\(utteranceId)",
"cmdSeq=\(currentCommandSeq)",
"lastCmd=\(lastHandledCommandSeq)",
"recording=\(isUtteranceRecording)",
"processing=\(isUtteranceProcessing)",
"storeEngine=\(store.engineMode)",
"boundEngine=\(sessionASREngineMode ?? "nil")",
"engineLive=\(capture.engineIsLive)",
"hostReady=\(FlowSessionBridge.isHostReady())",
"sessionActive=\(FlowSessionBridge.isSessionActive())",
"heartbeatStaleness=\(staleness)"
].joined(separator: " ")
if let extra, !extra.isEmpty {
debug("[trace] \(summary) \(extra)")
} else {
debug("[trace] \(summary)")
}
}
}
+21 -8
View File
@@ -38,6 +38,17 @@ struct FlowColdStartOverlay: View {
/// Fraction of the screen height the bottom gradient occupies.
private let gradientHeightFraction: CGFloat = 0.50
/// Ready and failure states dismiss on blank tap; preparing stays
/// informational only (no accidental dismiss while proving audio).
private var allowsBlankTapDismiss: Bool {
switch context.state {
case .ready, .failed:
return true
case .preparing:
return false
}
}
var body: some View {
GeometryReader { geo in
ZStack(alignment: .bottom) {
@@ -56,6 +67,16 @@ struct FlowColdStartOverlay: View {
.frame(maxWidth: .infinity, alignment: .bottom)
.allowsHitTesting(false)
if allowsBlankTapDismiss {
// Captures taps on empty overlay space (dismiss) and blocks
// pass-through to the host shell underneath. Action buttons in
// `content` sit above this layer and remain tappable.
Color.clear
.contentShape(Rectangle())
.onTapGesture(perform: onDismiss)
.ignoresSafeArea()
}
VStack(spacing: Spacing.lg) {
content
.padding(.horizontal, Spacing.xl)
@@ -63,14 +84,6 @@ struct FlowColdStartOverlay: View {
homeIndicator
.padding(.bottom, max(geo.safeAreaInsets.bottom, Spacing.sm))
}
.allowsHitTesting(false)
if context.state == .ready {
Color.clear
.contentShape(Rectangle())
.onTapGesture(perform: onDismiss)
.ignoresSafeArea()
}
}
.frame(maxWidth: .infinity, maxHeight: .infinity, alignment: .bottom)
.ignoresSafeArea()
@@ -42,6 +42,10 @@ final class KeyboardFlowCoordinator {
private static let hostReadyGrace: TimeInterval = 4
private var flowSessionMonitorTask: Task<Void, Never>?
private var isAwaitingFlowResult = false
private var activeSessionId: UUID?
private var currentUtteranceId: UUID?
private var currentCommandSeq: Int64 = 0
private var lastAvailabilityTraceSignature = ""
init(
state: KeyboardState,
@@ -133,7 +137,9 @@ final class KeyboardFlowCoordinator {
private func recomputeMicVoiceAvailability() {
FlowSessionBridge.reloadFromDisk()
let hostReady = FlowSessionBridge.isHostReady()
let readySnapshot = FlowSessionBridge.readySnapshot()
activeSessionId = readySnapshot?.sessionId ?? activeSessionId
let hostReady = readySnapshot?.ready == true && FlowSessionBridge.isHostReady()
let now = Date().timeIntervalSince1970
if hostReady { lastHostReadyAt = now }
// Grace window: the host was ready very recently, so treat a momentary
@@ -145,7 +151,7 @@ final class KeyboardFlowCoordinator {
let hostWarming = !hostReady
&& FlowSessionBridge.isSessionActive()
&& (FlowSessionBridge.isHostReachable() || isPendingFlowStart || withinReadyGrace)
state.flowSessionActive = hostReady
state.flowSessionActive = FlowSessionBridge.isSessionActive()
state.micVoiceAvailability = MicVoiceAvailabilityResolver.resolve(
phase: state.phase,
micDisabled: state.micDisabled,
@@ -154,6 +160,20 @@ final class KeyboardFlowCoordinator {
hostReady: hostReady,
isPreparingSession: isPendingFlowStart || hostWarming
)
let signature = [
"phase=\(String(describing: state.phase))",
"availability=\(String(describing: state.micVoiceAvailability))",
hostReady ? "hostReady=1" : "hostReady=0",
state.flowSessionActive ? "sessionActive=1" : "sessionActive=0",
isPendingFlowStart ? "pending=1" : "pending=0",
isFlowRecording ? "recording=1" : "recording=0",
isAwaitingFlowResult ? "awaiting=1" : "awaiting=0",
readySnapshot?.reason.rawValue ?? "snapshot=nil"
].joined(separator: "|")
if signature != lastAvailabilityTraceSignature {
lastAvailabilityTraceSignature = signature
traceState("availability.update", extra: signature)
}
}
/// Session is live but the ready contract has not landed yet poll
@@ -228,7 +248,8 @@ final class KeyboardFlowCoordinator {
scheduleAutoClearError()
recomputeMicVoiceAvailability()
case .unavailable(.preparingSession):
return
detectAndStoreAppContext()
beginFlowStart()
case .unavailable(.hostNotReady):
detectAndStoreAppContext()
beginFlowStart()
@@ -247,8 +268,8 @@ final class KeyboardFlowCoordinator {
isFlowRecording = false
stopUtteranceCountdown()
ExtensionScreenWakeLock.release()
FlowSessionBridge.setRecordingState(.stopped)
debug("pressEnded wrote .stopped (readback=\(FlowSessionBridge.recordingState().rawValue))")
writeCommand(.stopRecording)
debug("pressEnded wrote stop command")
state.phase = .processing
state.lastTranscript = ExtL10n.string("keyboard.flow.transcribing")
recomputeMicVoiceAvailability()
@@ -256,7 +277,10 @@ final class KeyboardFlowCoordinator {
}
func beginFlowStart() {
guard !isPendingFlowStart else { return }
guard !isPendingFlowStart else {
traceState("beginFlowStart.ignored", extra: "reason=pendingAlreadyTrue")
return
}
isPendingFlowStart = true
isFlowRecording = false
flowStartDeadline = Date().timeIntervalSince1970 + FlowWatchdog.startTimeout
@@ -264,11 +288,11 @@ final class KeyboardFlowCoordinator {
recomputeMicVoiceAvailability()
openHostApp("startflow")
startFlowStartWatchdog()
debug("beginFlowStart")
traceState("beginFlowStart.started")
}
func handleHostAppOpenResult(path: String, success: Bool) {
debug("openHostApp path=\(path) success=\(success)")
traceState("openHostApp.result", extra: "path=\(path) success=\(success)")
guard !success else { return }
// The open genuinely failed (iOS blocked it / no Full Access). Don't
@@ -278,6 +302,7 @@ final class KeyboardFlowCoordinator {
isPendingFlowStart = false
flowStartDeadline = 0
stopFlowWatchdog()
traceState("openHostApp.failed", extra: "path=startflow cancelPending=1")
showManualOpenHint(path: "startflow")
recomputeMicVoiceAvailability()
return
@@ -290,9 +315,10 @@ final class KeyboardFlowCoordinator {
guard !isAwaitingFlowResult else { return }
if isFlowRecording || isPendingFlowStart {
if isFlowRecording {
FlowSessionBridge.setRecordingState(.aborted)
writeCommand(.abort)
ExtensionScreenWakeLock.release()
}
currentUtteranceId = nil
isFlowRecording = false
isPendingFlowStart = false
stopUtteranceCountdown()
@@ -304,17 +330,56 @@ final class KeyboardFlowCoordinator {
// MARK: - Private
private func nextCommandSeq() -> Int64 {
let millis = Int64(Date().timeIntervalSince1970 * 1_000)
currentCommandSeq = max(currentCommandSeq + 1, millis)
return currentCommandSeq
}
private func writeCommand(_ action: FlowCommand.Action) {
guard let activeSessionId, let currentUtteranceId else { return }
let command = FlowCommand(
sessionId: activeSessionId,
utteranceId: currentUtteranceId,
commandSeq: nextCommandSeq(),
action: action,
localeId: state.localeId
)
FlowSessionBridge.writeCommand(command)
debug(
"command \(action.rawValue) seq=\(command.commandSeq) " +
"utterance=\(currentUtteranceId.uuidString)"
)
}
private func consumePendingFlowDeliveryIfNeeded() {
if isAwaitingFlowResult {
if let delivery = FlowSessionBridge.consumeTranscriptionDelivery() {
if let result = matchingResult(), result.status == .final, let text = result.text, !text.isEmpty {
isAwaitingFlowResult = false
stopFlowWatchdog()
textInserter.handleFlowTranscript(delivery)
FlowSessionBridge.writeAck(
FlowAck(
sessionId: result.sessionId,
utteranceId: result.utteranceId,
commandSeq: result.commandSeq
)
)
FlowSessionBridge.clearResult()
currentUtteranceId = nil
textInserter.handleFlowTranscript(
TranscriptionDelivery(text: text, polishWarning: result.warning)
)
return
}
if let error = FlowSessionBridge.consumeTranscriptionError() {
if let result = matchingResult(), isTerminalFailure(result) {
isAwaitingFlowResult = false
stopFlowWatchdog()
FlowSessionBridge.clearResult()
currentUtteranceId = nil
let error = FlowTranscriptionError(
message: result.text ?? ExtL10n.string("keyboard.flow.resultTimeout"),
kind: result.errorKind ?? .generic
)
state.phase = .error(
.fromFlowTranscription(error),
message: error.message
@@ -330,6 +395,20 @@ final class KeyboardFlowCoordinator {
}
}
private func matchingResult() -> FlowResult? {
guard let result = FlowSessionBridge.latestResult() else { return nil }
guard let activeSessionId, let currentUtteranceId else { return nil }
guard result.sessionId == activeSessionId,
result.utteranceId == currentUtteranceId else {
return nil
}
return result
}
private func isTerminalFailure(_ result: FlowResult) -> Bool {
result.status == .error || result.status == .timeout || result.status == .aborted
}
/// When the host process died mid-utterance, abort local recording / waiting
/// so the user is not stuck until the long result watchdog fires.
private func recoverFromDeadHostIfNeeded() {
@@ -339,7 +418,8 @@ final class KeyboardFlowCoordinator {
isFlowRecording = false
stopUtteranceCountdown()
ExtensionScreenWakeLock.release()
FlowSessionBridge.setRecordingState(.aborted)
writeCommand(.abort)
currentUtteranceId = nil
stopFlowWatchdog()
state.level = 0
state.phase = .idle
@@ -355,12 +435,14 @@ final class KeyboardFlowCoordinator {
}
private func failHostDisconnected() {
traceState("hostDisconnected.fail")
isAwaitingFlowResult = false
isFlowRecording = false
isPendingFlowStart = false
stopUtteranceCountdown()
ExtensionScreenWakeLock.release()
FlowSessionBridge.setRecordingState(.aborted)
writeCommand(.abort)
currentUtteranceId = nil
stopFlowWatchdog()
state.level = 0
let message = ExtL10n.string("keyboard.flow.hostDisconnected")
@@ -396,6 +478,7 @@ final class KeyboardFlowCoordinator {
private func startFlowRecording() {
recomputeMicVoiceAvailability()
guard state.micVoiceAvailability.isReady else {
traceState("startFlowRecording.blocked", extra: "availability=\(String(describing: state.micVoiceAvailability))")
beginFlowStart()
return
}
@@ -403,8 +486,14 @@ final class KeyboardFlowCoordinator {
flowStartDeadline = 0
stopFlowWatchdog()
FlowSessionBridge.setTranscriptionLanguage(state.localeId)
FlowSessionBridge.setRecordingState(.recording)
guard let sessionId = FlowSessionBridge.readySnapshot()?.sessionId else {
traceState("startFlowRecording.blocked", extra: "reason=missingSessionIdInReadySnapshot")
beginFlowStart()
return
}
activeSessionId = sessionId
currentUtteranceId = UUID()
writeCommand(.startRecording)
isFlowRecording = true
state.lastTranscript = ""
state.phase = .recording
@@ -414,9 +503,7 @@ final class KeyboardFlowCoordinator {
}
startUtteranceCountdown()
startFlowLevelWatchdog()
// Read back in-process to confirm the write landed before we rely on
// the host polling it out cross-process.
debug("startFlowRecording wrote .recording (readback=\(FlowSessionBridge.recordingState().rawValue))")
traceState("startFlowRecording.started")
}
private func startUtteranceCountdown() {
@@ -450,6 +537,7 @@ final class KeyboardFlowCoordinator {
state.phase = .idle
state.lastTranscript = ""
recomputeMicVoiceAvailability()
traceState("pendingStart.cancelledByUser")
}
private func startFlowStartWatchdog() {
@@ -465,6 +553,7 @@ final class KeyboardFlowCoordinator {
if self.flowStartDeadline > 0, now > self.flowStartDeadline {
self.isPendingFlowStart = false
self.flowStartDeadline = 0
self.traceState("startWatchdog.timeout")
self.showManualOpenHint(path: "startflow")
return
}
@@ -480,7 +569,7 @@ final class KeyboardFlowCoordinator {
state.lastTranscript = ""
refreshSessionState()
startFlowRecording()
debug("completeFlowStartHandoff → auto startFlowRecording")
traceState("completeFlowStartHandoff.done")
}
private func startFlowLevelWatchdog() {
@@ -492,6 +581,12 @@ final class KeyboardFlowCoordinator {
self.state.level = Double(peak)
}
self.refreshFlowPartialIfNeeded()
let staleness = FlowSessionBridge.heartbeatStaleness() ?? .infinity
if staleness > 5 {
self.debug("levelWatchdog: host heartbeat stale while recording")
self.failHostDisconnected()
return
}
try? await Task.sleep(nanoseconds: FlowWatchdog.pollIntervalNs)
}
}
@@ -501,7 +596,10 @@ final class KeyboardFlowCoordinator {
guard isFlowRecording || isAwaitingFlowResult else { return }
switch state.phase {
case .recording, .processing:
if let partial = FlowSessionBridge.transcriptionPartial() {
if let result = matchingResult(),
result.status == .partial,
let partial = result.text,
!partial.isEmpty {
state.lastTranscript = partial
}
default:
@@ -517,16 +615,33 @@ final class KeyboardFlowCoordinator {
debug("resultWatchdog started timeout=\(Int(resultTimeout))s engine=\(state.engineMode)")
flowWatchdogTask = Task { @MainActor [weak self] in
while let self, !Task.isCancelled {
if let delivery = FlowSessionBridge.consumeTranscriptionDelivery() {
if let result = self.matchingResult(), result.status == .final, let text = result.text, !text.isEmpty {
self.isAwaitingFlowResult = false
self.stopFlowWatchdog()
self.debug("resultWatchdog consumed delivery len=\(delivery.text.count)")
self.textInserter.handleFlowTranscript(delivery)
FlowSessionBridge.writeAck(
FlowAck(
sessionId: result.sessionId,
utteranceId: result.utteranceId,
commandSeq: result.commandSeq
)
)
FlowSessionBridge.clearResult()
self.currentUtteranceId = nil
self.debug("resultWatchdog consumed delivery len=\(text.count)")
self.textInserter.handleFlowTranscript(
TranscriptionDelivery(text: text, polishWarning: result.warning)
)
return
}
if let error = FlowSessionBridge.consumeTranscriptionError() {
if let result = self.matchingResult(), self.isTerminalFailure(result) {
self.isAwaitingFlowResult = false
self.stopFlowWatchdog()
FlowSessionBridge.clearResult()
self.currentUtteranceId = nil
let error = FlowTranscriptionError(
message: result.text ?? ExtL10n.string("keyboard.flow.resultTimeout"),
kind: result.errorKind ?? .generic
)
self.debug("resultWatchdog consumed error kind=\(error.kind.rawValue)")
self.state.phase = .error(
.fromFlowTranscription(error),
@@ -539,6 +654,11 @@ final class KeyboardFlowCoordinator {
self.refreshFlowPartialIfNeeded()
let now = Date().timeIntervalSince1970
let staleness = FlowSessionBridge.heartbeatStaleness() ?? .infinity
if self.isFlowRecording, staleness > 5 {
self.debug("level/result watchdog: host heartbeat stale while recording")
self.failHostDisconnected()
return
}
if staleness > FlowSessionKeys.heartbeatZombieInterval {
self.debug("resultWatchdog: host heartbeat zombie (staleness=\(String(format: "%.1f", staleness))s)")
self.failHostDisconnected()
@@ -553,6 +673,7 @@ final class KeyboardFlowCoordinator {
if now - startedAt > resultTimeout {
self.isAwaitingFlowResult = false
self.stopFlowWatchdog()
self.currentUtteranceId = nil
self.debug("resultWatchdog TIMEOUT after \(Int(resultTimeout))s — no result from host")
let msg = ExtL10n.string("keyboard.flow.resultTimeout")
self.state.phase = .error(.flowResultTimeout, message: msg)
@@ -573,4 +694,29 @@ final class KeyboardFlowCoordinator {
private func debug(_ message: String) {
OSGLog.keyboardExt.info("\(message, privacy: .public)")
}
private func traceState(_ event: String, extra: String? = nil) {
let staleness = FlowSessionBridge.heartbeatStaleness().map { String(format: "%.1f", $0) } ?? "nil"
let sessionId = activeSessionId?.uuidString ?? "nil"
let utteranceId = currentUtteranceId?.uuidString ?? "nil"
let summary = [
"event=\(event)",
"phase=\(String(describing: state.phase))",
"availability=\(String(describing: state.micVoiceAvailability))",
"pending=\(isPendingFlowStart)",
"recording=\(isFlowRecording)",
"awaiting=\(isAwaitingFlowResult)",
"sessionId=\(sessionId)",
"utteranceId=\(utteranceId)",
"cmdSeq=\(currentCommandSeq)",
"sessionActive=\(FlowSessionBridge.isSessionActive())",
"hostReady=\(FlowSessionBridge.isHostReady())",
"heartbeatStaleness=\(staleness)"
].joined(separator: " ")
if let extra, !extra.isEmpty {
debug("[trace] \(summary) \(extra)")
} else {
debug("[trace] \(summary)")
}
}
}
@@ -52,7 +52,7 @@ private struct FlowLiveActivityLockScreenView: View {
var body: some View {
HStack(spacing: 12) {
FlowLiveActivityBrandMark(height: 18)
FlowLiveActivityBrandMark(height: 13)
VStack(alignment: .leading, spacing: 4) {
Text("OSGKeyboard")
.font(.headline)
@@ -99,9 +99,10 @@ private struct FlowLiveActivityTrailingGlyph: View {
.foregroundStyle(.red)
.symbolEffect(.variableColor.iterative, options: .repeating)
case .processing:
ProgressView()
.progressViewStyle(.circular)
.tint(.white)
Image(systemName: "ellipsis")
.font(.title3.weight(.semibold))
.foregroundStyle(.white)
.symbolEffect(.variableColor.iterative, options: .repeating)
case .idle:
// Session ready but NOT listening avoid a mic glyph so users
// don't think the keyboard is recording in the background.
+20 -9
View File
@@ -183,11 +183,23 @@ struct BottomDictationBar: View {
.fixedSize()
}
//
//
// Option
private var recordControl: some View {
HStack(spacing: Spacing.sm) {
recordButton
.overlay(alignment: .top) {
if viewModel.isRecording {
MiniWaveform(level: viewModel.audioLevel)
Text(MacL10n.string("mac.record.pressStop", language: lang))
.font(TypeStyle.caption)
.foregroundStyle(palette.textTertiary)
.fixedSize()
.offset(y: -22)
}
}
}
private var recordButton: some View {
Button(action: viewModel.toggleRecording) {
ZStack {
Circle()
@@ -198,11 +210,16 @@ struct BottomDictationBar: View {
color: (viewModel.isRecording ? palette.recordRed : palette.accent).opacity(0.5),
radius: pulse ? 14 : 6
)
Image(systemName: viewModel.isRecording ? "stop.fill" : "mic.fill")
if viewModel.isRecording {
// iOS
MiniWaveform(level: viewModel.audioLevel, barCount: 4, tint: palette.textOnAccent)
} else {
Image(systemName: "mic.fill")
.font(.system(size: 20, weight: .bold))
.foregroundStyle(palette.textOnAccent)
}
}
}
.buttonStyle(.plain)
.disabled(viewModel.isProcessing)
.onAppear {
@@ -210,12 +227,6 @@ struct BottomDictationBar: View {
pulse = true
}
}
if viewModel.isRecording {
Text(MacL10n.string("mac.record.pressStop", language: lang))
.font(TypeStyle.caption)
.foregroundStyle(palette.textTertiary)
}
}
}
private var currentTranslationLabel: String {
+45 -4
View File
@@ -31,9 +31,30 @@ enum MacDictationPipeline {
let locale = Locale(identifier: store.localeId.isEmpty ? "zh-CN" : store.localeId)
let raw: String
var localBias: LocalASRBiasPayload?
if store.engineMode == "local" {
raw = try await MacLocalASRService.transcribe(samples: samples, locale: locale)
MacAppContextService.captureAndPersist(to: store)
let capabilities = MacLocalASRService.currentCapabilities()
let bias = LocalASRBiasAdapter.adapt(
LocalASRBiasRequest(
dictionary: store.personalDictionary,
locale: locale,
frontAppBundleId: MacAppContextService.frontmostBundleIdentifier(),
capabilities: capabilities
)
)
localBias = bias
LocalASRBiasDiagnosticsStore.save(
payload: bias,
modelId: MacLocalASRService.selectedModelDefinition()?.id,
backendLabel: MacLocalASRService.currentBackendLabel()
)
raw = try await MacLocalASRService.transcribe(
samples: samples,
locale: locale,
bias: bias
)
} else {
let strategy = CloudASRModelCatalog.strategy(for: store.providerId)
guard strategy != .localFallback else { throw MacDictationError.providerHasNoCloudASR }
@@ -51,13 +72,33 @@ enum MacDictationPipeline {
let trimmed = raw.trimmingCharacters(in: .whitespacesAndNewlines)
guard !trimmed.isEmpty else { throw MacDictationError.emptyTranscript }
let postASR: String
if let localBias, !localBias.correctionPairs.isEmpty {
postASR = LocalASRTranscriptCorrector.apply(trimmed, pairs: localBias.correctionPairs)
} else {
postASR = trimmed
}
let polishContext: PolishContext?
if let supplement = localBias?.polishFragment.trimmingCharacters(in: .whitespacesAndNewlines),
!supplement.isEmpty {
polishContext = PolishContext(
appContext: store.detectedAppContext?.context ?? .unknown,
intensity: store.polishIntensity,
dictionarySupplement: supplement
)
} else {
polishContext = nil
}
if let polished = try? await PolishingService(store: store).polish(
trimmed,
mode: store.polishModeForPipeline
postASR,
mode: store.polishModeForPipeline,
context: polishContext
),
!polished.isEmpty {
return polished
}
return trimmed
return postASR
}
}
+36 -2
View File
@@ -107,8 +107,9 @@ final class MacDictationViewModel: ObservableObject {
/// Pre-load MLX weights + Metal shaders so the first dictation is fast.
func warmUpQwen3IfNeeded() {
guard config.engineMode == "local",
MacLocalASRPreferences.backend == .qwen3MLX,
MacLocalASRPreferences.qwen3ModelIsInstalled() else { return }
let model = MacLocalASRService.selectedModelDefinition(),
model.backend == .mlx,
MacLocalASRService.isModelInstalled(model) else { return }
let path = MacLocalASRPreferences.qwen3ModelPath
Task.detached(priority: .utility) {
_ = try? await MacQwen3ASREngine.shared.prepareIfNeeded(modelPath: path)
@@ -156,6 +157,39 @@ final class MacDictationViewModel: ObservableObject {
return "\(seconds)s"
}
var localModelReady: Bool {
_ = localModelRevision
if let model = MacLocalASRService.selectedModelDefinition() {
return MacLocalASRService.isModelInstalled(model)
}
return MacLocalASRPreferences.qwen3ModelIsInstalled()
}
/// Context-aware warning when local engine is selected but the active model is not ready.
var localModelWarningMessage: String? {
_ = localModelRevision
guard config.engineMode == "local" else { return nil }
if localModelReady { return nil }
guard let model = MacLocalASRService.selectedModelDefinition() else {
return MacL10n.string("mac.settings.localModelFallbackApple", language: config.uiLanguage)
}
if model.installKind == .manual {
return MacL10n.string("mac.settings.mlxModelMissing", language: config.uiLanguage)
}
return MacL10n.format(
"mac.settings.selectedModelMissing",
language: config.uiLanguage,
model.displayName
)
}
@Published private(set) var localModelRevision = 0
func bumpLocalModelRevision() {
localModelRevision += 1
objectWillChange.send()
}
var qwen3ModelInstalled: Bool {
MacLocalASRPreferences.qwen3ModelIsInstalled()
}
@@ -0,0 +1,455 @@
// MacLocalASRModelSettingsView.swift
// OSGKeyboard · Mac
//
// Local ASR model catalog, download progress, MLX path, and bias diagnostics.
import AppKit
import SwiftUI
@MainActor
final class MacLocalASRModelSettingsViewModel: ObservableObject {
@Published var catalog: LocalASRCatalogDocument?
@Published var selectedModelId: String = MacLocalASRPreferences.selectedModelId
@Published var installProgress = LocalASRModelInstallProgress.idle
@Published var diagnosticsSnapshot = LocalASRBiasDiagnosticsStore.load()
@Published var statusMessage = ""
@Published var isInstalling = false
@Published var isDownloadPaused = false
var onLocalModelStateChanged: (() -> Void)?
private let manager = LocalASRModelManager.shared
private var progressPollTask: Task<Void, Never>?
deinit {
progressPollTask?.cancel()
}
func reload() {
catalog = try? LocalASRModelCatalog.loadBundled()
if let catalog {
let manifest = LocalASRInstalledManifestIO.load(defaultModelId: catalog.defaultModelId)
selectedModelId = manifest.selectedModelId.isEmpty
? MacLocalASRPreferences.selectedModelId
: manifest.selectedModelId
}
diagnosticsSnapshot = LocalASRBiasDiagnosticsStore.load()
onLocalModelStateChanged?()
}
func isInstalled(_ model: LocalASRModelDefinition) -> Bool {
MacLocalASRService.isModelInstalled(model)
}
func isInstallingModel(_ model: LocalASRModelDefinition) -> Bool {
isInstalling && installProgress.activeItemId == model.id
}
func installedDiskUsage(_ model: LocalASRModelDefinition) -> String? {
guard model.installKind == .archive,
let relative = model.installRelativePath,
isInstalled(model) else { return nil }
let dir = LocalASRModelInstallState.installDirectory(for: relative)
let bytes = LocalASRModelInstallState.directoryByteCount(at: dir)
guard bytes > 0 else { return nil }
return ByteCountFormatter.string(fromByteCount: bytes, countStyle: .file)
}
func currentRuntime(in catalog: LocalASRCatalogDocument) -> LocalASRRuntimeDefinition? {
LocalASRModelCatalog.runtime(for: LocalASRModelCatalog.currentRuntimePlatform(), in: catalog)
}
func isRuntimeInstalled(_ runtime: LocalASRRuntimeDefinition) -> Bool {
LocalASRModelInstallState.isRuntimeInstalled(runtime)
}
func selectModel(_ modelId: String) {
guard let catalog, !isInstalling else { return }
selectedModelId = modelId
MacLocalASRPreferences.selectedModelId = modelId
var manifest = LocalASRInstalledManifestIO.load(defaultModelId: catalog.defaultModelId)
manifest.selectedModelId = modelId
manifest.updatedAt = Date()
try? LocalASRInstalledManifestIO.save(manifest)
onLocalModelStateChanged?()
}
func installModel(_ model: LocalASRModelDefinition) {
guard let catalog, !isInstalling else { return }
statusMessage = ""
isInstalling = true
isDownloadPaused = false
startProgressPolling()
Task {
do {
try await manager.installModel(model, catalog: catalog)
installProgress = await manager.currentProgress()
selectModel(model.id)
statusMessage = MacL10n.string("mac.localASR.installDone")
} catch {
installProgress = await manager.currentProgress()
statusMessage = error.localizedDescription
}
isInstalling = false
isDownloadPaused = false
stopProgressPolling()
reload()
}
}
func pauseDownload() {
Task {
do {
try await manager.pauseDownload()
isDownloadPaused = true
installProgress = await manager.currentProgress()
} catch {
statusMessage = error.localizedDescription
}
}
}
func resumeDownload() {
Task {
do {
try await manager.resumeDownload()
isDownloadPaused = false
installProgress = await manager.currentProgress()
} catch {
statusMessage = error.localizedDescription
}
}
}
func deleteModel(_ model: LocalASRModelDefinition) {
guard let catalog, !isInstalling else { return }
Task {
do {
try await manager.deleteModel(model, catalog: catalog)
statusMessage = MacL10n.string("mac.localASR.deleteDone")
reload()
} catch {
statusMessage = error.localizedDescription
}
}
}
func revealModelInFinder(_ model: LocalASRModelDefinition) {
guard let relative = model.installRelativePath else { return }
let url = LocalASRModelInstallState.installDirectory(for: relative)
NSWorkspace.shared.activateFileViewerSelecting([url])
}
/// Opens (creating if needed) the model's shared subfolder so the user can
/// drop in manually-converted weights (used by the MLX model).
func revealModelFolder(_ model: LocalASRModelDefinition) {
guard let relative = model.installRelativePath else { return }
let url = LocalASRModelInstallState.installDirectory(for: relative)
try? FileManager.default.createDirectory(at: url, withIntermediateDirectories: true)
NSWorkspace.shared.open(url)
}
func revealStorageRoot() {
let url = LocalASRModelInstallState.rootDirectory()
try? FileManager.default.createDirectory(at: url, withIntermediateDirectories: true)
NSWorkspace.shared.open(url)
}
func progressLabel(for progress: LocalASRModelInstallProgress, language: AppUILanguage) -> String {
let phaseKey: String
switch progress.phase {
case .downloading: phaseKey = "mac.localASR.phase.downloading"
case .paused: phaseKey = "mac.localASR.phase.paused"
case .extracting: phaseKey = "mac.localASR.phase.extracting"
case .validating: phaseKey = "mac.localASR.phase.validating"
case .finalizing: phaseKey = "mac.localASR.phase.finalizing"
case .failed: phaseKey = "mac.localASR.phase.failed"
case .completed: phaseKey = "mac.localASR.phase.completed"
case .idle: return progress.message
}
let phase = MacL10n.string(phaseKey, language: language)
if let received = progress.bytesReceived, let total = progress.bytesTotal, total > 0 {
let recv = ByteCountFormatter.string(fromByteCount: received, countStyle: .file)
let tot = ByteCountFormatter.string(fromByteCount: total, countStyle: .file)
return "\(phase) · \(progress.message) (\(recv) / \(tot))"
}
return "\(phase) · \(progress.message)"
}
private func startProgressPolling() {
progressPollTask?.cancel()
progressPollTask = Task { [weak self] in
while !Task.isCancelled {
guard let self else { return }
let current = await manager.currentProgress()
await MainActor.run {
self.installProgress = current
self.isDownloadPaused = current.phase == .paused
}
try? await Task.sleep(for: .milliseconds(120))
}
}
}
private func stopProgressPolling() {
progressPollTask?.cancel()
progressPollTask = nil
}
func formattedSize(_ bytes: Int) -> String {
ByteCountFormatter.string(fromByteCount: Int64(bytes), countStyle: .file)
}
}
struct MacLocalASRModelSettingsView: View {
@ObservedObject var viewModel: MacDictationViewModel
@StateObject private var modelVM = MacLocalASRModelSettingsViewModel()
@Environment(\.themePalette) private var palette
private var lang: AppUILanguage { viewModel.config.uiLanguage }
var body: some View {
Group {
if let catalog = modelVM.catalog {
modelPickerSection(catalog: catalog)
runtimeSection(catalog: catalog)
} else {
Text(MacL10n.string("mac.localASR.catalogMissing", language: lang))
.foregroundStyle(palette.textSecondary)
}
}
.onAppear {
modelVM.onLocalModelStateChanged = { viewModel.bumpLocalModelRevision() }
modelVM.reload()
}
}
private func modelPickerSection(catalog: LocalASRCatalogDocument) -> some View {
Section {
ForEach(catalog.models) { model in
modelRow(model)
}
if modelVM.isInstalling,
modelVM.installProgress.phase == .extracting
|| modelVM.installProgress.phase == .validating
|| modelVM.installProgress.phase == .finalizing {
ProgressView(value: modelVM.installProgress.fraction) {
Text(modelVM.progressLabel(for: modelVM.installProgress, language: lang))
.font(TypeStyle.caption)
}
}
if !modelVM.statusMessage.isEmpty {
Text(modelVM.statusMessage)
.font(TypeStyle.caption)
.foregroundStyle(palette.textSecondary)
}
Button(MacL10n.string("mac.localASR.openStorage", language: lang)) {
modelVM.revealStorageRoot()
}
} header: {
Text(MacL10n.string("mac.localASR.models", language: lang))
} footer: {
Text(MacL10n.string("mac.localASR.modelsDesc", language: lang))
.font(TypeStyle.caption)
.foregroundStyle(palette.textSecondary)
}
}
private func runtimeSection(catalog: LocalASRCatalogDocument) -> some View {
Group {
if let runtime = modelVM.currentRuntime(in: catalog) {
Section {
LabeledContent(runtime.displayName) {
Text(
modelVM.isRuntimeInstalled(runtime)
? MacL10n.string("mac.localASR.installed", language: lang)
: MacL10n.string("mac.localASR.notInstalled", language: lang)
)
}
Text(MacL10n.string("mac.localASR.runtimeDesc", language: lang))
.font(TypeStyle.caption)
.foregroundStyle(palette.textSecondary)
} header: {
Text(MacL10n.string("mac.localASR.runtime", language: lang))
}
}
}
}
@ViewBuilder
private func modelRow(_ model: LocalASRModelDefinition) -> some View {
let installed = modelVM.isInstalled(model)
let selected = modelVM.selectedModelId == model.id
let installing = modelVM.isInstallingModel(model)
VStack(alignment: .leading, spacing: Spacing.xs) {
HStack(alignment: .top) {
Button {
modelVM.selectModel(model.id)
} label: {
HStack(spacing: Spacing.sm) {
Image(systemName: selected ? "largecircle.fill.circle" : "circle")
.foregroundStyle(selected ? palette.accent : palette.textTertiary)
VStack(alignment: .leading, spacing: 2) {
Text(model.displayName)
.foregroundStyle(palette.textPrimary)
Text(modelSubtitle(model, installed: installed))
.font(TypeStyle.caption)
.foregroundStyle(palette.textSecondary)
}
}
}
.buttonStyle(.plain)
.disabled(modelVM.isInstalling)
Spacer()
modelRowActions(model: model, installed: installed, installing: installing)
}
}
.padding(.vertical, 2)
}
@ViewBuilder
private func modelRowActions(
model: LocalASRModelDefinition,
installed: Bool,
installing: Bool
) -> some View {
if installing {
HStack(spacing: Spacing.sm) {
circularInstallProgress(for: model)
if modelVM.installProgress.phase == .downloading
|| modelVM.installProgress.phase == .paused {
Button {
if modelVM.isDownloadPaused {
modelVM.resumeDownload()
} else {
modelVM.pauseDownload()
}
} label: {
Image(systemName: modelVM.isDownloadPaused ? "play.fill" : "pause.fill")
.font(.system(size: 12, weight: .semibold))
.frame(width: 28, height: 28)
}
.buttonStyle(.bordered)
.controlSize(.small)
.help(
modelVM.isDownloadPaused
? MacL10n.string("mac.localASR.resume", language: lang)
: MacL10n.string("mac.localASR.pause", language: lang)
)
}
}
} else if model.installKind == .manual {
Button(MacL10n.string("mac.localASR.openFolder", language: lang)) {
modelVM.revealModelFolder(model)
}
.buttonStyle(.bordered)
.controlSize(.small)
} else if installed {
Button(MacL10n.string("mac.localASR.delete", language: lang), role: .destructive) {
modelVM.deleteModel(model)
}
.buttonStyle(.bordered)
.controlSize(.small)
} else {
Button(MacL10n.string("mac.localASR.download", language: lang)) {
modelVM.installModel(model)
}
.buttonStyle(.borderedProminent)
.controlSize(.small)
}
}
private func circularInstallProgress(for model: LocalASRModelDefinition) -> some View {
let fraction: Double = {
if modelVM.installProgress.phase == .downloading || modelVM.installProgress.phase == .paused,
let received = modelVM.installProgress.bytesReceived,
let total = modelVM.installProgress.bytesTotal,
total > 0 {
return min(1, max(0, Double(received) / Double(total)))
}
return modelVM.installProgress.fraction
}()
return ZStack {
Circle()
.stroke(palette.textTertiary.opacity(0.25), lineWidth: 3)
Circle()
.trim(from: 0, to: fraction)
.stroke(palette.accent, style: StrokeStyle(lineWidth: 3, lineCap: .round))
.rotationEffect(.degrees(-90))
.animation(.linear(duration: 0.15), value: fraction)
if modelVM.installProgress.phase == .paused {
Image(systemName: "pause.fill")
.font(.system(size: 10, weight: .bold))
.foregroundStyle(palette.textSecondary)
} else {
Text("\(Int(fraction * 100))%")
.font(.system(size: 9, weight: .medium, design: .rounded))
.foregroundStyle(palette.textSecondary)
}
}
.frame(width: 36, height: 36)
.accessibilityLabel(modelVM.progressLabel(for: modelVM.installProgress, language: lang))
}
private func modelSubtitle(_ model: LocalASRModelDefinition, installed: Bool) -> String {
let size = modelVM.formattedSize(model.sizeBytes)
let hotword = model.supportsHotwords
? MacL10n.string("mac.localASR.hotwordsYes", language: lang)
: MacL10n.string("mac.localASR.hotwordsNo", language: lang)
let state = installed
? MacL10n.string("mac.localASR.installed", language: lang)
: MacL10n.string("mac.localASR.notInstalled", language: lang)
if let usage = modelVM.installedDiskUsage(model) {
return "\(size) · \(hotword) · \(state) · \(usage)"
}
return "\(size) · \(hotword) · \(state)"
}
private var diagnosticsSection: some View {
Section {
if let snapshot = modelVM.diagnosticsSnapshot {
LabeledContent(MacL10n.string("mac.localASR.diagBackend", language: lang)) {
Text(snapshot.backendLabel ?? "")
}
LabeledContent(MacL10n.string("mac.localASR.diagUserTerms", language: lang)) {
Text("\(snapshot.diagnostics.userTermCount)")
}
LabeledContent(MacL10n.string("mac.localASR.diagBuiltinTerms", language: lang)) {
Text("\(snapshot.diagnostics.builtinTermCount)")
}
LabeledContent(MacL10n.string("mac.localASR.diagHotwords", language: lang)) {
Text("\(snapshot.hotwordCount)")
}
LabeledContent(MacL10n.string("mac.localASR.diagPrompt", language: lang)) {
Text("\(snapshot.promptBiasLength)")
}
if snapshot.diagnostics.truncated {
Label(
snapshot.diagnostics.truncationReason ?? MacL10n.string("mac.localASR.diagTruncated", language: lang),
systemImage: "exclamationmark.triangle"
)
.font(TypeStyle.caption)
.foregroundStyle(palette.warning)
}
Text(snapshot.diagnostics.selectedSources.joined(separator: ", "))
.font(TypeStyle.caption)
.foregroundStyle(palette.textSecondary)
} else {
Text(MacL10n.string("mac.localASR.diagEmpty", language: lang))
.foregroundStyle(palette.textSecondary)
}
} header: {
Text(MacL10n.string("mac.localASR.diagnostics", language: lang))
} footer: {
Text(MacL10n.string("mac.localASR.diagnosticsDesc", language: lang))
.font(TypeStyle.caption)
.foregroundStyle(palette.textSecondary)
}
}
}
+100 -24
View File
@@ -1,13 +1,15 @@
// MacLocalASRService.swift
// OSGKeyboard · Mac
//
// On-device ASR for macOS. Primary: Qwen3-ASR-1.7B (MLX via mlx-swift-asr).
// Falls back to Apple Speech when Qwen3 weights are absent or backend is Apple Speech.
// On-device ASR for macOS. Routes through the bundled local ASR catalog:
// Qwen3 MLX (default), Sherpa Qwen3 hotwords POC, SenseVoice, Apple Speech fallback.
import Foundation
enum MacLocalASRBackend: String, Sendable, CaseIterable {
case qwen3MLX
case sherpaQwen3
case sherpaSenseVoice
case appleSpeech
}
@@ -39,28 +41,33 @@ enum MacLocalASRError: Error, LocalizedError {
enum MacLocalASRPreferences {
static let backendKey = "mac.localASR.backend"
static let qwen3ModelPathKey = "mac.localASR.qwen3ModelPath"
static let selectedModelIdKey = LocalASRPreferenceKeys.selectedModelId
/// Shared managed subfolder for the manually-provided MLX weights.
static let qwen3ModelRelativePath = "models/qwen3-asr-1.7b-mlx"
static var backend: MacLocalASRBackend {
static var selectedModelId: String {
get {
if let raw = UserDefaults.standard.string(forKey: selectedModelIdKey), !raw.isEmpty {
return raw
}
return legacyBackend == .appleSpeech ? "apple-speech-fallback" : "qwen3-mlx-1.7b"
}
set { UserDefaults.standard.set(newValue, forKey: selectedModelIdKey) }
}
static var legacyBackend: MacLocalASRBackend {
guard let raw = UserDefaults.standard.string(forKey: backendKey),
let value = MacLocalASRBackend(rawValue: raw) else {
return .qwen3MLX
}
return value
}
set { UserDefaults.standard.set(newValue.rawValue, forKey: backendKey) }
}
/// Fixed location inside the shared managed model storage root. All three
/// catalog models live under the same directory, so MLX no longer needs a
/// per-model folder picker the user drops converted weights here.
static var qwen3ModelPath: String {
get { UserDefaults.standard.string(forKey: qwen3ModelPathKey) ?? defaultQwen3ModelPath }
set { UserDefaults.standard.set(newValue, forKey: qwen3ModelPathKey) }
}
/// Default install location for MLX-converted Qwen3-ASR weights.
static var defaultQwen3ModelPath: String {
let appSupport = FileManager.default.urls(for: .applicationSupportDirectory, in: .userDomainMask).first!
return appSupport.appendingPathComponent("OSGKeyboard/models/qwen3-asr-1.7b-mlx", isDirectory: true).path
LocalASRModelInstallState.installDirectory(for: qwen3ModelRelativePath).path
}
static func qwen3ModelIsInstalled(at path: String = qwen3ModelPath) -> Bool {
@@ -80,24 +87,93 @@ enum MacLocalASRPreferences {
}
enum MacLocalASRService {
/// Transcribe using the user's preferred local backend with automatic
/// fallback to Apple Speech when Qwen3 weights are not present.
static func transcribe(samples: [Float], locale: Locale) async throws -> String {
let preferQwen3 = MacLocalASRPreferences.backend == .qwen3MLX
if preferQwen3, MacLocalASRPreferences.qwen3ModelIsInstalled() {
static func loadCatalog() -> LocalASRCatalogDocument? {
try? LocalASRModelCatalog.loadBundled()
}
static func selectedModelDefinition() -> LocalASRModelDefinition? {
guard let catalog = loadCatalog() else { return nil }
let manifest = LocalASRInstalledManifestIO.load(defaultModelId: catalog.defaultModelId)
let selectedId = manifest.selectedModelId.isEmpty
? MacLocalASRPreferences.selectedModelId
: manifest.selectedModelId
if selectedId == "apple-speech-fallback" { return nil }
return LocalASRModelCatalog.model(selectedId, in: catalog)
?? LocalASRModelCatalog.model(catalog.defaultModelId, in: catalog)
}
static func currentCapabilities() -> LocalASRCapabilities {
guard let model = selectedModelDefinition() else { return .appleSpeech }
return LocalASRModelCatalog.capabilities(for: model)
}
static func currentBackendLabel() -> String {
guard let model = selectedModelDefinition() else { return "Apple Speech" }
return model.displayName
}
static func isModelInstalled(_ model: LocalASRModelDefinition) -> Bool {
LocalASRModelInstallState.isInstalled(
model,
manualMLXPath: MacLocalASRPreferences.qwen3ModelPath
)
}
/// Transcribe using the selected catalog model, with MLX Apple Speech fallback.
static func transcribe(
samples: [Float],
locale: Locale,
bias: LocalASRBiasPayload? = nil
) async throws -> String {
if let model = selectedModelDefinition(), isModelInstalled(model) {
do {
return try await transcribeWithModel(model, samples: samples, locale: locale, bias: bias)
} catch {
if model.backend != .mlx {
throw error
}
}
}
if MacLocalASRPreferences.qwen3ModelIsInstalled() {
return try await MacQwen3LocalASR.transcribe(
samples: samples,
sampleRate: 16_000,
locale: locale,
modelPath: MacLocalASRPreferences.qwen3ModelPath
modelPath: MacLocalASRPreferences.qwen3ModelPath,
bias: bias
)
} catch MacLocalASRError.qwen3ModelMissing {
// Fall through to Apple Speech when weights are absent.
} catch {
throw error
}
return try await MacSpeechLocalASR.transcribe(samples: samples, locale: locale)
}
private static func transcribeWithModel(
_ model: LocalASRModelDefinition,
samples: [Float],
locale: Locale,
bias: LocalASRBiasPayload?
) async throws -> String {
switch model.backend {
case .mlx:
return try await MacQwen3LocalASR.transcribe(
samples: samples,
sampleRate: 16_000,
locale: locale,
modelPath: MacLocalASRPreferences.qwen3ModelPath,
bias: bias
)
case .sherpaQwen3, .sherpaSenseVoice:
return try await MacSherpaLocalASR.transcribe(
samples: samples,
sampleRate: 16_000,
locale: locale,
model: model,
bias: bias
)
case .appleSpeech:
return try await MacSpeechLocalASR.transcribe(samples: samples, locale: locale)
}
}
}
+3 -2
View File
@@ -50,14 +50,15 @@ actor MacQwen3ASREngine {
func transcribe(
samples: [Float],
language: String?,
modelPath: String
modelPath: String,
context: String? = nil
) async throws -> String {
try await prepareIfNeeded(modelPath: modelPath)
guard let stt else {
throw MacLocalASRError.qwen3LoadFailed("Engine not initialized")
}
let result = try await stt.transcribe(audio: samples, language: language)
let result = try await stt.transcribe(audio: samples, language: language, context: context)
let text = result.text.trimmingCharacters(in: .whitespacesAndNewlines)
guard !text.isEmpty else {
throw MacLocalASRError.emptyTranscript
+6 -2
View File
@@ -12,7 +12,8 @@ enum MacQwen3LocalASR {
samples: [Float],
sampleRate: Int,
locale: Locale,
modelPath: String
modelPath: String,
bias: LocalASRBiasPayload? = nil
) async throws -> String {
guard MacLocalASRPreferences.qwen3ModelIsInstalled(at: modelPath) else {
throw MacLocalASRError.qwen3ModelMissing
@@ -24,11 +25,14 @@ enum MacQwen3LocalASR {
}
let language = MacQwen3LanguageHint.from(locale: locale)
let context = bias?.promptBias?.trimmingCharacters(in: .whitespacesAndNewlines)
let promptContext = (context?.isEmpty == false) ? context : nil
do {
return try await MacQwen3ASREngine.shared.transcribe(
samples: samples,
language: language,
modelPath: modelPath
modelPath: modelPath,
context: promptContext
)
} catch let error as MacLocalASRError {
throw error
-1
View File
@@ -56,7 +56,6 @@ struct MacRootView: View {
Spacer()
devicesFooter
}
.background(palette.surfaceMuted)
}
private func sidebarRow(_ section: MacSection) -> some View {
+2 -52
View File
@@ -37,7 +37,7 @@ struct MacSettingsView: View {
providerSection
}
if viewModel.config.engineMode == "local" {
qwen3Section
MacLocalASRModelSettingsView(viewModel: viewModel)
}
inputSection
syncSection
@@ -147,12 +147,6 @@ struct MacSettingsView: View {
systemImage: "cpu",
selected: viewModel.config.engineMode == "local"
) { viewModel.setEngineMode("local") }
if viewModel.config.engineMode == "local", !viewModel.qwen3ModelInstalled {
Label(MacL10n.string("mac.settings.qwen3Missing", language: lang), systemImage: "exclamationmark.triangle")
.font(TypeStyle.caption)
.foregroundStyle(palette.warning)
}
}
}
@@ -196,25 +190,7 @@ struct MacSettingsView: View {
}
}
// MARK: - Qwen3 model path
private var qwen3Section: some View {
Section {
HStack(spacing: Spacing.sm) {
TextField("", text: qwen3PathBinding, prompt: Text(verbatim: "~/Models/Qwen3-ASR"))
.macFieldStyle()
Button(MacL10n.string("mac.settings.qwen3Browse", language: lang)) {
pickQwen3Folder()
}
}
} header: {
Text(MacL10n.string("mac.settings.qwen3Model", language: lang))
} footer: {
Text(MacL10n.string("mac.settings.qwen3ModelDesc", language: lang))
.font(TypeStyle.caption)
.foregroundStyle(palette.textSecondary)
}
}
// MARK: - Qwen3 model path (legacy see MacLocalASRModelSettingsView)
// MARK: - Row helpers
@@ -366,17 +342,6 @@ struct MacSettingsView: View {
)
}
private var qwen3PathBinding: Binding<String> {
Binding(
get: { MacLocalASRPreferences.qwen3ModelPath },
set: { newPath in
MacLocalASRPreferences.qwen3ModelPath = newPath
Task { await MacQwen3ASREngine.shared.unload() }
viewModel.warmUpQwen3IfNeeded()
}
)
}
// MARK: - AppKit actions (macOS only)
private func openAccessibilitySettings() {
@@ -413,19 +378,4 @@ struct MacSettingsView: View {
private var accessibilityStatusNeeded: String {
lang.resolvedLanguageCode().hasPrefix("zh") ? "未授权" : "Needed"
}
private func pickQwen3Folder() {
#if os(macOS)
let panel = NSOpenPanel()
panel.canChooseDirectories = true
panel.canChooseFiles = false
panel.allowsMultipleSelection = false
panel.begin { response in
guard response == .OK, let url = panel.url else { return }
MacLocalASRPreferences.qwen3ModelPath = url.path
Task { await MacQwen3ASREngine.shared.unload() }
viewModel.warmUpQwen3IfNeeded()
}
#endif
}
}
+56
View File
@@ -0,0 +1,56 @@
// MacSherpaLocalASR.swift
// OSGKeyboard · Mac
//
// Sherpa-onnx backed local ASR (Qwen3 hotwords POC + SenseVoice baseline).
import Foundation
enum MacSherpaLocalASR {
static func transcribe(
samples: [Float],
sampleRate: Int,
locale: Locale,
model: LocalASRModelDefinition,
bias: LocalASRBiasPayload?
) async throws -> String {
let catalog = try LocalASRModelCatalog.loadBundled()
let manager = LocalASRModelManager.shared
guard let layout = model.layout,
let modelRoot = LocalASRModelInstallState.modelRootURL(model) else {
throw MacLocalASRError.qwen3ModelMissing
}
try await manager.ensureRuntimeInstalled(catalog: catalog)
guard let runtime = LocalASRModelCatalog.runtime(
for: LocalASRModelCatalog.currentRuntimePlatform(),
in: catalog
),
let binary = LocalASRModelInstallState.resolveRuntimeBinary(runtime: runtime) else {
throw MacLocalASRError.qwen3LoadFailed("Sherpa runtime binary missing")
}
switch model.backend {
case .sherpaQwen3:
return try await MacSherpaONNXRunner.transcribeQwen3(
samples: samples,
sampleRate: sampleRate,
locale: locale,
modelRoot: modelRoot,
layout: layout,
runtimeBinary: binary,
bias: bias
)
case .sherpaSenseVoice:
return try await MacSherpaONNXRunner.transcribeSenseVoice(
samples: samples,
sampleRate: sampleRate,
modelRoot: modelRoot,
layout: layout,
runtimeBinary: binary
)
default:
throw MacLocalASRError.qwen3InferenceFailed("Unsupported Sherpa backend")
}
}
}
+153
View File
@@ -0,0 +1,153 @@
// MacSherpaONNXRunner.swift
// OSGKeyboard · Mac
//
// Invokes the downloaded `sherpa-onnx-offline` binary for Sherpa-backed POC models.
import Foundation
enum MacSherpaONNXRunner {
static func transcribeQwen3(
samples: [Float],
sampleRate: Int,
locale: Locale,
modelRoot: URL,
layout: LocalASRModelLayout,
runtimeBinary: URL,
bias: LocalASRBiasPayload?
) async throws -> String {
guard sampleRate == 16_000 else {
throw MacLocalASRError.qwen3InferenceFailed("Sherpa expects 16 kHz audio")
}
guard let conv = layout.convFrontend,
let encoder = layout.encoder,
let decoder = layout.decoder,
let tokenizer = layout.tokenizer else {
throw MacLocalASRError.qwen3InferenceFailed("Incomplete Sherpa Qwen3 layout")
}
let wavURL = try writeTemporaryWAV(samples: samples, sampleRate: sampleRate)
defer { try? FileManager.default.removeItem(at: wavURL) }
var arguments = [
"--qwen3-asr-conv-frontend=\(modelRoot.appendingPathComponent(conv).path)",
"--qwen3-asr-encoder=\(modelRoot.appendingPathComponent(encoder).path)",
"--qwen3-asr-decoder=\(modelRoot.appendingPathComponent(decoder).path)",
"--qwen3-asr-tokenizer=\(modelRoot.appendingPathComponent(tokenizer).path)",
"--qwen3-asr-max-new-tokens=512",
"--num-threads=2",
]
if let language = MacQwen3LanguageHint.from(locale: locale) {
arguments.append("--qwen3-asr-language=\(language)")
}
if let hotwords = bias?.hardHotwords, !hotwords.isEmpty {
arguments.append("--qwen3-asr-hotwords=\(hotwords.joined(separator: ","))")
}
arguments.append(wavURL.path)
return try await run(binary: runtimeBinary, arguments: arguments)
}
static func transcribeSenseVoice(
samples: [Float],
sampleRate: Int,
modelRoot: URL,
layout: LocalASRModelLayout,
runtimeBinary: URL
) async throws -> String {
guard sampleRate == 16_000 else {
throw MacLocalASRError.qwen3InferenceFailed("Sherpa expects 16 kHz audio")
}
guard let model = layout.senseVoiceModel,
let tokens = layout.tokens else {
throw MacLocalASRError.qwen3InferenceFailed("Incomplete SenseVoice layout")
}
let wavURL = try writeTemporaryWAV(samples: samples, sampleRate: sampleRate)
defer { try? FileManager.default.removeItem(at: wavURL) }
let arguments = [
"--tokens=\(modelRoot.appendingPathComponent(tokens).path)",
"--sense-voice-model=\(modelRoot.appendingPathComponent(model).path)",
"--num-threads=2",
wavURL.path,
]
return try await run(binary: runtimeBinary, arguments: arguments)
}
// MARK: - Private
private static func writeTemporaryWAV(samples: [Float], sampleRate: Int) throws -> URL {
let data = PCMSampleWavEncoder.encode(samples: samples, sampleRate: sampleRate)
let url = FileManager.default.temporaryDirectory
.appendingPathComponent("osg-sherpa-\(UUID().uuidString).wav")
try data.write(to: url, options: .atomic)
return url
}
private static func run(binary: URL, arguments: [String]) async throws -> String {
try await withCheckedThrowingContinuation { continuation in
let process = Process()
process.executableURL = binary
process.arguments = arguments
process.currentDirectoryURL = binary.deletingLastPathComponent()
let outputPipe = Pipe()
let errorPipe = Pipe()
process.standardOutput = outputPipe
process.standardError = errorPipe
process.terminationHandler = { proc in
let outputData = outputPipe.fileHandleForReading.readDataToEndOfFile()
let errorData = errorPipe.fileHandleForReading.readDataToEndOfFile()
let stdout = String(data: outputData, encoding: .utf8) ?? ""
let stderr = String(data: errorData, encoding: .utf8) ?? ""
guard proc.terminationStatus == 0 else {
let detail = stderr.isEmpty ? stdout : stderr
continuation.resume(
throwing: MacLocalASRError.qwen3InferenceFailed(
detail.trimmingCharacters(in: .whitespacesAndNewlines)
)
)
return
}
let text = parseTranscript(stdout: stdout)
if text.isEmpty {
continuation.resume(throwing: MacLocalASRError.emptyTranscript)
} else {
continuation.resume(returning: text)
}
}
do {
try process.run()
} catch {
continuation.resume(throwing: MacLocalASRError.qwen3InferenceFailed(error.localizedDescription))
}
}
}
private static func parseTranscript(stdout: String) -> String {
let lines = stdout
.split(whereSeparator: \.isNewline)
.map { String($0).trimmingCharacters(in: .whitespacesAndNewlines) }
.filter { !$0.isEmpty }
for line in lines.reversed() {
if line.hasPrefix("{"), let data = line.data(using: .utf8),
let object = try? JSONSerialization.jsonObject(with: data) as? [String: Any],
let text = object["text"] as? String {
let trimmed = text.trimmingCharacters(in: .whitespacesAndNewlines)
if !trimmed.isEmpty { return trimmed }
}
if !line.hasPrefix("/"), !line.hasPrefix("--"), line.count > 1 {
return line
}
}
return ""
}
}
+1 -1
View File
@@ -3,7 +3,7 @@
<plist version="1.0">
<dict>
<key>com.apple.security.app-sandbox</key>
<true/>
<false/>
<key>com.apple.security.device.audio-input</key>
<true/>
<key>com.apple.security.network.client</key>
+50 -7
View File
@@ -86,6 +86,20 @@ final class MacAppDelegate: NSObject, NSApplicationDelegate {
MacAppearancePreference.applyToApp(.current)
configurePopover()
configureStatusItem()
// The menu bar always follows the *system* appearance, so the status
// item must ignore the app's forced light/dark override. Re-pin the
// button appearance whenever the system theme flips.
DistributedNotificationCenter.default.addObserver(
self,
selector: #selector(systemAppearanceDidChange),
name: NSNotification.Name("AppleInterfaceThemeChangedNotification"),
object: nil
)
}
deinit {
DistributedNotificationCenter.default.removeObserver(self)
}
/// Keep the app alive after the last window closes it lives in the menu bar.
@@ -96,18 +110,47 @@ final class MacAppDelegate: NSObject, NSApplicationDelegate {
private func configureStatusItem() {
let item = NSStatusBar.system.statusItem(withLength: NSStatusItem.variableLength)
if let button = item.button {
// Prefer the brand mark; fall back to an SF Symbol so the item is
// never invisible even if the asset fails to resolve.
let image = NSImage(named: "OSGBrandMark")
?? NSImage(systemSymbolName: "mic.circle.fill", accessibilityDescription: "OSGKeyboard")
image?.isTemplate = true
image?.size = NSSize(width: 18, height: 18)
button.image = image
button.image = Self.makeStatusBarImage()
button.image?.accessibilityDescription = "OSGKeyboard"
button.action = #selector(togglePopover(_:))
button.target = self
}
statusItem = item
applyStatusItemAppearance()
}
/// Builds the menu-bar glyph from the dedicated horizontal status mark.
/// Height is pinned to the tallest practical menu-bar slot so the logo reads
/// clearly; width follows the asset's aspect ratio.
private static func makeStatusBarImage() -> NSImage? {
guard let image = NSImage(named: "OSGStatusMark")
?? NSImage(named: "OSGBrandMark")
?? NSImage(systemSymbolName: "mic.circle.fill", accessibilityDescription: "OSGKeyboard") else {
return nil
}
let height: CGFloat = 9
let aspect = max(image.size.width / max(image.size.height, 1), 1)
image.size = NSSize(width: height * aspect, height: height)
image.isTemplate = true
return image
}
/// Pins the status-bar button to the current *system* appearance so its
/// template image tint matches the real menu-bar background regardless of
/// the in-app light/dark preference forced on `NSApp.appearance`.
private func applyStatusItemAppearance() {
guard let button = statusItem?.button else { return }
let isDark = UserDefaults.standard.string(forKey: "AppleInterfaceStyle")?
.lowercased().contains("dark") ?? false
button.appearance = NSAppearance(named: isDark ? .darkAqua : .aqua)
}
@objc private func systemAppearanceDidChange() {
// The global-domain default lags the notification by a hair; hop to the
// next runloop tick so `AppleInterfaceStyle` reflects the new value.
DispatchQueue.main.async { [weak self] in
self?.applyStatusItemAppearance()
}
}
private func configurePopover() {
@@ -0,0 +1,99 @@
// LocalASRBiasPayload.swift
// OSGKeyboard · Shared
//
// Output of `LocalASRBiasAdapter` vocabulary signals for each pipeline layer.
import Foundation
public struct LocalASRCorrectionPair: Sendable, Equatable {
public let alias: String
public let term: String
public init(alias: String, term: String) {
self.alias = alias
self.term = term
}
}
public struct LocalASRBiasDiagnostics: Sendable, Equatable, Codable {
public var userTermCount: Int
public var builtinTermCount: Int
public var truncated: Bool
public var truncationReason: String?
public var selectedSources: [String]
public init(
userTermCount: Int = 0,
builtinTermCount: Int = 0,
truncated: Bool = false,
truncationReason: String? = nil,
selectedSources: [String] = []
) {
self.userTermCount = userTermCount
self.builtinTermCount = builtinTermCount
self.truncated = truncated
self.truncationReason = truncationReason
self.selectedSources = selectedSources
}
}
public struct LocalASRBiasPayload: Sendable, Equatable {
public var hardHotwords: [String]
public var promptBias: String?
public var corpusContext: String?
public var polishFragment: String
public var correctionPairs: [LocalASRCorrectionPair]
public var diagnostics: LocalASRBiasDiagnostics
public static let empty = LocalASRBiasPayload(
hardHotwords: [],
promptBias: nil,
corpusContext: nil,
polishFragment: "",
correctionPairs: [],
diagnostics: LocalASRBiasDiagnostics()
)
public init(
hardHotwords: [String],
promptBias: String?,
corpusContext: String?,
polishFragment: String,
correctionPairs: [LocalASRCorrectionPair],
diagnostics: LocalASRBiasDiagnostics
) {
self.hardHotwords = hardHotwords
self.promptBias = promptBias
self.corpusContext = corpusContext
self.polishFragment = polishFragment
self.correctionPairs = correctionPairs
self.diagnostics = diagnostics
}
}
public struct LocalASRBiasRequest: Sendable {
public var dictionary: PersonalDictionary
public var locale: Locale
public var frontAppBundleId: String?
public var capabilities: LocalASRCapabilities
/// Max builtin `phrases.tsv` terms considered for ASR bias (not polish-only).
public var builtinASRLimit: Int
/// Max builtin terms referenced in the polish supplement block.
public var builtinPolishLimit: Int
public init(
dictionary: PersonalDictionary,
locale: Locale,
frontAppBundleId: String? = nil,
capabilities: LocalASRCapabilities,
builtinASRLimit: Int = 300,
builtinPolishLimit: Int = 40
) {
self.dictionary = dictionary
self.locale = locale
self.frontAppBundleId = frontAppBundleId
self.capabilities = capabilities
self.builtinASRLimit = builtinASRLimit
self.builtinPolishLimit = builtinPolishLimit
}
}
@@ -0,0 +1,82 @@
// LocalASRCapabilities.swift
// OSGKeyboard · Shared
//
// Declares what each on-device ASR backend can accept for vocabulary bias.
// Callers must consult capabilities before building a `LocalASRBiasPayload`.
import Foundation
/// How a backend accepts vocabulary hints (honest matrix not every model
/// supports hard hotwords).
public enum LocalASRHotwordMode: String, Sendable, Codable, Equatable {
case none
case promptOnly
case perRequest
case recognizerScoped
case cloudVocabulary
}
/// Cost of refreshing hotwords on a backend (e.g. Sherpa Qwen3 reloads recognizer).
public enum LocalASRHotwordReloadCost: String, Sendable, Codable, Equatable {
case none
case recognizerReload
case modelReload
}
public struct LocalASRCapabilities: Sendable, Equatable {
public let hotwordMode: LocalASRHotwordMode
public let maxHotwordCount: Int
public let maxPromptCharacters: Int
public let supportsStreaming: Bool
public let hotwordReloadCost: LocalASRHotwordReloadCost
public init(
hotwordMode: LocalASRHotwordMode,
maxHotwordCount: Int,
maxPromptCharacters: Int,
supportsStreaming: Bool,
hotwordReloadCost: LocalASRHotwordReloadCost
) {
self.hotwordMode = hotwordMode
self.maxHotwordCount = maxHotwordCount
self.maxPromptCharacters = maxPromptCharacters
self.supportsStreaming = supportsStreaming
self.hotwordReloadCost = hotwordReloadCost
}
/// Qwen3 MLX via mlx-swift-asr `context` soft prompt on `transcribe`.
public static let qwen3MLX = LocalASRCapabilities(
hotwordMode: .promptOnly,
maxHotwordCount: 0,
maxPromptCharacters: 800,
supportsStreaming: false,
hotwordReloadCost: .none
)
/// Apple Speech on macOS no project-controlled hotword API today.
public static let appleSpeech = LocalASRCapabilities(
hotwordMode: .none,
maxHotwordCount: 0,
maxPromptCharacters: 0,
supportsStreaming: false,
hotwordReloadCost: .none
)
/// Sherpa Qwen3 hard hotwords via `--qwen3-asr-hotwords`.
public static let sherpaQwen3 = LocalASRCapabilities(
hotwordMode: .recognizerScoped,
maxHotwordCount: 100,
maxPromptCharacters: 0,
supportsStreaming: false,
hotwordReloadCost: .recognizerReload
)
/// Sherpa SenseVoice fast Chinese baseline without hotwords.
public static let sherpaSenseVoice = LocalASRCapabilities(
hotwordMode: .none,
maxHotwordCount: 0,
maxPromptCharacters: 0,
supportsStreaming: false,
hotwordReloadCost: .none
)
}
@@ -0,0 +1,134 @@
// LocalASRModelCatalog.swift
// OSGKeyboard · Shared
//
// Bundled catalog of downloadable / manual local ASR models and Sherpa runtimes.
import Foundation
public enum LocalASRModelBackend: String, Codable, Sendable, Equatable {
case mlx
case sherpaQwen3
case sherpaSenseVoice
case appleSpeech
}
public enum LocalASRInstallKind: String, Codable, Sendable, Equatable {
case manual
case archive
case runtime
}
public struct LocalASRDownloadSource: Codable, Sendable, Equatable {
public let type: String
public let priority: Int
public let url: String
}
public struct LocalASRModelLayout: Codable, Sendable, Equatable {
public var convFrontend: String?
public var encoder: String?
public var decoder: String?
public var tokenizer: String?
public var senseVoiceModel: String?
public var tokens: String?
}
public struct LocalASRRuntimeDefinition: Codable, Sendable, Equatable, Identifiable {
public let id: String
public let displayName: String
public let installRelativePath: String
public let binaryCandidates: [String]
public let archiveFileName: String
public let sizeBytes: Int
public let platform: String
public let sources: [LocalASRDownloadSource]
}
public struct LocalASRModelDefinition: Codable, Sendable, Equatable, Identifiable {
public let id: String
public let displayName: String
public let backend: LocalASRModelBackend
public let sizeBytes: Int
public let recommendedLocales: [String]
public let supportsHotwords: Bool
public let hotwordMode: LocalASRHotwordMode
public let installKind: LocalASRInstallKind
public let installRelativePath: String?
public let archiveBaseName: String?
public let layout: LocalASRModelLayout?
public let requiredRelativeFiles: [String]?
public let runtimePlatform: String?
public let sources: [LocalASRDownloadSource]?
}
public struct LocalASRCatalogDocument: Codable, Sendable, Equatable {
public let schemaVersion: Int
public let defaultModelId: String
public let runtimes: [LocalASRRuntimeDefinition]
public let models: [LocalASRModelDefinition]
}
public enum LocalASRModelCatalog {
public static func loadBundled() throws -> LocalASRCatalogDocument {
let bundle = Bundle(for: LocalASRCatalogBundleToken.self)
guard let url = bundle.url(forResource: "local-asr-catalog", withExtension: "json") else {
throw LocalASRModelCatalogError.missingBundledCatalog
}
let data = try Data(contentsOf: url)
return try JSONDecoder().decode(LocalASRCatalogDocument.self, from: data)
}
public static func model(_ id: String, in catalog: LocalASRCatalogDocument) -> LocalASRModelDefinition? {
catalog.models.first { $0.id == id }
}
public static func capabilities(for model: LocalASRModelDefinition) -> LocalASRCapabilities {
switch model.backend {
case .mlx:
return .qwen3MLX
case .sherpaQwen3:
return .sherpaQwen3
case .sherpaSenseVoice:
return .sherpaSenseVoice
case .appleSpeech:
return .appleSpeech
}
}
#if os(macOS)
public static func runtime(for platform: String, in catalog: LocalASRCatalogDocument) -> LocalASRRuntimeDefinition? {
if platform == "macos-arm64" {
return catalog.runtimes.first { $0.platform == "macos-arm64" }
}
if platform == "macos-x64" {
return catalog.runtimes.first { $0.platform == "macos-x64" }
}
return catalog.runtimes.first
}
public static func currentRuntimePlatform() -> String {
#if arch(arm64)
return "macos-arm64"
#else
return "macos-x64"
#endif
}
#endif
}
public enum LocalASRModelCatalogError: Error, LocalizedError {
case missingBundledCatalog
case modelNotFound(String)
public var errorDescription: String? {
switch self {
case .missingBundledCatalog:
return "Missing bundled local ASR catalog."
case .modelNotFound(let id):
return "Local ASR model not found: \(id)"
}
}
}
private final class LocalASRCatalogBundleToken {}
@@ -111,4 +111,29 @@ extension PersonalDictionary {
if hasNonASCII { return "zh" }
return "en"
}
/// Alias canonical term pairs for deterministic post-ASR correction.
/// Sorted longest-alias-first by the caller (`LocalASRTranscriptCorrector`).
public func localCorrectionPairs() -> [LocalASRCorrectionPair] {
var seen = Set<String>()
var pairs: [LocalASRCorrectionPair] = []
for entry in effectiveEntries {
let term = entry.term.trimmingCharacters(in: .whitespacesAndNewlines)
guard !term.isEmpty else { continue }
for alias in entry.aliases {
let trimmed = alias.trimmingCharacters(in: .whitespacesAndNewlines)
guard !trimmed.isEmpty else { continue }
guard trimmed.caseInsensitiveCompare(term) != .orderedSame else { continue }
let key = "\(trimmed.lowercased())|\(term.lowercased())"
guard seen.insert(key).inserted else { continue }
pairs.append(LocalASRCorrectionPair(alias: trimmed, term: term))
}
}
return pairs.sorted { lhs, rhs in
if lhs.alias.count != rhs.alias.count {
return lhs.alias.count > rhs.alias.count
}
return lhs.alias.localizedCaseInsensitiveCompare(rhs.alias) == .orderedAscending
}
}
}
@@ -24,6 +24,10 @@ public struct PolishContext: Sendable {
/// bias terminology choices.
public let precedingText: String?
/// Extra dictionary block appended after `PersonalDictionary.promptFragment()`
/// (e.g. builtin `phrases.tsv` terms on macOS local ASR).
public let dictionarySupplement: String?
/// Cap on how many characters of `precedingText` we actually
/// include in the prompt. The full preceding text is often
/// hundreds of KB in a long note we only need the tail.
@@ -33,11 +37,13 @@ public struct PolishContext: Sendable {
appContext: AppContext = .unknown,
intensity: PolishIntensity = .default,
precedingText: String? = nil,
dictionarySupplement: String? = nil,
maxPrecedingChars: Int = 500
) {
self.appContext = appContext
self.intensity = intensity
self.precedingText = precedingText
self.dictionarySupplement = dictionarySupplement
self.maxPrecedingChars = maxPrecedingChars
}
@@ -74,7 +74,7 @@ public final class ProviderConfig: ObservableObject, @unchecked Sendable {
guard !isApplyingConfiguration, engineMode != configuration.engineMode else { return }
configuration.engineMode = engineMode
applyEngineModeSideEffects()
persistConfiguration()
persistConfiguration(postConfigChanged: true)
}
}
@Published public var hasCompletedOnboarding: Bool {
@@ -0,0 +1,102 @@
{
"schemaVersion": 1,
"defaultModelId": "qwen3-mlx-1.7b",
"runtimes": [
{
"id": "sherpa-onnx-1.13.4-macos-arm64",
"displayName": "sherpa-onnx 1.13.4 (Apple Silicon)",
"installRelativePath": "runtimes/sherpa-onnx-1.13.4-macos-arm64",
"binaryCandidates": ["bin/sherpa-onnx-offline", "sherpa-onnx-offline"],
"archiveFileName": "sherpa-onnx-v1.13.4-osx-arm64-static-no-tts.tar.bz2",
"sizeBytes": 120000000,
"platform": "macos-arm64",
"sources": [
{
"type": "github",
"priority": 1,
"url": "https://github.com/k2-fsa/sherpa-onnx/releases/download/v1.13.4/sherpa-onnx-v1.13.4-osx-arm64-static-no-tts.tar.bz2"
}
]
},
{
"id": "sherpa-onnx-1.13.4-macos-x64",
"displayName": "sherpa-onnx 1.13.4 (Intel)",
"installRelativePath": "runtimes/sherpa-onnx-1.13.4-macos-x64",
"binaryCandidates": ["bin/sherpa-onnx-offline", "sherpa-onnx-offline"],
"archiveFileName": "sherpa-onnx-v1.13.4-osx-x64-static-no-tts.tar.bz2",
"sizeBytes": 130000000,
"platform": "macos-x64",
"sources": [
{
"type": "github",
"priority": 1,
"url": "https://github.com/k2-fsa/sherpa-onnx/releases/download/v1.13.4/sherpa-onnx-v1.13.4-osx-x64-static-no-tts.tar.bz2"
}
]
}
],
"models": [
{
"id": "qwen3-mlx-1.7b",
"displayName": "Qwen3-ASR 1.7B (MLX)",
"backend": "mlx",
"sizeBytes": 1400000000,
"recommendedLocales": ["zh-CN", "en-US"],
"supportsHotwords": true,
"hotwordMode": "promptOnly",
"installKind": "manual",
"installRelativePath": "models/qwen3-asr-1.7b-mlx",
"requiredRelativeFiles": ["config.json", "model.safetensors", "vocab.json", "merges.txt"]
},
{
"id": "sherpa-qwen3-0.6b-int8",
"displayName": "Qwen3-ASR 0.6B (Sherpa · hotwords)",
"backend": "sherpaQwen3",
"runtimePlatform": "macos",
"sizeBytes": 650000000,
"recommendedLocales": ["zh-CN", "en-US"],
"supportsHotwords": true,
"hotwordMode": "recognizerScoped",
"installKind": "archive",
"installRelativePath": "models/sherpa-qwen3-0.6b-int8",
"archiveBaseName": "sherpa-onnx-qwen3-asr-0.6B-int8-2026-03-25",
"layout": {
"convFrontend": "conv_frontend.onnx",
"encoder": "encoder.int8.onnx",
"decoder": "decoder.int8.onnx",
"tokenizer": "tokenizer"
},
"sources": [
{
"type": "github",
"priority": 1,
"url": "https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-qwen3-asr-0.6B-int8-2026-03-25.tar.bz2"
}
]
},
{
"id": "sherpa-sensevoice-small-int8",
"displayName": "SenseVoice Small (Sherpa)",
"backend": "sherpaSenseVoice",
"runtimePlatform": "macos",
"sizeBytes": 250000000,
"recommendedLocales": ["zh-CN", "en-US", "ja-JP", "ko-KR"],
"supportsHotwords": false,
"hotwordMode": "none",
"installKind": "archive",
"installRelativePath": "models/sherpa-sensevoice-small-int8",
"archiveBaseName": "sherpa-onnx-sense-voice-zh-en-ja-ko-yue-int8-2024-07-17",
"layout": {
"senseVoiceModel": "model.int8.onnx",
"tokens": "tokens.txt"
},
"sources": [
{
"type": "github",
"priority": 1,
"url": "https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-sense-voice-zh-en-ja-ko-yue-int8-2024-07-17.tar.bz2"
}
]
}
]
}
@@ -92,6 +92,7 @@ public struct AppGroupStore: @unchecked Sendable {
config.model = openAI.defaultModel
}
}
AppGroupConfigDarwin.postConfigChanged()
}
public func setUILanguage(_ language: AppUILanguage) {
@@ -0,0 +1,152 @@
// BuiltinLexiconIndex.swift
// OSGKeyboard · Shared
//
// In-memory index over bundled `phrases.tsv` (~10k computer terms).
// macOS local ASR consumes a Top-N subset; the full index also backs
// polish supplements and future retrieval.
import Foundation
public final class BuiltinLexiconIndex: @unchecked Sendable {
public struct Term: Sendable, Equatable {
public let word: String
public let pinyin: String
public let source: String
public let weight: Int
}
public static let shared = BuiltinLexiconIndex()
private let lock = NSLock()
private var cachedTerms: [Term]?
private let injectedURL: URL?
/// Production singleton loads from the app bundle.
private init() {
injectedURL = nil
}
/// Test / preview hook with an explicit TSV file or inline fixture.
init(fixtureURL: URL) {
injectedURL = fixtureURL
}
/// Parse TSV content without touching the bundle (unit tests).
public static func parseTSV(_ content: String) -> [Term] {
var terms: [Term] = []
terms.reserveCapacity(256)
for (lineIndex, line) in content.split(whereSeparator: \.isNewline).enumerated() {
if lineIndex == 0, line.hasPrefix("word\t") { continue }
let columns = line.split(separator: "\t", omittingEmptySubsequences: false)
guard columns.count >= 4 else { continue }
let word = String(columns[0]).trimmingCharacters(in: .whitespacesAndNewlines)
guard !word.isEmpty else { continue }
let pinyin = String(columns[1])
let source = String(columns[2])
let weight = Int(columns[3]) ?? 1
terms.append(Term(word: word, pinyin: pinyin, source: source, weight: weight))
}
return terms
}
public func termCount() -> Int {
lock.lock()
defer { lock.unlock() }
return loadTermsLocked().count
}
/// Returns canonical words ranked for ASR bias injection.
public func topTerms(
limit: Int,
minimumWeight: Int = 4,
preferredSources: Set<String>? = nil
) -> [String] {
guard limit > 0 else { return [] }
lock.lock()
let all = loadTermsLocked()
lock.unlock()
let filtered = all.filter { term in
guard term.weight >= minimumWeight else { return false }
if let preferredSources, !preferredSources.isEmpty {
return preferredSources.contains(term.source)
}
return true
}
let ranked = filtered.sorted { lhs, rhs in
let leftScore = Self.rankingScore(lhs)
let rightScore = Self.rankingScore(rhs)
if leftScore != rightScore { return leftScore > rightScore }
return lhs.word.localizedCaseInsensitiveCompare(rhs.word) == .orderedAscending
}
var seen = Set<String>()
var words: [String] = []
words.reserveCapacity(min(limit, ranked.count))
for term in ranked {
let key = term.word.lowercased()
guard seen.insert(key).inserted else { continue }
words.append(term.word)
if words.count >= limit { break }
}
return words
}
// MARK: - Private
private func loadTermsLocked() -> [Term] {
if let cachedTerms { return cachedTerms }
let loaded: [Term]
if let injectedURL {
loaded = Self.load(from: injectedURL)
} else if let url = Self.locateBundledPhrasesURL() {
loaded = Self.load(from: url)
} else {
loaded = []
}
cachedTerms = loaded
return loaded
}
private static func load(from url: URL) -> [Term] {
guard let data = try? Data(contentsOf: url),
let content = String(data: data, encoding: .utf8) else {
return []
}
return parseTSV(content)
}
private static func locateBundledPhrasesURL() -> URL? {
let candidates: [Bundle] = [Bundle.main, Bundle(for: BuiltinLexiconIndex.self)]
for bundle in candidates {
if let url = bundle.url(
forResource: "phrases",
withExtension: "tsv",
subdirectory: "CustomLanguageModel/v1"
) {
return url
}
if let url = bundle.url(forResource: "phrases", withExtension: "tsv") {
return url
}
}
return nil
}
private static func rankingScore(_ term: Term) -> Int {
var score = term.weight * 100
if containsLatinLetters(term.word) { score += 50 }
if term.word.count <= 8 { score += 10 }
return score
}
private static func containsLatinLetters(_ text: String) -> Bool {
text.unicodeScalars.contains { scalar in
scalar.isASCII && CharacterSet.letters.contains(scalar)
}
}
}
@@ -384,19 +384,26 @@ public final class FlowContinuousCapture {
/// Re-activate capture after returning from background without
/// reinstalling the tap (iOS may deactivate the audio session).
public func reassertIfRunning() {
guard isRunning else { return }
@discardableResult
public func reassertIfRunning() -> Bool {
guard isRunning else { return false }
let session = AVAudioSession.sharedInstance()
try? session.setCategory(
do {
try session.setCategory(
.playAndRecord,
mode: .measurement,
options: [.defaultToSpeaker, .allowBluetoothHFP, .mixWithOthers]
)
try? session.setActive(true, options: .notifyOthersOnDeactivation)
try session.setActive(true, options: .notifyOthersOnDeactivation)
if !audioEngine.isRunning {
try? audioEngine.start()
try audioEngine.start()
}
notifyEngineLiveChanged()
return engineIsLive
} catch {
notifyEngineLiveChanged()
return false
}
}
public func awaitAudioFlowing(
@@ -6,6 +6,158 @@
import Foundation
public struct FlowCommand: Codable, Equatable, Sendable {
public enum Action: String, Codable, Sendable {
case startRecording
case stopRecording
case abort
}
public let protocolVersion: Int
public let sessionId: UUID
public let utteranceId: UUID
public let commandSeq: Int64
public let action: Action
public let localeId: String
public let createdAt: TimeInterval
public init(
protocolVersion: Int = 1,
sessionId: UUID,
utteranceId: UUID,
commandSeq: Int64,
action: Action,
localeId: String,
createdAt: TimeInterval = Date().timeIntervalSince1970
) {
self.protocolVersion = protocolVersion
self.sessionId = sessionId
self.utteranceId = utteranceId
self.commandSeq = commandSeq
self.action = action
self.localeId = localeId
self.createdAt = createdAt
}
}
public struct FlowResult: Codable, Equatable, Sendable {
public enum Status: String, Codable, Sendable {
case partial
case final
case error
case aborted
case timeout
}
public let protocolVersion: Int
public let sessionId: UUID
public let utteranceId: UUID
public let commandSeq: Int64
public let status: Status
public let text: String?
public let warning: String?
public let errorKind: FlowSessionKeys.TranscriptionErrorKind?
public let createdAt: TimeInterval
public init(
protocolVersion: Int = 1,
sessionId: UUID,
utteranceId: UUID,
commandSeq: Int64,
status: Status,
text: String? = nil,
warning: String? = nil,
errorKind: FlowSessionKeys.TranscriptionErrorKind? = nil,
createdAt: TimeInterval = Date().timeIntervalSince1970
) {
self.protocolVersion = protocolVersion
self.sessionId = sessionId
self.utteranceId = utteranceId
self.commandSeq = commandSeq
self.status = status
self.text = text
self.warning = warning
self.errorKind = errorKind
self.createdAt = createdAt
}
}
public struct FlowAck: Codable, Equatable, Sendable {
public let protocolVersion: Int
public let sessionId: UUID
public let utteranceId: UUID
public let commandSeq: Int64
public let consumedAt: TimeInterval
public init(
protocolVersion: Int = 1,
sessionId: UUID,
utteranceId: UUID,
commandSeq: Int64,
consumedAt: TimeInterval = Date().timeIntervalSince1970
) {
self.protocolVersion = protocolVersion
self.sessionId = sessionId
self.utteranceId = utteranceId
self.commandSeq = commandSeq
self.consumedAt = consumedAt
}
}
public struct FlowReadySnapshot: Codable, Equatable, Sendable {
public enum Reason: String, Codable, Sendable {
case ready
case noSession
case starting
case audioEngineNotLive
case waitingForAudioProof
case recording
case processing
case permissionMissing
case appGroupUnavailable
case hostLost
case error
}
public let protocolVersion: Int
public let sessionId: UUID?
public let ready: Bool
public let reason: Reason
public let heartbeatAt: TimeInterval
public let readyAt: TimeInterval?
public let audioProofAt: TimeInterval?
public let engineMode: String
public let localeId: String
public let busyUtteranceId: UUID?
public let sessionExpiresAt: TimeInterval?
public init(
protocolVersion: Int = 1,
sessionId: UUID?,
ready: Bool,
reason: Reason,
heartbeatAt: TimeInterval = Date().timeIntervalSince1970,
readyAt: TimeInterval? = nil,
audioProofAt: TimeInterval? = nil,
engineMode: String,
localeId: String,
busyUtteranceId: UUID? = nil,
sessionExpiresAt: TimeInterval? = nil
) {
self.protocolVersion = protocolVersion
self.sessionId = sessionId
self.ready = ready
self.reason = reason
self.heartbeatAt = heartbeatAt
self.readyAt = readyAt
self.audioProofAt = audioProofAt
self.engineMode = engineMode
self.localeId = localeId
self.busyUtteranceId = busyUtteranceId
self.sessionExpiresAt = sessionExpiresAt
}
}
public struct FlowTranscriptionError: Equatable, Sendable {
public let message: String
public let kind: FlowSessionKeys.TranscriptionErrorKind
@@ -36,6 +188,15 @@ public enum FlowSessionBridge {
}
}
private static func encode<T: Encodable>(_ value: T) -> Data? {
try? JSONEncoder().encode(value)
}
private static func decode<T: Decodable>(_ type: T.Type, from data: Data?) -> T? {
guard let data else { return nil }
return try? JSONDecoder().decode(type, from: data)
}
/// Keyboard/read side: refresh App Group defaults after the extension was
/// suspended so decisions are not based on stale in-process caches.
public static func reloadFromDisk(defaults: UserDefaults? = nil) {
@@ -45,10 +206,87 @@ public enum FlowSessionBridge {
}
}
// MARK: - Typed Flow protocol
public static func writeCommand(_ command: FlowCommand, defaults: UserDefaults? = nil) {
let store = resolvedDefaults(defaults)
if let data = encode(command) {
store.set(data, forKey: FlowSessionKeys.flowCommandPayload)
}
flush(store)
FlowSessionDarwin.postCommandChanged()
}
public static func latestCommand(defaults: UserDefaults? = nil) -> FlowCommand? {
let store = resolvedDefaults(defaults)
return decode(FlowCommand.self, from: store.data(forKey: FlowSessionKeys.flowCommandPayload))
}
public static func writeResult(_ result: FlowResult, defaults: UserDefaults? = nil) {
let store = resolvedDefaults(defaults)
if let data = encode(result) {
store.set(data, forKey: FlowSessionKeys.flowResultPayload)
}
flush(store)
FlowSessionDarwin.postTranscriptionChanged()
}
public static func latestResult(defaults: UserDefaults? = nil) -> FlowResult? {
let store = resolvedDefaults(defaults)
return decode(FlowResult.self, from: store.data(forKey: FlowSessionKeys.flowResultPayload))
}
public static func clearResult(defaults: UserDefaults? = nil) {
let store = resolvedDefaults(defaults)
store.removeObject(forKey: FlowSessionKeys.flowResultPayload)
flush(store)
}
public static func writeAck(_ ack: FlowAck, defaults: UserDefaults? = nil) {
let store = resolvedDefaults(defaults)
if let data = encode(ack) {
store.set(data, forKey: FlowSessionKeys.flowAckPayload)
}
flush(store)
}
public static func latestAck(defaults: UserDefaults? = nil) -> FlowAck? {
let store = resolvedDefaults(defaults)
return decode(FlowAck.self, from: store.data(forKey: FlowSessionKeys.flowAckPayload))
}
public static func writeReadySnapshot(_ snapshot: FlowReadySnapshot, defaults: UserDefaults? = nil) {
let store = resolvedDefaults(defaults)
if let data = encode(snapshot) {
store.set(data, forKey: FlowSessionKeys.flowReadyPayload)
}
if snapshot.ready {
store.set(true, forKey: FlowSessionKeys.flowHostReady)
if let readyAt = snapshot.readyAt {
store.set(readyAt, forKey: FlowSessionKeys.flowHostReadyAt)
}
} else {
clearHostReady(defaults: store, notify: false)
store.removeObject(forKey: FlowSessionKeys.flowReadyPayload)
}
if let expires = snapshot.sessionExpiresAt {
store.set(expires, forKey: FlowSessionKeys.flowSessionExpires)
}
store.set(snapshot.heartbeatAt, forKey: FlowSessionKeys.flowHeartbeat)
flush(store)
FlowSessionDarwin.postHostReadyChanged()
}
public static func readySnapshot(defaults: UserDefaults? = nil) -> FlowReadySnapshot? {
let store = resolvedDefaults(defaults)
return decode(FlowReadySnapshot.self, from: store.data(forKey: FlowSessionKeys.flowReadyPayload))
}
// MARK: - Session lifecycle (host app)
public static func markSessionActive(
duration: TimeInterval? = nil,
sessionId: UUID? = nil,
defaults: UserDefaults? = nil
) {
let store = resolvedDefaults(defaults)
@@ -59,8 +297,26 @@ public enum FlowSessionBridge {
store.set(expires, forKey: FlowSessionKeys.flowSessionExpires)
store.set(now, forKey: FlowSessionKeys.lastActivityAt)
writeHeartbeat(defaults: store)
setRecordingState(.idle, defaults: store)
clearTranscription(defaults: store)
store.removeObject(forKey: FlowSessionKeys.flowCommandPayload)
store.removeObject(forKey: FlowSessionKeys.flowResultPayload)
store.removeObject(forKey: FlowSessionKeys.flowAckPayload)
if let sessionId {
let snapshot = FlowReadySnapshot(
sessionId: sessionId,
ready: false,
reason: .starting,
heartbeatAt: now,
engineMode: AppGroupConfiguration.load(fromAvailable: store).engineMode,
localeId: AppGroupConfiguration.load(fromAvailable: store).localeId,
sessionExpiresAt: expires
)
if let data = encode(snapshot) {
store.set(data, forKey: FlowSessionKeys.flowReadyPayload)
}
} else {
store.removeObject(forKey: FlowSessionKeys.flowReadyPayload)
}
flush(store)
}
@@ -69,8 +325,11 @@ public enum FlowSessionBridge {
store.set(false, forKey: FlowSessionKeys.flowSessionActive)
store.removeObject(forKey: FlowSessionKeys.flowSessionExpires)
store.removeObject(forKey: FlowSessionKeys.flowHeartbeat)
setRecordingState(.idle, defaults: store)
clearTranscription(defaults: store)
store.removeObject(forKey: FlowSessionKeys.flowCommandPayload)
store.removeObject(forKey: FlowSessionKeys.flowResultPayload)
store.removeObject(forKey: FlowSessionKeys.flowAckPayload)
store.removeObject(forKey: FlowSessionKeys.flowReadyPayload)
clearHostReady(defaults: store, notify: false)
flush(store)
}
@@ -185,6 +444,15 @@ public enum FlowSessionBridge {
/// True when the host has published a fresh ready contract (stricter than heartbeat alone).
public static func isHostReady(defaults: UserDefaults? = nil) -> Bool {
let store = resolvedDefaults(defaults)
if let snapshot = readySnapshot(defaults: store) {
guard snapshot.ready else { return false }
guard isHostReachable(defaults: store) else { return false }
if let readyAt = snapshot.readyAt {
let skew = abs(snapshot.heartbeatAt - readyAt)
guard skew <= FlowSessionKeys.hostReadyMaxHeartbeatSkew else { return false }
}
return true
}
guard isHostReachable(defaults: store) else { return false }
return store.bool(forKey: FlowSessionKeys.flowHostReady)
}
@@ -393,6 +661,10 @@ public enum FlowSessionBridge {
store.removeObject(forKey: FlowSessionKeys.flowHeartbeat)
store.removeObject(forKey: FlowSessionKeys.keyboardRecordingState)
store.removeObject(forKey: FlowSessionKeys.transcriptionLanguage)
store.removeObject(forKey: FlowSessionKeys.flowCommandPayload)
store.removeObject(forKey: FlowSessionKeys.flowResultPayload)
store.removeObject(forKey: FlowSessionKeys.flowAckPayload)
store.removeObject(forKey: FlowSessionKeys.flowReadyPayload)
clearTranscription(defaults: store)
store.removeObject(forKey: FlowSessionKeys.audioLevels)
store.removeObject(forKey: FlowSessionKeys.pendingHostBundleId)
@@ -8,6 +8,8 @@ import Foundation
public enum FlowSessionDarwin {
public static let notificationName = "com.osgkeyboard.flow.session.changed"
/// Posted when the keyboard writes a command for the host app.
public static let commandNotificationName = "com.osgkeyboard.flow.command.changed"
/// Posted when the host app writes a transcription result or error.
public static let transcriptionNotificationName = "com.osgkeyboard.flow.transcription.changed"
/// Posted when the host app publishes or clears the ready contract.
@@ -23,6 +25,16 @@ public enum FlowSessionDarwin {
)
}
public static func postCommandChanged() {
CFNotificationCenterPostNotification(
CFNotificationCenterGetDarwinNotifyCenter(),
CFNotificationName(commandNotificationName as CFString),
nil,
nil,
true
)
}
public static func postTranscriptionChanged() {
CFNotificationCenterPostNotification(
CFNotificationCenterGetDarwinNotifyCenter(),
@@ -7,6 +7,10 @@
import Foundation
public enum FlowSessionKeys {
public static let flowCommandPayload = "flow.commandPayload.v1"
public static let flowResultPayload = "flow.resultPayload.v1"
public static let flowAckPayload = "flow.ackPayload.v1"
public static let flowReadyPayload = "flow.readyPayload.v1"
public static let flowSessionActive = "flow.flowSessionActive"
public static let flowSessionExpires = "flow.flowSessionExpires"
public static let flowHeartbeat = "flow.flowHeartbeat"
@@ -74,7 +78,7 @@ public enum FlowSessionKeys {
}
/// Structured host keyboard transcription failure kind.
public enum TranscriptionErrorKind: String, Sendable, Equatable {
public enum TranscriptionErrorKind: String, Sendable, Equatable, Codable {
case noSpeech
case recognitionInterrupted
case audioUnavailable
@@ -0,0 +1,159 @@
// LocalASRBiasAdapter.swift
// OSGKeyboard · Shared
//
// Maps `PersonalDictionary` + builtin lexicon + runtime context into the
// layered bias outputs consumed by local ASR, correction, and polish.
import Foundation
public enum LocalASRBiasAdapter {
/// Bundle IDs where computer-science vocabulary is especially likely.
private static let codeEditorBundleIDs: Set<String> = [
"com.apple.dt.Xcode",
"com.microsoft.VSCode",
"com.google.android.studio",
"com.jetbrains.intellij",
"com.jetbrains.AppCode",
"com.sublimetext.4",
"com.apple.Terminal",
"com.googlecode.iterm2",
"dev.warp.Warp-Stable",
]
public static func adapt(
_ request: LocalASRBiasRequest,
lexicon: BuiltinLexiconIndex = .shared
) -> LocalASRBiasPayload {
let capabilities = request.capabilities
let dictionary = request.dictionary
var selectedSources = ["user"]
let preferredSources = Self.preferredLexiconSources(for: request.frontAppBundleId)
if preferredSources != nil {
selectedSources.append("builtin-computer")
} else {
selectedSources.append("builtin-top")
}
let userSorted = dictionary.effectiveEntries.sorted { $0.usageCount > $1.usageCount }
var mergedTerms: [String] = []
var seen = Set<String>()
func appendTerm(_ term: String) {
let trimmed = term.trimmingCharacters(in: .whitespacesAndNewlines)
guard !trimmed.isEmpty else { return }
let key = trimmed.lowercased()
guard seen.insert(key).inserted else { return }
mergedTerms.append(trimmed)
}
for entry in userSorted {
appendTerm(entry.term)
}
let userTermCount = mergedTerms.count
let builtinWords = lexicon.topTerms(
limit: request.builtinASRLimit,
minimumWeight: 4,
preferredSources: preferredSources
)
let beforeBuiltin = mergedTerms.count
for word in builtinWords {
appendTerm(word)
}
let builtinTermCount = mergedTerms.count - beforeBuiltin
var hardHotwords: [String] = []
switch capabilities.hotwordMode {
case .perRequest, .recognizerScoped:
let cap = max(capabilities.maxHotwordCount, 1)
hardHotwords = Self.hardHotwordList(from: mergedTerms, maxCount: cap)
case .cloudVocabulary:
hardHotwords = dictionary.asrHotwords(maxCount: max(capabilities.maxHotwordCount, 1))
case .none, .promptOnly:
break
}
var promptBias: String?
var truncated = false
var truncationReason: String?
if capabilities.hotwordMode == .promptOnly, capabilities.maxPromptCharacters > 0 {
let built = Self.buildPromptBias(
dictionary: dictionary,
builtinTerms: builtinWords,
maxCharacters: capabilities.maxPromptCharacters
)
if built.count > capabilities.maxPromptCharacters {
truncated = true
truncationReason = "promptBias exceeded \(capabilities.maxPromptCharacters) characters"
}
promptBias = built.isEmpty ? nil : built
}
let polishFragment = Self.buildPolishFragment(
dictionary: dictionary,
builtinTerms: builtinWords,
maxTerms: request.builtinPolishLimit
)
let correctionPairs = dictionary.localCorrectionPairs()
return LocalASRBiasPayload(
hardHotwords: hardHotwords,
promptBias: promptBias,
corpusContext: promptBias,
polishFragment: polishFragment,
correctionPairs: correctionPairs,
diagnostics: LocalASRBiasDiagnostics(
userTermCount: userTermCount,
builtinTermCount: builtinTermCount,
truncated: truncated,
truncationReason: truncationReason,
selectedSources: selectedSources
)
)
}
// MARK: - Private
private static func preferredLexiconSources(for bundleId: String?) -> Set<String>? {
guard let bundleId, codeEditorBundleIDs.contains(bundleId) else { return nil }
return ["computer_terms"]
}
private static func hardHotwordList(from terms: [String], maxCount: Int) -> [String] {
Array(terms.prefix(maxCount))
}
private static func buildPromptBias(
dictionary: PersonalDictionary,
builtinTerms: [String],
maxCharacters: Int
) -> String {
let userBias = dictionary.asrPromptBias(maxCharacters: maxCharacters)
let userTermsLower = Set(dictionary.effectiveEntries.map { $0.term.lowercased() })
let extras = builtinTerms.filter { !userTermsLower.contains($0.lowercased()) }
guard !extras.isEmpty else { return userBias }
let extraBlock = "常见技术词汇:\(extras.prefix(80).joined(separator: ""))"
if userBias.isEmpty {
return String(extraBlock.prefix(maxCharacters))
}
let combined = userBias + "" + extraBlock
return String(combined.prefix(maxCharacters))
}
private static func buildPolishFragment(
dictionary: PersonalDictionary,
builtinTerms: [String],
maxTerms: Int
) -> String {
let userTermsLower = Set(dictionary.effectiveEntries.map { $0.term.lowercased() })
let extras = builtinTerms
.filter { !userTermsLower.contains($0.lowercased()) }
.prefix(maxTerms)
guard !extras.isEmpty else { return "" }
return "内置技术词汇参考(需原样保留):\(extras.joined(separator: ""))"
}
}
@@ -0,0 +1,56 @@
// LocalASRBiasDiagnosticsStore.swift
// OSGKeyboard · Shared
//
// Persists the most recent local ASR bias diagnostics for settings / debug UI.
import Foundation
public struct LocalASRBiasDiagnosticsSnapshot: Codable, Sendable, Equatable {
public var capturedAt: Date
public var modelId: String?
public var backendLabel: String?
public var diagnostics: LocalASRBiasDiagnostics
public var hotwordCount: Int
public var promptBiasLength: Int
public init(
capturedAt: Date = Date(),
modelId: String? = nil,
backendLabel: String? = nil,
diagnostics: LocalASRBiasDiagnostics,
hotwordCount: Int = 0,
promptBiasLength: Int = 0
) {
self.capturedAt = capturedAt
self.modelId = modelId
self.backendLabel = backendLabel
self.diagnostics = diagnostics
self.hotwordCount = hotwordCount
self.promptBiasLength = promptBiasLength
}
}
public enum LocalASRBiasDiagnosticsStore {
private static let defaultsKey = "mac.localASR.lastBiasDiagnostics"
public static func save(payload: LocalASRBiasPayload, modelId: String?, backendLabel: String?) {
let snapshot = LocalASRBiasDiagnosticsSnapshot(
modelId: modelId,
backendLabel: backendLabel,
diagnostics: payload.diagnostics,
hotwordCount: payload.hardHotwords.count,
promptBiasLength: payload.promptBias?.count ?? 0
)
guard let data = try? JSONEncoder().encode(snapshot) else { return }
UserDefaults.standard.set(data, forKey: defaultsKey)
}
public static func load() -> LocalASRBiasDiagnosticsSnapshot? {
guard let data = UserDefaults.standard.data(forKey: defaultsKey) else { return nil }
return try? JSONDecoder().decode(LocalASRBiasDiagnosticsSnapshot.self, from: data)
}
public static func clear() {
UserDefaults.standard.removeObject(forKey: defaultsKey)
}
}
@@ -0,0 +1,32 @@
// LocalASRInstalledManifestIO.swift
// OSGKeyboard · Shared
import Foundation
public enum LocalASRInstalledManifestIO {
public static func manifestURL(fileManager: FileManager = .default) -> URL {
let appSupport = fileManager.urls(for: .applicationSupportDirectory, in: .userDomainMask).first!
return appSupport
.appendingPathComponent("OSGKeyboard/LocalASRModels/installed-manifest.json")
}
public static func load(defaultModelId: String, fileManager: FileManager = .default) -> LocalASRInstalledManifest {
let url = manifestURL(fileManager: fileManager)
guard let data = try? Data(contentsOf: url),
let manifest = try? JSONDecoder().decode(LocalASRInstalledManifest.self, from: data) else {
return LocalASRInstalledManifest(selectedModelId: defaultModelId)
}
return manifest
}
public static func save(_ manifest: LocalASRInstalledManifest, fileManager: FileManager = .default) throws {
let url = manifestURL(fileManager: fileManager)
try fileManager.createDirectory(at: url.deletingLastPathComponent(), withIntermediateDirectories: true)
let encoder = JSONEncoder()
encoder.outputFormatting = [.prettyPrinted, .sortedKeys]
encoder.dateEncodingStrategy = .iso8601
let data = try encoder.encode(manifest)
try data.write(to: url, options: .atomic)
}
}
@@ -0,0 +1,157 @@
// LocalASRModelDownloadClient.swift
// OSGKeyboard · Shared
//
// URLSession download with byte-level progress and pause/resume (macOS local model installs).
import Foundation
#if os(macOS)
public struct LocalASRDownloadProgressUpdate: Sendable {
public let bytesReceived: Int64
public let bytesTotal: Int64
public var fraction: Double {
guard bytesTotal > 0 else { return 0 }
return min(1, max(0, Double(bytesReceived) / Double(bytesTotal)))
}
}
/// Controls an in-flight URLSession download; supports pause via resume data.
public final class LocalASRModelDownloadController: NSObject, URLSessionDownloadDelegate, @unchecked Sendable {
private let destinationURL: URL
private let onProgress: @Sendable (LocalASRDownloadProgressUpdate) -> Void
private lazy var delegateSession: URLSession = {
URLSession(configuration: .default, delegate: self, delegateQueue: nil)
}()
private var remoteURL: URL?
private var task: URLSessionDownloadTask?
private var completionContinuation: CheckedContinuation<Void, Error>?
private var isPausing = false
private var finished = false
init(
destinationURL: URL,
onProgress: @escaping @Sendable (LocalASRDownloadProgressUpdate) -> Void
) {
self.destinationURL = destinationURL
self.onProgress = onProgress
super.init()
}
/// Runs until the archive is fully written to `destinationURL` (survives pause/resume).
public func download(from remoteURL: URL) async throws {
self.remoteURL = remoteURL
try await withCheckedThrowingContinuation { (continuation: CheckedContinuation<Void, Error>) in
completionContinuation = continuation
startTask(resumeData: nil)
}
}
public func pause() async throws -> Data {
guard task != nil, !finished else {
throw LocalASRModelManagerError.downloadFailed("No active download to pause.")
}
return try await withCheckedThrowingContinuation { (continuation: CheckedContinuation<Data, Error>) in
isPausing = true
task?.cancel(byProducingResumeData: { [weak self] data in
guard let self else { return }
self.isPausing = false
if let data {
continuation.resume(returning: data)
} else {
continuation.resume(throwing: LocalASRModelManagerError.downloadFailed("Pause failed."))
}
})
}
}
/// Continues a paused download; `download(from:)` must still be awaiting.
public func resumeFromPause(_ resumeData: Data) {
finished = false
startTask(resumeData: resumeData)
}
public func cancel() {
finished = true
task?.cancel()
completionContinuation?.resume(throwing: CancellationError())
completionContinuation = nil
delegateSession.invalidateAndCancel()
}
private func startTask(resumeData: Data?) {
if let resumeData {
task = delegateSession.downloadTask(withResumeData: resumeData)
} else if let remoteURL {
task = delegateSession.downloadTask(with: remoteURL)
}
task?.resume()
}
// MARK: - URLSessionDownloadDelegate
public func urlSession(
_ session: URLSession,
downloadTask: URLSessionDownloadTask,
didWriteData bytesWritten: Int64,
totalBytesWritten: Int64,
totalBytesExpectedToWrite: Int64
) {
onProgress(
LocalASRDownloadProgressUpdate(
bytesReceived: totalBytesWritten,
bytesTotal: max(totalBytesExpectedToWrite, 1)
)
)
}
public func urlSession(
_ session: URLSession,
downloadTask: URLSessionDownloadTask,
didFinishDownloadingTo location: URL
) {
guard !finished else { return }
finished = true
do {
let fm = FileManager.default
if fm.fileExists(atPath: destinationURL.path) {
try fm.removeItem(at: destinationURL)
}
try fm.moveItem(at: location, to: destinationURL)
completionContinuation?.resume()
} catch {
completionContinuation?.resume(
throwing: LocalASRModelManagerError.downloadFailed(error.localizedDescription)
)
}
completionContinuation = nil
session.finishTasksAndInvalidate()
}
public func urlSession(_ session: URLSession, task: URLSessionTask, didCompleteWithError error: Error?) {
guard !finished else { return }
if isPausing { return }
if let error {
finished = true
completionContinuation?.resume(
throwing: LocalASRModelManagerError.downloadFailed(error.localizedDescription)
)
completionContinuation = nil
session.finishTasksAndInvalidate()
}
}
}
public enum LocalASRModelDownloadClient {
public static func makeController(
destinationURL: URL,
onProgress: @escaping @Sendable (LocalASRDownloadProgressUpdate) -> Void
) -> LocalASRModelDownloadController {
LocalASRModelDownloadController(destinationURL: destinationURL, onProgress: onProgress)
}
}
#endif
@@ -0,0 +1,138 @@
// LocalASRModelInstallState.swift
// OSGKeyboard · Shared
import Foundation
public enum LocalASRModelInstallState {
public static func rootDirectory(fileManager: FileManager = .default) -> URL {
let appSupport = fileManager.urls(for: .applicationSupportDirectory, in: .userDomainMask).first!
return appSupport.appendingPathComponent("OSGKeyboard/LocalASRModels", isDirectory: true)
}
public static func installDirectory(for relativePath: String, fileManager: FileManager = .default) -> URL {
rootDirectory(fileManager: fileManager).appendingPathComponent(relativePath, isDirectory: true)
}
public static func isInstalled(
_ model: LocalASRModelDefinition,
manualMLXPath: String?,
fileManager: FileManager = .default
) -> Bool {
switch model.installKind {
case .manual:
guard let required = model.requiredRelativeFiles, !required.isEmpty else { return false }
let base = URL(fileURLWithPath: manualMLXPath ?? "", isDirectory: true)
guard fileManager.fileExists(atPath: base.path) else { return false }
return required.allSatisfy { fileManager.fileExists(atPath: base.appendingPathComponent($0).path) }
case .archive:
guard let relative = model.installRelativePath,
let layout = model.layout,
let baseName = model.archiveBaseName else { return false }
let root = installDirectory(for: relative, fileManager: fileManager)
.appendingPathComponent(baseName, isDirectory: true)
return validateArchiveModel(at: root, model: model, layout: layout, fileManager: fileManager)
case .runtime:
return false
}
}
public static func modelRootURL(
_ model: LocalASRModelDefinition,
fileManager: FileManager = .default
) -> URL? {
guard model.installKind == .archive,
let relative = model.installRelativePath,
let baseName = model.archiveBaseName else { return nil }
return installDirectory(for: relative, fileManager: fileManager)
.appendingPathComponent(baseName, isDirectory: true)
}
public static func resolveRuntimeBinary(
runtime: LocalASRRuntimeDefinition,
fileManager: FileManager = .default
) -> URL? {
let root = installDirectory(for: runtime.installRelativePath, fileManager: fileManager)
for candidate in runtime.binaryCandidates {
let direct = root.appendingPathComponent(candidate)
if fileManager.isExecutableFile(atPath: direct.path) {
return direct
}
}
for candidate in runtime.binaryCandidates {
let name = (candidate as NSString).lastPathComponent
if let found = findExecutable(named: name, under: root, fileManager: fileManager) {
return found
}
}
return nil
}
public static func isRuntimeInstalled(
_ runtime: LocalASRRuntimeDefinition,
fileManager: FileManager = .default
) -> Bool {
resolveRuntimeBinary(runtime: runtime, fileManager: fileManager) != nil
}
// MARK: - Private
private static func validateArchiveModel(
at root: URL,
model: LocalASRModelDefinition,
layout: LocalASRModelLayout,
fileManager: FileManager
) -> Bool {
switch model.backend {
case .sherpaQwen3:
guard let conv = layout.convFrontend,
let encoder = layout.encoder,
let decoder = layout.decoder,
let tokenizer = layout.tokenizer else { return false }
return fileManager.fileExists(atPath: root.appendingPathComponent(conv).path)
&& fileManager.fileExists(atPath: root.appendingPathComponent(encoder).path)
&& fileManager.fileExists(atPath: root.appendingPathComponent(decoder).path)
&& fileManager.fileExists(atPath: root.appendingPathComponent(tokenizer, isDirectory: true).path)
case .sherpaSenseVoice:
guard let onnx = layout.senseVoiceModel,
let tokens = layout.tokens else { return false }
return fileManager.fileExists(atPath: root.appendingPathComponent(onnx).path)
&& fileManager.fileExists(atPath: root.appendingPathComponent(tokens).path)
default:
return false
}
}
private static func findExecutable(
named name: String,
under root: URL,
fileManager: FileManager
) -> URL? {
guard let enumerator = fileManager.enumerator(
at: root,
includingPropertiesForKeys: [.isExecutableKey],
options: [.skipsHiddenFiles]
) else { return nil }
for case let url as URL in enumerator {
guard url.lastPathComponent == name else { continue }
if fileManager.isExecutableFile(atPath: url.path) {
return url
}
}
return nil
}
public static func directoryByteCount(at url: URL, fileManager: FileManager = .default) -> Int64 {
guard let enumerator = fileManager.enumerator(
at: url,
includingPropertiesForKeys: [.fileSizeKey],
options: [.skipsHiddenFiles]
) else { return 0 }
var total: Int64 = 0
for case let fileURL as URL in enumerator {
let size = (try? fileURL.resourceValues(forKeys: [.fileSizeKey]).fileSize) ?? 0
total += Int64(size)
}
return total
}
}
@@ -0,0 +1,492 @@
// LocalASRModelManager.swift
// OSGKeyboard · Shared
//
// Installs local ASR model archives and Sherpa runtimes under Application Support.
// Catalog is bundled; installed state is persisted in `installed-manifest.json`.
import Foundation
public struct LocalASRInstalledManifest: Codable, Sendable, Equatable {
public var schemaVersion: Int
public var selectedModelId: String
public var installedModelIDs: [String]
public var installedRuntimeIDs: [String]
public var updatedAt: Date
public init(
schemaVersion: Int = 1,
selectedModelId: String,
installedModelIDs: [String] = [],
installedRuntimeIDs: [String] = [],
updatedAt: Date = Date()
) {
self.schemaVersion = schemaVersion
self.selectedModelId = selectedModelId
self.installedModelIDs = installedModelIDs
self.installedRuntimeIDs = installedRuntimeIDs
self.updatedAt = updatedAt
}
}
public enum LocalASRModelInstallPhase: String, Sendable, Equatable {
case idle
case downloading
case paused
case extracting
case validating
case finalizing
case failed
case completed
}
public struct LocalASRModelInstallProgress: Sendable, Equatable {
public var phase: LocalASRModelInstallPhase
public var fraction: Double
public var message: String
public var bytesReceived: Int64?
public var bytesTotal: Int64?
public var activeItemId: String?
public init(
phase: LocalASRModelInstallPhase,
fraction: Double,
message: String,
bytesReceived: Int64? = nil,
bytesTotal: Int64? = nil,
activeItemId: String? = nil
) {
self.phase = phase
self.fraction = fraction
self.message = message
self.bytesReceived = bytesReceived
self.bytesTotal = bytesTotal
self.activeItemId = activeItemId
}
public static let idle = LocalASRModelInstallProgress(phase: .idle, fraction: 0, message: "")
}
public enum LocalASRModelManagerError: Error, LocalizedError {
case downloadFailed(String)
case extractFailed(String)
case validationFailed(String)
case runtimeMissing
case binaryMissing
public var errorDescription: String? {
switch self {
case .downloadFailed(let detail): return "Download failed: \(detail)"
case .extractFailed(let detail): return "Extract failed: \(detail)"
case .validationFailed(let detail): return "Validation failed: \(detail)"
case .runtimeMissing: return "Sherpa runtime is not installed."
case .binaryMissing: return "Sherpa binary not found in runtime bundle."
}
}
}
public actor LocalASRModelManager {
public static let shared = LocalASRModelManager()
private let fileManager = FileManager.default
private var progress = LocalASRModelInstallProgress.idle
#if os(macOS)
private var activeDownloadController: LocalASRModelDownloadController?
private var pausedResumeData: Data?
#endif
private init() {}
public func currentProgress() -> LocalASRModelInstallProgress {
progress
}
#if os(macOS)
public func pauseDownload() async throws {
guard progress.phase == .downloading, let controller = activeDownloadController else { return }
let resumeData = try await controller.pause()
pausedResumeData = resumeData
progress = LocalASRModelInstallProgress(
phase: .paused,
fraction: progress.fraction,
message: progress.message,
bytesReceived: progress.bytesReceived,
bytesTotal: progress.bytesTotal,
activeItemId: progress.activeItemId
)
}
public func resumeDownload() async throws {
guard progress.phase == .paused,
let resumeData = pausedResumeData,
let controller = activeDownloadController else {
throw LocalASRModelManagerError.downloadFailed("No paused download to resume.")
}
pausedResumeData = nil
progress = LocalASRModelInstallProgress(
phase: .downloading,
fraction: progress.fraction,
message: progress.message,
bytesReceived: progress.bytesReceived,
bytesTotal: progress.bytesTotal,
activeItemId: progress.activeItemId
)
controller.resumeFromPause(resumeData)
}
public func isDownloadPaused() -> Bool {
progress.phase == .paused
}
#endif
public func rootDirectory() -> URL {
let appSupport = fileManager.urls(for: .applicationSupportDirectory, in: .userDomainMask).first!
return appSupport.appendingPathComponent("OSGKeyboard/LocalASRModels", isDirectory: true)
}
public func manifestURL() -> URL {
LocalASRInstalledManifestIO.manifestURL(fileManager: fileManager)
}
public func loadManifest(defaultModelId: String) -> LocalASRInstalledManifest {
LocalASRInstalledManifestIO.load(defaultModelId: defaultModelId, fileManager: fileManager)
}
public func saveManifest(_ manifest: LocalASRInstalledManifest) throws {
try LocalASRInstalledManifestIO.save(manifest, fileManager: fileManager)
}
public func setSelectedModelId(_ modelId: String, catalog: LocalASRCatalogDocument) throws {
var manifest = loadManifest(defaultModelId: catalog.defaultModelId)
manifest.selectedModelId = modelId
manifest.updatedAt = Date()
try saveManifest(manifest)
}
public func installDirectory(for relativePath: String) -> URL {
rootDirectory().appendingPathComponent(relativePath, isDirectory: true)
}
public func isModelInstalled(_ model: LocalASRModelDefinition, manualMLXPath: String?) -> Bool {
LocalASRModelInstallState.isInstalled(model, manualMLXPath: manualMLXPath, fileManager: fileManager)
}
public func isRuntimeInstalled(_ runtime: LocalASRRuntimeDefinition) -> Bool {
LocalASRModelInstallState.isRuntimeInstalled(runtime, fileManager: fileManager)
}
#if os(macOS)
public func resolveRuntimeBinary(runtime: LocalASRRuntimeDefinition) -> URL? {
LocalASRModelInstallState.resolveRuntimeBinary(runtime: runtime, fileManager: fileManager)
}
public func installModel(
_ model: LocalASRModelDefinition,
catalog: LocalASRCatalogDocument
) async throws {
guard model.installKind == .archive,
let relative = model.installRelativePath,
let baseName = model.archiveBaseName,
let sources = model.sources,
!sources.isEmpty else {
throw LocalASRModelManagerError.validationFailed("Model is not downloadable.")
}
progress = LocalASRModelInstallProgress(
phase: .downloading,
fraction: 0.05,
message: model.displayName,
activeItemId: model.id
)
if model.backend == .sherpaQwen3 || model.backend == .sherpaSenseVoice {
try await ensureRuntimeInstalled(catalog: catalog)
}
let sortedSources = sources.sorted { $0.priority < $1.priority }
var lastError: Error?
for source in sortedSources {
do {
try await installArchive(
from: source.url,
installRelativePath: relative,
archiveBaseName: baseName,
layoutModel: model,
itemId: model.id,
displayName: model.displayName
)
var manifest = loadManifest(defaultModelId: catalog.defaultModelId)
if !manifest.installedModelIDs.contains(model.id) {
manifest.installedModelIDs.append(model.id)
}
manifest.updatedAt = Date()
try saveManifest(manifest)
progress = LocalASRModelInstallProgress(
phase: .completed,
fraction: 1,
message: model.displayName,
activeItemId: model.id
)
return
} catch {
lastError = error
}
}
progress = LocalASRModelInstallProgress(
phase: .failed,
fraction: 0,
message: lastError?.localizedDescription ?? "Download failed"
)
throw lastError ?? LocalASRModelManagerError.downloadFailed("All mirrors failed")
}
public func installRuntime(
_ runtime: LocalASRRuntimeDefinition,
catalog: LocalASRCatalogDocument
) async throws {
guard let source = runtime.sources.sorted(by: { $0.priority < $1.priority }).first else {
throw LocalASRModelManagerError.downloadFailed("No runtime source configured.")
}
progress = LocalASRModelInstallProgress(
phase: .downloading,
fraction: 0.05,
message: runtime.displayName,
activeItemId: runtime.id
)
try await installArchive(
from: source.url,
installRelativePath: runtime.installRelativePath,
archiveBaseName: runtime.installRelativePath.split(separator: "/").last.map(String.init) ?? runtime.id,
layoutModel: nil,
expectedBinaryCandidates: runtime.binaryCandidates,
itemId: runtime.id,
displayName: runtime.displayName
)
guard isRuntimeInstalled(runtime) else {
throw LocalASRModelManagerError.binaryMissing
}
var manifest = loadManifest(defaultModelId: catalog.defaultModelId)
if !manifest.installedRuntimeIDs.contains(runtime.id) {
manifest.installedRuntimeIDs.append(runtime.id)
}
manifest.updatedAt = Date()
try saveManifest(manifest)
progress = LocalASRModelInstallProgress(phase: .completed, fraction: 1, message: runtime.displayName)
}
public func modelRootURL(_ model: LocalASRModelDefinition) -> URL? {
LocalASRModelInstallState.modelRootURL(model, fileManager: fileManager)
}
public func installDirectoryURL(for model: LocalASRModelDefinition) -> URL? {
guard let relative = model.installRelativePath else { return nil }
return installDirectory(for: relative)
}
public func deleteModel(
_ model: LocalASRModelDefinition,
catalog: LocalASRCatalogDocument
) throws {
guard model.installKind == .archive, let relative = model.installRelativePath else { return }
let dir = installDirectory(for: relative)
if fileManager.fileExists(atPath: dir.path) {
try fileManager.removeItem(at: dir)
}
var manifest = loadManifest(defaultModelId: catalog.defaultModelId)
manifest.installedModelIDs.removeAll { $0 == model.id }
if manifest.selectedModelId == model.id {
manifest.selectedModelId = catalog.defaultModelId
UserDefaults.standard.set(catalog.defaultModelId, forKey: LocalASRPreferenceKeys.selectedModelId)
}
manifest.updatedAt = Date()
try saveManifest(manifest)
if progress.activeItemId == model.id {
progress = .idle
}
}
public func deleteRuntime(
_ runtime: LocalASRRuntimeDefinition,
catalog: LocalASRCatalogDocument
) throws {
let dir = installDirectory(for: runtime.installRelativePath)
if fileManager.fileExists(atPath: dir.path) {
try fileManager.removeItem(at: dir)
}
var manifest = loadManifest(defaultModelId: catalog.defaultModelId)
manifest.installedRuntimeIDs.removeAll { $0 == runtime.id }
manifest.updatedAt = Date()
try saveManifest(manifest)
}
private func setProgress(_ update: LocalASRModelInstallProgress) {
progress = update
}
func updateDownloadProgress(
itemId: String,
displayName: String,
update: LocalASRDownloadProgressUpdate
) {
reportDownloadProgress(itemId: itemId, displayName: displayName, update: update)
}
private func reportDownloadProgress(
itemId: String,
displayName: String,
update: LocalASRDownloadProgressUpdate
) {
// Download phase occupies 10%55% of the overall install bar.
let mapped = 0.10 + update.fraction * 0.45
progress = LocalASRModelInstallProgress(
phase: .downloading,
fraction: mapped,
message: displayName,
bytesReceived: update.bytesReceived,
bytesTotal: update.bytesTotal,
activeItemId: itemId
)
}
#endif
// MARK: - Private
#if os(macOS)
private func installArchive(
from urlString: String,
installRelativePath: String,
archiveBaseName: String,
layoutModel: LocalASRModelDefinition?,
expectedBinaryCandidates: [String]? = nil,
itemId: String,
displayName: String
) async throws {
guard let remoteURL = URL(string: urlString) else {
throw LocalASRModelManagerError.downloadFailed("Invalid URL")
}
let stagingRoot = rootDirectory().appendingPathComponent("staging/\(UUID().uuidString)", isDirectory: true)
let destinationParent = installDirectory(for: installRelativePath)
try fileManager.createDirectory(at: stagingRoot, withIntermediateDirectories: true)
defer { try? fileManager.removeItem(at: stagingRoot) }
let archiveURL = stagingRoot.appendingPathComponent(remoteURL.lastPathComponent)
progress = LocalASRModelInstallProgress(
phase: .downloading,
fraction: 0.10,
message: displayName,
activeItemId: itemId
)
do {
let controller = LocalASRModelDownloadClient.makeController(destinationURL: archiveURL) { update in
Task {
await LocalASRModelManager.shared.updateDownloadProgress(
itemId: itemId,
displayName: displayName,
update: update
)
}
}
activeDownloadController = controller
try await controller.download(from: remoteURL)
activeDownloadController = nil
pausedResumeData = nil
} catch {
activeDownloadController = nil
pausedResumeData = nil
throw LocalASRModelManagerError.downloadFailed(error.localizedDescription)
}
progress = LocalASRModelInstallProgress(
phase: .extracting,
fraction: 0.58,
message: displayName,
activeItemId: itemId
)
try fileManager.createDirectory(at: destinationParent, withIntermediateDirectories: true)
let extractOK = try await extractTarBz2(archiveURL: archiveURL, destination: destinationParent)
guard extractOK else {
throw LocalASRModelManagerError.extractFailed("tar extraction failed")
}
progress = LocalASRModelInstallProgress(
phase: .validating,
fraction: 0.82,
message: displayName,
activeItemId: itemId
)
if let layoutModel {
guard LocalASRModelInstallState.isInstalled(
layoutModel,
manualMLXPath: nil,
fileManager: fileManager
) else {
throw LocalASRModelManagerError.validationFailed("Required model files missing after extract.")
}
}
if let expectedBinaryCandidates {
let runtimeRoot = destinationParent
let found = expectedBinaryCandidates.contains { candidate in
let direct = runtimeRoot.appendingPathComponent(candidate)
if fileManager.isExecutableFile(atPath: direct.path) { return true }
let name = (candidate as NSString).lastPathComponent
return findExecutable(named: name, under: runtimeRoot) != nil
}
guard found else {
throw LocalASRModelManagerError.binaryMissing
}
}
progress = LocalASRModelInstallProgress(
phase: .finalizing,
fraction: 0.95,
message: displayName,
activeItemId: itemId
)
try? fileManager.removeItem(at: archiveURL)
}
public func ensureRuntimeInstalled(catalog: LocalASRCatalogDocument) async throws {
guard let runtime = LocalASRModelCatalog.runtime(
for: LocalASRModelCatalog.currentRuntimePlatform(),
in: catalog
) else {
throw LocalASRModelManagerError.runtimeMissing
}
if isRuntimeInstalled(runtime) { return }
try await installRuntime(runtime, catalog: catalog)
}
private func findExecutable(named name: String, under root: URL) -> URL? {
guard let enumerator = fileManager.enumerator(
at: root,
includingPropertiesForKeys: [.isExecutableKey],
options: [.skipsHiddenFiles]
) else { return nil }
for case let url as URL in enumerator {
guard url.lastPathComponent == name else { continue }
if fileManager.isExecutableFile(atPath: url.path) {
return url
}
}
return nil
}
private func extractTarBz2(archiveURL: URL, destination: URL) async throws -> Bool {
try await withCheckedThrowingContinuation { continuation in
let process = Process()
process.executableURL = URL(fileURLWithPath: "/usr/bin/tar")
process.arguments = ["-xjf", archiveURL.path, "-C", destination.path]
process.standardOutput = FileHandle.nullDevice
process.standardError = FileHandle.nullDevice
process.terminationHandler = { proc in
continuation.resume(returning: proc.terminationStatus == 0)
}
do {
try process.run()
} catch {
continuation.resume(throwing: LocalASRModelManagerError.extractFailed(error.localizedDescription))
}
}
}
#endif
}
@@ -0,0 +1,8 @@
// LocalASRPreferenceKeys.swift
// OSGKeyboard · Shared
import Foundation
enum LocalASRPreferenceKeys {
static let selectedModelId = "mac.localASR.selectedModelId"
}
@@ -0,0 +1,68 @@
// LocalASRTranscriptCorrector.swift
// OSGKeyboard · Shared
//
// Deterministic alias canonical term replacement between raw ASR output
// and the LLM polish step. Only applies whole-phrase matches.
import Foundation
public enum LocalASRTranscriptCorrector {
/// Applies high-confidence alias replacements (longest match first).
public static func apply(
_ text: String,
pairs: [LocalASRCorrectionPair]
) -> String {
guard !text.isEmpty, !pairs.isEmpty else { return text }
let sorted = pairs.sorted { lhs, rhs in
if lhs.alias.count != rhs.alias.count {
return lhs.alias.count > rhs.alias.count
}
return lhs.alias.localizedCaseInsensitiveCompare(rhs.alias) == .orderedAscending
}
var result = text
for pair in sorted {
result = replaceWholeMatches(
in: result,
alias: pair.alias,
term: pair.term
)
}
return result
}
// MARK: - Private
private static func replaceWholeMatches(
in text: String,
alias: String,
term: String
) -> String {
guard !alias.isEmpty, alias != term else { return text }
if alias.unicodeScalars.allSatisfy({ $0.isASCII }) {
return replaceASCIIWord(in: text, alias: alias, term: term)
}
return text.replacingOccurrences(of: alias, with: term)
}
private static func replaceASCIIWord(
in text: String,
alias: String,
term: String
) -> String {
let escaped = NSRegularExpression.escapedPattern(for: alias)
let pattern = "(?i)(?<![A-Za-z0-9_])\(escaped)(?![A-Za-z0-9_])"
guard let regex = try? NSRegularExpression(pattern: pattern) else {
return text
}
let range = NSRange(text.startIndex..<text.endIndex, in: text)
return regex.stringByReplacingMatches(
in: text,
range: range,
withTemplate: term
)
}
}
@@ -252,7 +252,10 @@ public actor PolishingService {
providerId: String
) -> String {
let dictionary = store.personalDictionary
let dictionaryBlock = dictionary.promptFragment()
let dictionaryBlock = Self.mergedDictionaryBlock(
dictionary: dictionary,
supplement: context.dictionarySupplement
)
let contextGuideline = context.appContext.polishGuideline
let intensityGuideline = context.intensity.promptGuideline
let contract = Self.globalOutputContract(useChinese: shouldUseChineseGuidance(providerId: providerId))
@@ -321,6 +324,17 @@ public actor PolishingService {
}
}
internal static func mergedDictionaryBlock(
dictionary: PersonalDictionary,
supplement: String?
) -> String {
let base = dictionary.promptFragment()
let extra = supplement?.trimmingCharacters(in: .whitespacesAndNewlines) ?? ""
if base.isEmpty { return extra }
if extra.isEmpty { return base }
return base + "\n" + extra
}
private func shouldUseChineseGuidance(providerId: String) -> Bool {
switch providerId {
case "zhipu", "moonshot", "qwen", "deepseek":
+42 -2
View File
@@ -157,8 +157,8 @@
"mac.settings.recognition" = "RECOGNITION METHOD";
"mac.settings.cloudEngine" = "Cloud Engine & AI Refinement";
"mac.settings.cloudEngineDesc" = "Premium transcription via your provider's API, plus AI grammar and style polishing.";
"mac.settings.localEngine" = "Local Recognition (Qwen3-ASR)";
"mac.settings.localEngineDesc" = "On-device ASR with Qwen3-ASR 1.7B (MLX). High privacy, zero latency.";
"mac.settings.localEngine" = "Local Recognition";
"mac.settings.localEngineDesc" = "On-device transcription with a local model. High privacy, zero latency.";
"mac.settings.localSpeechFallback" = "Local Recognition (Apple Speech)";
"mac.settings.localSpeechFallbackDesc" = "On-device Apple Speech when the Qwen3 model is not installed.";
"mac.settings.about" = "About";
@@ -174,6 +174,9 @@
"mac.settings.qwen3ModelDesc" = "Folder with config.json, model.safetensors, vocab.json, and merges.txt.";
"mac.settings.qwen3Browse" = "Choose folder…";
"mac.settings.qwen3Missing" = "Qwen3 model not found — using Apple Speech for now.";
"mac.settings.mlxModelMissing" = "Select a Qwen3 MLX model folder below, or choose another installed model.";
"mac.settings.selectedModelMissing" = "%@ is not installed — using Apple Speech for now.";
"mac.settings.localModelFallbackApple" = "No local model is ready — using Apple Speech for now.";
"mac.settings.accessibility" = "Accessibility";
"mac.settings.accessibilityDesc" = "Required for global shortcut and auto-paste.";
"mac.settings.openAccessibility" = "Open System Settings";
@@ -188,6 +191,43 @@
"mac.error.qwen3ModelMissing" = "Qwen3-ASR model not installed";
"mac.error.qwen3LoadFailed" = "Failed to load Qwen3 model: %@";
"mac.error.qwen3InferenceFailed" = "Qwen3 transcription failed: %@";
"mac.localASR.models" = "Local ASR Models";
"mac.localASR.modelsDesc" = "Sherpa models download directly. For MLX Qwen3, drop your converted weights into the folder opened by “Open folder”. All three share one storage directory.";
"mac.localASR.download" = "Download";
"mac.localASR.selectFolder" = "Choose folder";
"mac.localASR.openFolder" = "Open folder";
"mac.localASR.pause" = "Pause";
"mac.localASR.resume" = "Resume";
"mac.localASR.needsFolder" = "Folder required";
"mac.localASR.installDone" = "Install completed.";
"mac.localASR.installed" = "Installed";
"mac.localASR.notInstalled" = "Not installed";
"mac.localASR.hotwordsYes" = "Hotwords";
"mac.localASR.hotwordsNo" = "No hotwords";
"mac.localASR.catalogMissing" = "Local ASR catalog is missing from the app bundle.";
"mac.localASR.diagnostics" = "Last Bias Diagnostics";
"mac.localASR.diagnosticsDesc" = "Captured after your most recent local dictation.";
"mac.localASR.diagEmpty" = "No local dictation yet.";
"mac.localASR.diagBackend" = "Backend";
"mac.localASR.diagUserTerms" = "User terms";
"mac.localASR.diagBuiltinTerms" = "Builtin terms";
"mac.localASR.diagHotwords" = "Hotwords sent";
"mac.localASR.diagPrompt" = "Prompt chars";
"mac.localASR.diagTruncated" = "Prompt truncated";
"mac.localASR.delete" = "Delete";
"mac.localASR.deleteDone" = "Model deleted.";
"mac.localASR.redownload" = "Re-download";
"mac.localASR.revealInFinder" = "Reveal in Finder";
"mac.localASR.openStorage" = "Open model storage folder";
"mac.localASR.runtime" = "Sherpa Runtime";
"mac.localASR.runtimeDesc" = "Required for Sherpa Qwen3 and SenseVoice models. Installed automatically with those models.";
"mac.localASR.phase.downloading" = "Downloading";
"mac.localASR.phase.paused" = "Paused";
"mac.localASR.phase.extracting" = "Extracting";
"mac.localASR.phase.validating" = "Validating";
"mac.localASR.phase.finalizing" = "Finalizing";
"mac.localASR.phase.failed" = "Failed";
"mac.localASR.phase.completed" = "Completed";
"mac.error.accessibilityRequired" = "Enable Accessibility for OSGKeyboard in System Settings";
"mac.foregroundApp" = "Front app: %@";
"mac.sync.settingsTitle" = "iCloud Sync";
+42 -2
View File
@@ -157,8 +157,8 @@
"mac.settings.recognition" = "识别方式";
"mac.settings.cloudEngine" = "云端引擎与 AI 润色";
"mac.settings.cloudEngineDesc" = "通过服务商 API 进行高质量转写,并自动润色语法与风格。";
"mac.settings.localEngine" = "本地识别Qwen3-ASR";
"mac.settings.localEngineDesc" = "使用 Qwen3-ASR 1.7BMLX)本地转写,高隐私、低延迟。";
"mac.settings.localEngine" = "本地识别";
"mac.settings.localEngineDesc" = "在本机使用本地模型转写,高隐私、低延迟。";
"mac.settings.localSpeechFallback" = "本地识别(Apple Speech";
"mac.settings.localSpeechFallbackDesc" = "未安装 Qwen3 模型时,使用 Apple 本地语音识别。";
"mac.settings.about" = "关于";
@@ -174,6 +174,9 @@
"mac.settings.qwen3ModelDesc" = "需包含 config.json、model.safetensors、vocab.json 与 merges.txt。";
"mac.settings.qwen3Browse" = "选择文件夹…";
"mac.settings.qwen3Missing" = "未找到 Qwen3 模型,暂时使用 Apple Speech。";
"mac.settings.mlxModelMissing" = "请在下方选择 Qwen3 MLX 模型目录,或改用其他已安装的模型。";
"mac.settings.selectedModelMissing" = "「%@」尚未安装,暂时使用 Apple Speech。";
"mac.settings.localModelFallbackApple" = "没有可用的本地模型,暂时使用 Apple Speech。";
"mac.settings.accessibility" = "辅助功能";
"mac.settings.accessibilityDesc" = "全局快捷键与自动粘贴需要此权限。";
"mac.settings.openAccessibility" = "打开系统设置";
@@ -188,6 +191,43 @@
"mac.error.qwen3ModelMissing" = "未安装 Qwen3-ASR 模型";
"mac.error.qwen3LoadFailed" = "Qwen3 模型加载失败:%@";
"mac.error.qwen3InferenceFailed" = "Qwen3 转写失败:%@";
"mac.localASR.models" = "本地 ASR 模型";
"mac.localASR.modelsDesc" = "Sherpa 模型可直接下载;MLX Qwen3 请将转换好的权重放入「打开目录」指向的文件夹。三个模型共用同一存储目录。";
"mac.localASR.download" = "下载";
"mac.localASR.selectFolder" = "选择目录";
"mac.localASR.openFolder" = "打开目录";
"mac.localASR.pause" = "暂停";
"mac.localASR.resume" = "继续";
"mac.localASR.needsFolder" = "需选择目录";
"mac.localASR.installDone" = "安装完成。";
"mac.localASR.installed" = "已安装";
"mac.localASR.notInstalled" = "未安装";
"mac.localASR.hotwordsYes" = "支持热词";
"mac.localASR.hotwordsNo" = "无热词";
"mac.localASR.catalogMissing" = "应用包内缺少本地 ASR 模型目录。";
"mac.localASR.diagnostics" = "最近一次词库诊断";
"mac.localASR.diagnosticsDesc" = "在上一轮本地听写后记录。";
"mac.localASR.diagEmpty" = "尚无本地听写记录。";
"mac.localASR.diagBackend" = "后端";
"mac.localASR.diagUserTerms" = "用户词条";
"mac.localASR.diagBuiltinTerms" = "内置词条";
"mac.localASR.diagHotwords" = "热词数量";
"mac.localASR.diagPrompt" = "Prompt 字符";
"mac.localASR.diagTruncated" = "Prompt 已截断";
"mac.localASR.delete" = "删除";
"mac.localASR.deleteDone" = "模型已删除。";
"mac.localASR.redownload" = "重新下载";
"mac.localASR.revealInFinder" = "在 Finder 中显示";
"mac.localASR.openStorage" = "打开模型存储目录";
"mac.localASR.runtime" = "Sherpa 运行时";
"mac.localASR.runtimeDesc" = "Sherpa Qwen3 与 SenseVoice 模型需要此运行时;下载上述模型时会自动安装。";
"mac.localASR.phase.downloading" = "下载中";
"mac.localASR.phase.paused" = "已暂停";
"mac.localASR.phase.extracting" = "解压中";
"mac.localASR.phase.validating" = "校验中";
"mac.localASR.phase.finalizing" = "完成安装";
"mac.localASR.phase.failed" = "失败";
"mac.localASR.phase.completed" = "已完成";
"mac.error.accessibilityRequired" = "请在系统设置中为 OSGKeyboard 启用辅助功能";
"mac.foregroundApp" = "前台应用:%@";
"mac.sync.settingsTitle" = "iCloud 同步";
@@ -141,6 +141,7 @@ final class FlowSessionBridgeTests: XCTestCase {
func testDarwinNotificationPostsWithoutCrashing() {
FlowSessionDarwin.postSessionChanged()
FlowSessionDarwin.postCommandChanged()
FlowSessionDarwin.postHostReadyChanged()
}
@@ -183,4 +184,112 @@ final class FlowSessionBridgeTests: XCTestCase {
XCTAssertFalse(defaults.bool(forKey: FlowSessionKeys.flowHostReady))
XCTAssertFalse(FlowSessionBridge.isHostReady(defaults: defaults))
}
func testFlowCommandRoundTrip() {
let defaults = makeDefaults()
let sessionId = UUID()
let utteranceId = UUID()
let command = FlowCommand(
sessionId: sessionId,
utteranceId: utteranceId,
commandSeq: 42,
action: .startRecording,
localeId: "zh-Hans",
createdAt: 123
)
FlowSessionBridge.writeCommand(command, defaults: defaults)
XCTAssertEqual(FlowSessionBridge.latestCommand(defaults: defaults), command)
}
func testFlowResultRoundTripPreservesUtteranceIdentity() {
let defaults = makeDefaults()
let sessionId = UUID()
let utteranceId = UUID()
let result = FlowResult(
sessionId: sessionId,
utteranceId: utteranceId,
commandSeq: 43,
status: .final,
text: "hello",
warning: "raw fallback",
createdAt: 124
)
FlowSessionBridge.writeResult(result, defaults: defaults)
XCTAssertEqual(FlowSessionBridge.latestResult(defaults: defaults), result)
FlowSessionBridge.clearResult(defaults: defaults)
XCTAssertNil(FlowSessionBridge.latestResult(defaults: defaults))
}
func testFlowAckRoundTrip() {
let defaults = makeDefaults()
let ack = FlowAck(
sessionId: UUID(),
utteranceId: UUID(),
commandSeq: 44,
consumedAt: 125
)
FlowSessionBridge.writeAck(ack, defaults: defaults)
XCTAssertEqual(FlowSessionBridge.latestAck(defaults: defaults), ack)
}
func testReadySnapshotDrivesHostReady() {
let defaults = makeDefaults()
let sessionId = UUID()
let now = Date().timeIntervalSince1970
FlowSessionBridge.markSessionActive(duration: 60, sessionId: sessionId, defaults: defaults)
let snapshot = FlowReadySnapshot(
sessionId: sessionId,
ready: true,
reason: .ready,
heartbeatAt: now,
readyAt: now,
audioProofAt: now,
engineMode: "local",
localeId: "zh-Hans",
sessionExpiresAt: now + 60
)
FlowSessionBridge.writeReadySnapshot(snapshot, defaults: defaults)
XCTAssertEqual(FlowSessionBridge.readySnapshot(defaults: defaults), snapshot)
XCTAssertTrue(FlowSessionBridge.isHostReady(defaults: defaults))
}
func testClearFlowStateRemovesProtocolPayloads() {
let defaults = makeDefaults()
let sessionId = UUID()
let utteranceId = UUID()
FlowSessionBridge.writeCommand(
FlowCommand(
sessionId: sessionId,
utteranceId: utteranceId,
commandSeq: 1,
action: .startRecording,
localeId: "en-US"
),
defaults: defaults
)
FlowSessionBridge.writeResult(
FlowResult(
sessionId: sessionId,
utteranceId: utteranceId,
commandSeq: 1,
status: .partial,
text: "hello"
),
defaults: defaults
)
FlowSessionBridge.clearFlowState(defaults: defaults)
XCTAssertNil(FlowSessionBridge.latestCommand(defaults: defaults))
XCTAssertNil(FlowSessionBridge.latestResult(defaults: defaults))
XCTAssertNil(FlowSessionBridge.readySnapshot(defaults: defaults))
}
}
@@ -0,0 +1,124 @@
// LocalASRBiasAdapterTests.swift
// OSGKeyboardTests
import XCTest
@testable import OSGKeyboardShared
final class LocalASRBiasAdapterTests: XCTestCase {
private func makeFixtureLexicon() throws -> BuiltinLexiconIndex {
let dir = FileManager.default.temporaryDirectory
.appendingPathComponent("osg-phrases-\(UUID().uuidString)", isDirectory: true)
try FileManager.default.createDirectory(at: dir, withIntermediateDirectories: true)
let url = dir.appendingPathComponent("phrases.tsv")
let tsv = """
word\tpinyin\tsource\tweight
SwiftUI\tswift ui\tcomputer_terms\t5
Kubernetes\tku bo ne si\tcomputer_terms\t5
\tyi zhi xing\tcomputer_terms\t5
"""
try tsv.write(to: url, atomically: true, encoding: .utf8)
addTeardownBlock {
try? FileManager.default.removeItem(at: dir)
}
return BuiltinLexiconIndex(fixtureURL: url)
}
func testAdaptBuildsPromptBiasForQwen3MLX() throws {
let lexicon = try makeFixtureLexicon()
var dict = PersonalDictionary.empty
_ = dict.upsertManual(term: "Cursor")
dict.updateAliases(for: dict.entries[0].id, aliases: ["cursor"])
let payload = LocalASRBiasAdapter.adapt(
LocalASRBiasRequest(
dictionary: dict,
locale: Locale(identifier: "zh-CN"),
capabilities: .qwen3MLX
),
lexicon: lexicon
)
XCTAssertNotNil(payload.promptBias)
XCTAssertTrue(payload.promptBias?.contains("Cursor") == true)
XCTAssertTrue(payload.promptBias?.contains("SwiftUI") == true)
XCTAssertEqual(payload.diagnostics.userTermCount, 2) // OSGKeyboard system + Cursor
XCTAssertGreaterThan(payload.diagnostics.builtinTermCount, 0)
}
func testAdaptProducesPolishFragmentWithoutUserDuplicates() throws {
let lexicon = try makeFixtureLexicon()
var dict = PersonalDictionary.empty
_ = dict.upsertManual(term: "SwiftUI")
let payload = LocalASRBiasAdapter.adapt(
LocalASRBiasRequest(
dictionary: dict,
locale: Locale(identifier: "zh-CN"),
capabilities: .qwen3MLX
),
lexicon: lexicon
)
XCTAssertFalse(payload.polishFragment.contains("SwiftUI"))
XCTAssertTrue(payload.polishFragment.contains("Kubernetes"))
}
func testCorrectionPairsFromAliases() {
var dict = PersonalDictionary.empty
_ = dict.upsertManual(term: "Kubernetes")
dict.updateAliases(for: dict.entries[0].id, aliases: ["k8s"])
let payload = LocalASRBiasAdapter.adapt(
LocalASRBiasRequest(
dictionary: dict,
locale: Locale(identifier: "zh-CN"),
capabilities: .qwen3MLX
),
lexicon: BuiltinLexiconIndex.shared
)
XCTAssertEqual(payload.correctionPairs.count, 1)
XCTAssertEqual(payload.correctionPairs[0].alias, "k8s")
XCTAssertEqual(payload.correctionPairs[0].term, "Kubernetes")
}
func testTranscriptCorrectorReplacesASCIIAlias() {
let pairs = [LocalASRCorrectionPair(alias: "k8s", term: "Kubernetes")]
let result = LocalASRTranscriptCorrector.apply(
"部署 k8s 集群",
pairs: pairs
)
XCTAssertEqual(result, "部署 Kubernetes 集群")
}
func testTranscriptCorrectorSkipsPartialASCIIMatch() {
let pairs = [LocalASRCorrectionPair(alias: "k8s", term: "Kubernetes")]
let result = LocalASRTranscriptCorrector.apply(
"xk8s集群",
pairs: pairs
)
XCTAssertEqual(result, "xk8s集群")
}
func testBuiltinLexiconParsesTSV() {
let terms = BuiltinLexiconIndex.parseTSV(
"word\tpinyin\tsource\tweight\nFoo\tfoo\tcomputer_terms\t5\n"
)
XCTAssertEqual(terms.count, 1)
XCTAssertEqual(terms[0].word, "Foo")
XCTAssertEqual(terms[0].weight, 5)
}
func testPolishingServiceMergesDictionarySupplement() {
let dict = PersonalDictionary(entries: [
PersonalDictionary.Entry(term: "Cursor", category: .productName, source: .manual),
])
let merged = PolishingService.mergedDictionaryBlock(
dictionary: dict,
supplement: "内置技术词汇参考:SwiftUI"
)
XCTAssertTrue(merged.contains("Cursor"))
XCTAssertTrue(merged.contains("SwiftUI"))
}
}
@@ -0,0 +1,88 @@
// LocalASRModelCatalogTests.swift
// OSGKeyboardTests
import XCTest
@testable import OSGKeyboardShared
final class LocalASRModelCatalogTests: XCTestCase {
func testBundledCatalogLoads() throws {
let catalog = try LocalASRModelCatalog.loadBundled()
XCTAssertEqual(catalog.schemaVersion, 1)
XCTAssertFalse(catalog.models.isEmpty)
XCTAssertTrue(catalog.models.contains { $0.id == "qwen3-mlx-1.7b" })
XCTAssertTrue(catalog.models.contains { $0.id == "sherpa-qwen3-0.6b-int8" })
}
func testCapabilitiesForSherpaQwen3() throws {
let catalog = try LocalASRModelCatalog.loadBundled()
let model = try XCTUnwrap(LocalASRModelCatalog.model("sherpa-qwen3-0.6b-int8", in: catalog))
let caps = LocalASRModelCatalog.capabilities(for: model)
XCTAssertEqual(caps.hotwordMode, .recognizerScoped)
XCTAssertTrue(model.supportsHotwords)
}
func testManifestRoundTrip() throws {
let manifest = LocalASRInstalledManifest(
selectedModelId: "sherpa-qwen3-0.6b-int8",
installedModelIDs: ["sherpa-qwen3-0.6b-int8"]
)
let url = FileManager.default.temporaryDirectory
.appendingPathComponent("manifest-\(UUID().uuidString).json")
defer { try? FileManager.default.removeItem(at: url) }
let encoder = JSONEncoder()
encoder.dateEncodingStrategy = .iso8601
try encoder.encode(manifest).write(to: url)
let decoder = JSONDecoder()
decoder.dateDecodingStrategy = .iso8601
let loaded = try decoder.decode(LocalASRInstalledManifest.self, from: Data(contentsOf: url))
XCTAssertEqual(loaded.selectedModelId, manifest.selectedModelId)
XCTAssertEqual(loaded.installedModelIDs, manifest.installedModelIDs)
}
func testBiasDiagnosticsStoreRoundTrip() {
LocalASRBiasDiagnosticsStore.clear()
let payload = LocalASRBiasPayload(
hardHotwords: ["Cursor"],
promptBias: "test",
corpusContext: nil,
polishFragment: "fragment",
correctionPairs: [],
diagnostics: LocalASRBiasDiagnostics(userTermCount: 2, builtinTermCount: 3)
)
LocalASRBiasDiagnosticsStore.save(
payload: payload,
modelId: "qwen3-mlx-1.7b",
backendLabel: "MLX"
)
let snapshot = LocalASRBiasDiagnosticsStore.load()
XCTAssertEqual(snapshot?.modelId, "qwen3-mlx-1.7b")
XCTAssertEqual(snapshot?.diagnostics.userTermCount, 2)
XCTAssertEqual(snapshot?.hotwordCount, 1)
LocalASRBiasDiagnosticsStore.clear()
}
func testSherpaAdapterProducesHardHotwords() throws {
let fixtureURL = FileManager.default.temporaryDirectory
.appendingPathComponent("phrases-\(UUID().uuidString).tsv")
try "word\tpinyin\tsource\tweight\nSwiftUI\tswift ui\tcomputer_terms\t5\n"
.write(to: fixtureURL, atomically: true, encoding: .utf8)
defer { try? FileManager.default.removeItem(at: fixtureURL) }
var dict = PersonalDictionary.empty
_ = dict.upsertManual(term: "Kubernetes")
let payload = LocalASRBiasAdapter.adapt(
LocalASRBiasRequest(
dictionary: dict,
locale: Locale(identifier: "zh-CN"),
capabilities: .sherpaQwen3
),
lexicon: BuiltinLexiconIndex(fixtureURL: fixtureURL)
)
XCTAssertFalse(payload.hardHotwords.isEmpty)
XCTAssertTrue(payload.hardHotwords.contains("Kubernetes"))
}
}
+585
View File
@@ -0,0 +1,585 @@
# OSGKeyboard 本地 ASR 技术架构
> **文档状态**:架构规划(非实现规格)
> **适用范围**:macOS 本地听写;与 iOS 键盘扩展、云 ASR 路径的关系见各节说明。
> **核心结论**:短期不换主模型,优先打通 **词库感知管道**;中期用 POC 验证 **Sherpa Qwen3 hard hotwords** 是否值得成为热词主线。
---
## 1. Executive Summary
OSGKeyboard 的本地 ASR 竞争力不来自单一模型,而来自:
1. **用户 PersonalDictionary**term / aliases / iCloud
2. **内置技术词库**`phrases.tsv` ≈ 1 万词,iOS 已用于 Apple CLM
3. **分层 bias**:ASR 偏置 → 后处理纠错 → Polish 保真
4. **可替换的 Local ASR Provider**Qwen3 MLX 主线,Sherpa / SenseVoice / Apple Speech 对照)
当前最大缺口:**macOS 本地路径未消费任何词库**;云路径已通过 `PersonalDictionary+ASRBias` 完整接线。
推荐路线:
| 阶段 | 动作 |
|------|------|
| **短期** | 保留 Qwen3 MLX;实现 `LocalASRBiasAdapter`;接 soft prompt + polish + aliases 后处理 |
| **中期** | ModelScope 优先的本地模型 catalogSherpa Qwen3 hotwords POC |
| **长期** | 按评测数据决定是否新增默认 provider 或保留 Qwen3 MLX |
---
## 2. 背景与问题定义
### 2.1 为什么本地 ASR 不能只讨论模型
语音输入的「专有名词准确率」由多层共同决定:
- **ASR 层**:听出 `Claude``SwiftUI``Qwen3-ASR`
- **后处理层**`克劳德``Claude`
- **润色层**:保留品牌名、变量名,不擅自改写
闭源产品(Typeless 等)常把词典效果归因于云端 ASR;开源竞品(OpenLess、Typeflux、SayIt)表明:**词典必须按 backend 能力分层注入**,不能假设「一个 hotwords 数组走天下」。
### 2.2 OSG 相对竞品的结构性优势
| 能力 | OSGKeyboard | 典型开源竞品 |
|------|-------------|----------------|
| 用户词库 | `PersonalDictionary`term + aliases + category + iCloud | 多为 phrase-only |
| 内置领域词库 | ~10k `phrases.tsv` + iOS CLM | OpenLess preset ~20 词;SayIt server hotwords.txt ~30 词 |
| 云 ASR bias | 智谱 / 阿里 vocabulary / Whisper prompt | 单云或单 provider |
| iOS 本地 CLM | `SFCustomLanguageModelData` | macOS 路径未等价 |
### 2.3 设计目标
- 离线、隐私友好的 macOS 本地听写
- 复用 `PersonalDictionary``phrases.tsv` **源数据**(非 iOS `.bin` 直用)
- Provider 可替换;能力矩阵诚实声明(尤其热词模式)
- 模型下载可管理(**ModelScope 优先**HF / GitHub 备用)
- 可评测、可灰度、可回退
### 2.4 非目标
- 不立即将主路径切到 FunASR Python server 或 Sherpa
- 不把 1 万词全量塞入 ASR prompt
- 不把 Polish 当作唯一纠错层
- 不承诺未 POC 验证的模型效果
- 第一期不强制实现 Typeflux 式「自动词库学习」(仅作可选实验设计)
---
## 3. 当前架构(代码事实)
### 3.1 端到端数据流
```mermaid
flowchart LR
subgraph macOS["macOS"]
Rec["MacAudioRecorder"]
Pipe["MacDictationPipeline"]
Local["MacLocalASRService"]
Cloud["CloudASRClient"]
Polish["PolishingService"]
Insert["MacTextInsertionService"]
end
Rec --> Pipe
Pipe -->|engineMode local| Local
Pipe -->|engineMode cloud| Cloud
Local --> Polish
Cloud --> Polish
Polish --> Insert
```
### 3.2 云路径 vs 本地路径
| 环节 | 云 ASR | 本地 ASR(当前) |
|------|--------|------------------|
| 入口 | `MacDictationPipeline.run` | 同左 |
| ASR | `CloudASRClientFactory` + `dictionary: store.personalDictionary` | `MacLocalASRService.transcribe(samples, locale)` **无 dictionary** |
| 词库 bias | `PersonalDictionary+ASRBias`(按 provider | **无** |
| 润色 | `PolishingService` + `promptFragment()` | 同左(仅用词典做 polish,不经 ASR) |
| 默认模型 | 用户所选云 provider | Qwen3 MLX 1.7B;缺权重 → Apple Speech |
关键代码:
- [`OSGKeyboardMac/MacDictationPipeline.swift`](../OSGKeyboardMac/MacDictationPipeline.swift) — 本地分支未传 `personalDictionary`
- [`OSGKeyboardMac/MacLocalASRService.swift`](../OSGKeyboardMac/MacLocalASRService.swift) — Qwen3 / Apple Speech 二选一
- [`OSGKeyboardShared/Models/PersonalDictionary+ASRBias.swift`](../OSGKeyboardShared/Models/PersonalDictionary+ASRBias.swift) — 云侧 `asrHotwords` / `asrPromptBias` / 阿里热词表
- [`OSGKeyboardShared/Services/PolishingService.swift`](../OSGKeyboardShared/Services/PolishingService.swift) — 润色层消费 `dictionary.promptFragment()`
### 3.3 iOS 词库资产(macOS 不可直接复用)
- **内置词库**[`OSGKeyboard/Resources/CustomLanguageModel/v1/phrases.tsv`](../OSGKeyboard/Resources/CustomLanguageModel/v1/phrases.tsv)(约 10,301 行,含 `word` / `pinyin` / `source` / `weight`
- **Apple CLM**[`OSGKeyboardShared/Services/CustomLanguageModelManager.swift`](../OSGKeyboardShared/Services/CustomLanguageModelManager.swift) → `SFSpeechLanguageModel.Configuration`,仅 iOS 26+ 中文本地 ASR 路径
- **结论**macOS 需从 TSV + `PersonalDictionary` **重新适配**为 Qwen3 prompt / Sherpa hotwords / polish fragment,不能加载 `OSGKeyboardCLM.bin`
### 3.4 本地模型现状
- 默认路径:`~/Library/Application Support/OSGKeyboard/models/qwen3-asr-1.7b-mlx`
- 校验:`config.json``model.safetensors``vocab.json``merges.txt`
- Qwen3 引擎:[`MacQwen3LocalASR.swift`](../OSGKeyboardMac/MacQwen3LocalASR.swift) / [`MacQwen3ASREngine.swift`](../OSGKeyboardMac/MacQwen3ASREngine.swift) — **无 prompt / hotword 入参**
---
## 4. 开源竞品源码观察
基于 GitHub topic `typeless-alternative` 及关联仓库(2026-03 快照)。
### 4.1 对比总表
| 项目 | 技术栈 | 本地 ASR | 热词 / 词库 | 可借鉴 | 不宜照搬 |
|------|--------|----------|-------------|--------|----------|
| **OpenLess** | Tauri/Rust | Qwen3 C 引擎、Apple Speech、SherpaWin | 火山 `context.hotwords`Whisper promptPolish hotword block**本地 Qwen3 未接词典** | 多 provider + polish 双层 | 本地热词叙事过度乐观 |
| **Typeflux** | Swift/macOS | SenseVoice、FunASR、Qwen3 Sherpa CLI、WhisperKit | `VocabularyStore` cap 500Doubao hotwordsWhisper prompt**Sherpa 本地无热词**;自动项目词 + 编辑后学习 | Swift 原生、词库排序、自动学习(作实验) | Sherpa 仅 CLI 离线,无 hotwords 接线 |
| **SayIt** | Tauri + FastAPI | sherpa-onnx Rust**Qwen3 recognizer 创建时写 hotwords** | 内置主题词 + 自定义;云 Qwen corpusserver `hotwords.txt` | **本地 Qwen3 hard hotwords 实证**;跨 provider `StartOptions.hotwords` | 服务端 vLLM 非 macOS 客户端主线 |
| **VoiceSnap** | Go/Wails | SenseVoice + sherpa-onnx | **无个性化词库** | 离线体验:静音截断、剪贴板保护、填充词过滤 | 无万级词库场景 |
| **OpenBroca** | Electron | Sherpa 等 | Dictionary hotword/replacement | **one-shot `recognize` first**model catalog、sha256、selected model | Electron 栈 |
### 4.2 对 OSG 的启示
1. **学架构,不学「本地已完整支持热词」的 README 叙事**OpenLess 本地 Qwen3 缺口与 OSG 类似)。
2. **SayIt 证明**sherpa-onnx `OfflineQwen3ASRModelConfig.hotwords` 可在 recognizer 创建时注入;热词变化需 **重建 recognizer**(缓存 key 含 hotwords 字符串)。
3. **Typeflux 证明**`activeTerms()` 上限 500 + 动态排序;但 Sherpa Qwen3/SenseVoice 命令行路径**未传** vocabulary prompt。
4. **OpenBroca 证明**:runtime 不得静默选「目录里第一个模型」;须 `selectedModelId` + manifest。
---
## 5. 目标架构
### 5.1 管道总览
```mermaid
flowchart TD
audioSamples["Audio Samples 16kHz"] --> macPipeline["MacDictationPipeline"]
personalDict["PersonalDictionary"] --> biasAdapter["LocalASRBiasAdapter"]
phrasesTSV["phrases.tsv Index"] --> biasAdapter
runtimeCtx["Runtime Context locale app recentHits"] --> biasAdapter
providerCap["Provider Capability"] --> biasAdapter
biasAdapter --> hotwords["hardHotwords"]
biasAdapter --> promptBias["promptBias"]
biasAdapter --> corpusText["corpusContext"]
biasAdapter --> correctionPairs["correctionPairs"]
biasAdapter --> polishFrag["polishFragment"]
biasAdapter --> diag["diagnostics"]
macPipeline --> recognize["LocalASRProvider.recognize"]
hotwords --> recognize
promptBias --> recognize
corpusText --> recognize
recognize --> rawTranscript["Raw Transcript"]
rawTranscript --> correctionLayer["Correction Layer"]
correctionPairs --> correctionLayer
correctionLayer --> polishingService["PolishingService"]
polishFrag --> polishingService
polishingService --> finalText["Final Text"]
```
### 5.2 三层职责边界
| 层 | 职责 | 禁止 |
|----|------|------|
| **ASR bias** | 提高听写阶段专有名词概率 | 承担全文语法润色 |
| **Correction** | 高置信 `aliases → term` 替换 | 凭热词表改写普通句意 |
| **Polish** | 标点、口语转书面、热词保真 | 单独承担全部专名纠错 |
---
## 6. Local ASR Provider 抽象
### 6.1 One-shot first
借鉴 OpenBrocamacOS 听写主路径为 **录完后一次性 `recognize`**;流式预览(`transcribe` / partial)为可选能力,非第一期必做。
建议协议(概念层):
```swift
protocol LocalASRProvider {
var capabilities: LocalASRCapabilities { get }
func recognize(
samples: [Float],
sampleRate: Int,
locale: Locale,
bias: LocalASRBiasPayload?,
options: LocalASRRecognizeOptions?
) async throws -> LocalASRResult
}
```
### 6.2 能力矩阵(须诚实声明)
`supportsStreaming``maxHotwordCount` 外,**必须区分热词模式**
| 字段 | 含义 |
|------|------|
| `hotwordMode` | `none` / `promptOnly` / `perRequest` / `recognizerScoped` / `cloudVocabulary` |
| `hotwordStrength` | `weak` / `medium` / `strong`(产品文案用,非科学绝对值) |
| `hotwordReloadCost` | `none` / `recognizerReload` / `modelReload` |
| `maxPromptCharacters` | soft prompt 上限 |
| `maxHotwordCount` | hard hotwords 上限 |
| `supportsLanguageHint` | 是否接受 locale → language hint |
### 6.3 各 Backend 定位(规划)
| Provider | 角色 | hotwordMode(规划) | 备注 |
|----------|------|---------------------|------|
| **Qwen3 MLX** | 短期主线 | `promptOnly`(待接 `qwen_set_prompt` 等价 API) | 已有权重路径;改动面最小 |
| **Sherpa Qwen3** | 中期 POC | `recognizerScoped` | SayIt 同款;热词变更加载成本 |
| **SenseVoice** | 对照 | `none` 或弱 prompt | 速度/中文基线;非热词主线 |
| **Apple Speech** | Fallback | `none`macOS CLM 待验证) | 系统稳定 |
| **Cloud ASR** | 质量上限 | 各云 `PersonalDictionary+ASRBias` | 非离线 |
**暂不主推**Sherpa Paraformer 作为热词主线(官方不支持 Paraformer hotwords,与 transducer/Qwen3 不同)。
---
## 7. LocalASRBiasAdapter 设计
### 7.1 输入
| 输入 | 说明 |
|------|------|
| `PersonalDictionary.effectiveEntries` | 用户词;最高优先级 |
| `BuiltinLexiconIndex` | 自 `phrases.tsv` 构建;按 weight / 场景筛选 |
| `locale` | `store.localeId` |
| `frontAppBundleId` | 可选;技术类 App 提升 IT 子集权重 |
| `recentHitTerms` | 历史命中统计(若已有) |
| `providerCapabilities` | 决定输出哪些字段、如何截断 |
### 7.2 输出 `LocalASRBiasPayload`
```swift
struct LocalASRBiasPayload {
var hardHotwords: [String] // Sherpa Qwen3、部分云 API
var promptBias: String? // Qwen3 MLX、Whisper 系
var corpusContext: String? // Qwen 云 corpus 风格(若将来统一)
var polishFragment: String // PolishingService 追加块
var correctionPairs: [(alias: String, term: String)]
var diagnostics: BiasDiagnostics // 供设置页 / 调试
}
struct BiasDiagnostics {
var userTermCount: Int
var builtinTermCount: Int
var truncated: Bool
var truncationReason: String?
var selectedSources: [String] // e.g. user, builtin-it, builtin-top
}
```
### 7.3 优先级与截断
```
用户高频 / 最近命中
> PersonalDictionary(全部有效 term
> 当前 App 相关内置词(phrases 子集)
> 高 weight 内置技术词(Top-N
> 其余内置词(仅 polish / 检索,不进 ASR
```
默认建议(可 POC 调参):
| 输出 | 默认上限 |
|------|----------|
| `hardHotwords` | 100Qwen3 Sherpa);对齐 `asrHotwords(maxCount: 100)` |
| `promptBias` | 800 字符;复用 `asrPromptBias(maxCharacters:)` 逻辑 |
| ASR 层内置词 | 200500**不全量 1 万** |
| `correctionPairs` | aliases 全量可进后处理,但仅 **整词 / 高置信** 替换 |
### 7.4 防污染规则
- 近静音、极短音频:减少或跳过内置词,保留用户词。
- 用户词始终优先于内置词。
- diagnostics 必须记录「为何丢弃」某批词(超 cap、provider 不支持、场景不匹配)。
### 7.5 与现有云代码复用
扩展 [`PersonalDictionary+ASRBias.swift`](../OSGKeyboardShared/Models/PersonalDictionary+ASRBias.swift) 为 **单一事实来源**,新增例如:
- `localPromptBias(maxCharacters:builtinTerms:)`
- `correctionPairs()`
- `rankedTermsForASR(limit:builtinBoost:)`
避免 macOS / iOS / Cloud 三套独立拼接逻辑。
---
## 8. 词库策略
### 8.1 PersonalDictionary
| 字段 | ASR | Correction | Polish |
|------|-----|------------|--------|
| `term` | hotword / prompt | 标准写法 | 必须保留 |
| `aliases` | 可进 prompt 提示 | **主战场** | 语义纠错参考 |
| `category` | 排序权重 | — | 分组展示 |
| `usageCount` | 排序权重 | — | — |
iCloud[`PersonalDictionaryCloudSync`](../OSGKeyboardShared/Services/PersonalDictionaryCloudSync/) 保证 Mac / iOS / Extension 一致;本地 ASR 只读 `AppGroupStore.personalDictionary`
### 8.2 phrases.tsv 分层
**不全量进入 ASR prompt。**
| 层级 | 用途 | 规模建议 |
|------|------|----------|
| L1 ASR 高价值 | `weight >= 4` 或 curated IT 品牌缩写 | 200500 |
| L2 场景相关 | 按 `frontApp` / 用户最近命中动态加入 | +0–100 |
| L3 全量索引 | 后处理模糊匹配、polish 检索 | ~10k |
TSV 列:`word``pinyin``source``weight` — 构建索引时保留 weight 用于排序。
### 8.3 iOS CLM 与 macOS 关系
- iOSTSV → export script → `.bin``CustomLanguageModelManager`
- macOSTSV → `BuiltinLexiconIndex``LocalASRBiasAdapter` → Qwen3 / Sherpa / Polish
- **同一 TSV 源**,两种消费格式;不尝试把 `.bin` 喂给 Sherpa/MLX
---
## 9. 自动词库学习(可选实验,非第一期)
借鉴 Typeflux `WorkflowController+AutomaticVocabulary`
- 听写插入后,短时观察用户在前台可编辑框内的修改
- LLM 或规则判断是否为「专名 / 品牌 / 大小写修正」
- 候选进入 **待确认队列**,不直接写入 `PersonalDictionary`
**约束(必须写进隐私说明)**
- 默认关闭
- 不自动 iCloud 同步待确认项
- 可一键清空、可审计来源
- 拒绝:整句改写、纯语法修正、过短词条、编辑幅度过大
OSG 已有 `PersonalDictionary.Entry.Source.recentEdit` 与合并逻辑,可与之对齐而非新建平行存储。
---
## 10. 本地模型管理
### 10.1 原则
- **Catalog 与 Runtime 分离**:下载源只影响安装;推理只读本地 **已验证 manifest**
- **Selected model 显式**:禁止「扫描目录用第一个 onnx」
- **完整性**sha256 或 size 校验 + staging 目录原子发布
### 10.2 存储布局(建议)
```
~/Library/Application Support/OSGKeyboard/
LocalASRModels/
manifest.json # 已安装模型、版本、backend、capabilities
qwen3-mlx-1.7b/ # 当前 MLX 布局(可与现路径兼容)
sherpa-qwen3-0.6b/
sherpa-sensevoice-small/
```
### 10.3 Catalog 条目(概念)
```json
{
"modelId": "sherpa-qwen3-asr-0.6b-int8",
"displayName": "Qwen3-ASR 0.6B (Sherpa)",
"backend": "sherpaQwen3",
"sizeBytes": 1200000000,
"recommendedLocales": ["zh-CN", "en-US"],
"supportsHotwords": true,
"hotwordMode": "recognizerScoped",
"sources": [
{
"type": "modelscope",
"url": "https://www.modelscope.cn/api/v1/models/.../repo?Revision=master&FilePath=...",
"sha256": "...",
"priority": 1
},
{
"type": "huggingface",
"url": "https://huggingface.co/...",
"priority": 2
},
{
"type": "github",
"url": "https://github.com/k2-fsa/sherpa-onnx/releases/download/...",
"priority": 3
}
]
}
```
### 10.4 ModelScope 策略
| 场景 | 策略 |
|------|------|
| 中国大陆用户默认 | **ModelScope 优先**Qwen3-ASR、SenseVoice、FunASR 相关 ONNX |
| 国际 / ModelScope 失败 | Hugging Face → GitHub Releases |
| 企业内网 | `custom` mirror URL(用户配置) |
| 安装流程 | download → verify → extract → validate required files → rename staging → update manifest |
| 失败 | 清理 staging / 临时文件;不留下半安装状态 |
MLX Qwen3 权重:可继续支持用户自选目录(现状),逐步纳入统一 catalog 的 `type: mlx` 条目。
### 10.5 UI / 设置需求(规划)
- 模型列表:体积、语言、安装状态、是否支持热词
- 下载进度:phasedownloading / extracting / validating / finalizing
- 切换模型:仅允许 **installed + manifest 合法** 的项为默认
- 诊断:当前 provider、capability、上次 bias diagnostics
---
## 11. 后端对比与决策矩阵
### 11.1 产品分层
```text
主线: Qwen3 MLX + LocalASRBiasAdapter + Polish/Correction
重点 POC: Sherpa Qwen3 hotwords
对照: SenseVoice(速度)、Apple Speechfallback
参考上限: Cloud ASR + PersonalDictionary
团队部署: Qwen3-ASR vLLMSayIt 式,非客户端主线)
暂不主推: FunASR Paraformer hotwords、纯 OpenLess 本地词典叙事
```
### 11.2 详细对比
| 维度 | Qwen3 MLX | Sherpa Qwen3 | SenseVoice | Apple Speech | Cloud |
|------|-----------|--------------|------------|--------------|-------|
| 离线 | ✅ | ✅ | ✅ | ✅ | ❌ |
| 中英混合技术词 | 强(经验性) | 待 POC | 中 | 中 | 强 |
| Hard hotwords | ❌→prompt | ✅ recognizerScoped | ❌ | ❌ | ✅ 因 provider 异 |
| 实现成本 | 低(已有) | 高(runtime 体积) | 中 | 低 | 已有 |
| 模型体积 | ~1.3GB+ | 类似 | ~350MB 级 | 0 | N/A |
| 隐私 | 本地 | 本地 | 本地 | 本地 | 依配置 |
### 11.3 Sherpa POC 通过阈值(建议)
相对 **当前 Qwen3 MLX + 仅 polish** 基线:
| 指标 | 建议阈值 |
|------|----------|
| 用户热词召回率 | 提升 ≥ 20% |
| 误触发率(未说热词却被改成热词) | ≤ 2% |
| 30s 音频端到端延迟 | ≤ 基线 × 1.5 |
| 内存峰值(8GB Mac 目标机) | 可接受且无 OOM |
| 安装成功率 | 普通用户可完成 ModelScope/HF 下载 |
未达阈值:**保留 Qwen3 MLX 主线**Sherpa 仅作高级选项。
---
## 12. POC 评测计划
### 12.1 测试集
| 类别 | 内容 | 目的 |
|------|------|------|
| A 普通中文 | 日常口语 50 句 | 基线 WER / 误触发 |
| B 技术术语 | SwiftUI、Cursor、Qwen3-ASR 等 50 句 | 专名召回 |
| C 用户词典 | 模拟 PersonalDictionary 20 词 × 多句 | 热词核心场景 |
| D 长句润色 | 30s+ 口语 | polish 兜底 |
| E 噪声 / 短句 | 低 SNR、<2s | 防污染规则 |
### 12.2 对照矩阵
| 配置 | 说明 |
|------|------|
| Baseline | Qwen3 MLX,无 bias |
| B1 | Qwen3 MLX + promptBias |
| B2 | B1 + polishFragment + correction |
| POC1 | Sherpa Qwen3 + hardHotwords |
| POC2 | SenseVoice,无 hotwords |
| Ref | 云 ASR + PersonalDictionary |
### 12.3 指标
- Raw CER/WER(中文可用字错误率)
- **Hotword recall**(用户词是否出现在 raw 或 final)
- **False hotword rate**
- Final accuracy(用户主观或编辑距离)
- Latencyrecord end → text inserted
- Memory / CPU、模型加载时间
- 离线可靠性(无网络完成全流程)
---
## 13. 失败回退策略
```mermaid
flowchart TD
start["recognize 开始"] --> qwen{"Qwen3 MLX 可用?"}
qwen -->|是| qwenRun["Qwen3 + bias"]
qwen -->|否| apple["Apple Speech"]
qwenRun -->|失败| apple
qwenRun -->|成功| post["Correction + Polish"]
apple --> post
post -->|Polish 失败| raw["返回 raw transcript"]
post -->|成功| done["插入 final"]
```
| 条件 | 行为 |
|------|------|
| Qwen3 权重缺失 | Apple Speech(现状) |
| Qwen3 推理失败 | 可配置:重试一次 → Apple Speech |
| Sherpa 未安装 | 不回退云;提示下载 |
| 模型 manifest 损坏 | 标记 invalid,禁止设为默认 |
| Polish 失败 | 使用 raw(现状) |
| 用户禁用云 | 不静默切云 |
---
## 14. 分阶段落地路线
| Phase | 内容 | 交付物 |
|-------|------|--------|
| **1** | 本文档定稿;`LocalASRCapabilities` + `LocalASRBiasPayload` 类型设计 | 架构文档 + ADR 可选 |
| **2** | `MacDictationPipeline` 接入 adapterQwen3 MLX `promptBias`polish + correction | 实现 PR |
| **3** | `BuiltinLexiconIndex`Top-Ndiagnostics UI | 实现 PR |
| **4** | Model catalog + ModelScope 下载 + manifest | 实现 PR |
| **5** | Sherpa Qwen3 POC + 评测报告 | 决策是否默认切换 |
| **6** | 可选:自动词库学习实验(默认关) | 功能 flag |
---
## 15. 风险与待确认问题
| 风险 | 缓解 |
|------|------|
| Qwen3 prompt bias 过弱 | POC 对比 Sherpa hard hotwords;保留 correction + polish |
| 热词过多污染识别 | cap + 场景筛选 + diagnostics |
| aliases 后处理误改 | 整词边界、低置信跳过 |
| Sherpa 分发体积 / 签名 / 公证 | 单独评估;可选按需下载 |
| Apple Speech macOS CLM | 调研 macOS 26+ 是否可接 CLM;否则仅 fallback |
| ModelScope API 变更 | 多 mirrormanifest 可更新 URL |
| 自动学习隐私 | 默认关、本地、待确认 |
| 低配 Mac 内存 | 单模型常驻策略;SenseVoice 作轻量选项 |
**待确认**
1. MLX Swift API 是否暴露等价 `setPrompt`(对标 Open-Less/qwen-asr `qwen_set_prompt`
2. Sherpa-ONNX Swift/SPM 与 App Store 公证路径
3. `phrases.tsv` Top-N 是否按 `weight` 静态裁剪即可,或需按 App 动态检索
---
## 16. 相关代码索引
| 主题 | 路径 |
|------|------|
| Mac 听写管道 | `OSGKeyboardMac/MacDictationPipeline.swift` |
| 本地 ASR 入口 | `OSGKeyboardMac/MacLocalASRService.swift` |
| Qwen3 MLX | `OSGKeyboardMac/MacQwen3LocalASR.swift`, `MacQwen3ASREngine.swift` |
| Apple Speech fallback | `OSGKeyboardMac/MacSpeechLocalASR.swift` |
| 用户词库 | `OSGKeyboardShared/Models/PersonalDictionary.swift` |
| 云 bias | `OSGKeyboardShared/Models/PersonalDictionary+ASRBias.swift` |
| 内置 TSV | `OSGKeyboard/Resources/CustomLanguageModel/v1/phrases.tsv` |
| iOS CLM | `OSGKeyboardShared/Services/CustomLanguageModelManager.swift` |
| 润色 | `OSGKeyboardShared/Services/PolishingService.swift` |
| 文本插入 | `OSGKeyboardMac/MacTextInsertionService.swift` |
---
## 17. 修订记录
| 日期 | 说明 |
|------|------|
| 2026-03-31 | 初版:基于 OSG 代码审计 + typeless-alternative 竞品源码 + 计划评审(ModelScope、hotwordMode、回退策略) |
+17 -3
View File
@@ -50,8 +50,8 @@ settings:
ASSETCATALOG_COMPILER_GENERATE_SWIFT_ASSET_SYMBOL_EXTENSIONS: YES
STRING_CATALOG_GENERATE_SYMBOLS: YES
CLANG_CXX_LANGUAGE_STANDARD: c++17
MARKETING_VERSION: "0.5.0"
CURRENT_PROJECT_VERSION: "18"
MARKETING_VERSION: "0.5.2"
CURRENT_PROJECT_VERSION: "19"
# 签名配置来自 Signing.local.xcconfiggitignored,不会被覆盖)
# 项目级签名 xcconfig,适用于所有 target
@@ -327,6 +327,7 @@ targets:
resources:
- path: OSGKeyboardShared/Resources/CustomLanguageModel/v1/OSGKeyboardCLM.bin
- path: OSGKeyboardShared/Resources/CustomLanguageModel/v1/compiled-manifest.json
- path: OSGKeyboardShared/Resources/LocalASR/local-asr-catalog.json
info:
path: OSGKeyboardShared/Info.plist
settings:
@@ -429,10 +430,18 @@ targets:
buildPhase: resources
- path: OSGKeyboardShared/zh-Hans.lproj/Shared.strings
buildPhase: resources
- path: OSGKeyboard/Resources/CustomLanguageModel/v1/phrases.tsv
buildPhase: resources
- path: OSGKeyboardShared/Resources/LocalASR/local-asr-catalog.json
buildPhase: resources
entitlements:
path: OSGKeyboardMac/OSGKeyboardMac.entitlements
properties:
com.apple.security.app-sandbox: true
# Sandbox disabled on purpose: this menu-bar dictation utility uses the
# Accessibility API to monitor a global hotkey and inject ⌘V into other
# apps — both are forbidden by App Sandbox. Ship via Developer ID +
# notarization (not Mac App Store / TestFlight, which force the sandbox).
com.apple.security.app-sandbox: false
com.apple.security.device.audio-input: true
com.apple.security.network.client: true
com.apple.developer.ubiquity-kvstore-identifier: $(TeamIdentifierPrefix)com.osgkeyboard.ios
@@ -455,6 +464,9 @@ targets:
ITSAppUsesNonExemptEncryption: false
settings:
base:
PRODUCT_NAME: OSGKeyboard
INFOPLIST_KEY_CFBundleDisplayName: OSGKeyboard
INFOPLIST_KEY_CFBundleName: OSGKeyboard
PRODUCT_BUNDLE_IDENTIFIER: com.osgkeyboard.mac
MACOSX_DEPLOYMENT_TARGET: "15.0"
ASSETCATALOG_COMPILER_APPICON_NAME: AppIcon
@@ -462,6 +474,8 @@ targets:
APPLICATION_EXTENSION_API_ONLY: NO
CODE_SIGN_STYLE: Automatic
DEVELOPMENT_TEAM: X329MZU23S
# Required for Developer ID distribution + notarization (outside App Store).
ENABLE_HARDENED_RUNTIME: YES
dependencies:
- sdk: Speech.framework
- sdk: AVFoundation.framework