diff --git a/CHANGELOG.md b/CHANGELOG.md index 2703ab3..637796a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,20 +7,29 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [0.5.2] - 2026-07-09 + ### Added +- **macOS local ASR models**: the desktop app ships a bundled model catalog with one-click download of Sherpa Qwen3 (hotwords) and SenseVoice models, plus a shared model storage directory; downloads show a circular progress ring with pause / resume, and each row has inline Download / Delete actions. / **macOS 本地 ASR 模型**：桌面 App 内置模型目录，可一键下载 Sherpa Qwen3（热词）与 SenseVoice 模型，并共用同一模型存储目录；下载显示带暂停 / 继续的环形进度，每行提供内联的下载 / 删除操作。 +- **Shared model directory for MLX**: Qwen3-ASR MLX now uses a fixed subfolder inside the shared model storage — drop converted weights into the folder opened by "Open folder"; no per-model directory picker. / **MLX 共用模型目录**：Qwen3-ASR MLX 改用共享模型存储中的固定子目录——把转换好的权重放入「打开目录」指向的文件夹即可，不再逐模型选目录。 - **iCloud sync hardening**: per-field settings merge (`appSettings.v2`), per-device usage statistics (G-Counter), tombstoned dictionary/history merge, and a low-risk **Sync Now** action in Settings. / **iCloud 同步加固**：设置按字段合并（`appSettings.v2`）、统计按设备 G-Counter 累计、词库/历史带墓碑合并，并在设置中新增低风险的**立即同步**操作。 ### Changed - **API key sync**: cloud provider API keys now replicate through **iCloud Keychain** when settings sync is on — never through iCloud KVS JSON. / **API 密钥同步**：开启设置同步后，云端服务商 API 密钥改由 **iCloud 钥匙串**复制，不再写入 iCloud KVS JSON。 - **Speech history cap**: synced history limit is **300** entries (aligned with the sync payload). / **语音历史上限**：可同步历史上限为 **300** 条（与同步载荷一致）。 +- **macOS app name**: the built product is now `OSGKeyboard.app` (was `OSGKeyboardMac.app`); Dock, About, and Finder all read **OSGKeyboard**. / **macOS 应用名称**：编译产物改为 `OSGKeyboard.app`（原 `OSGKeyboardMac.app`）；Dock、关于窗口与 Finder 均显示 **OSGKeyboard**。 +- **macOS local recognition label**: the Settings entry is now simply "Local Recognition" and no longer names a specific model. / **macOS 本地识别标签**：设置项改为「本地识别」，不再绑定具体模型名称。 ### Fixed +- **macOS menu-bar icon in light mode**: the status-bar icon now follows the *system* menu-bar appearance, so forcing the app into Light while the system is Dark no longer renders an unreadable dark icon; a refreshed status mark is used. / **macOS 菜单栏图标（浅色模式）**：状态栏图标改为跟随*系统*菜单栏外观，App 强制浅色而系统为深色时不再出现看不清的深色图标；并更新了状态栏图标。 +- **macOS light-mode sidebar**: restored the native translucent sidebar material so the light appearance matches system apps (e.g. System Settings, Notes) instead of a flat grey fill. / **macOS 浅色侧边栏**：恢复原生半透明侧栏材质，浅色外观与系统应用（如系统设置、备忘录）一致，不再是扁平灰底。 - **Settings sync wiping API keys**: pulling a legacy settings blob without API key fields no longer deletes local Keychain entries. / **设置同步清空 API 密钥**：拉取不含 API 密钥字段的旧版设置包时，不再删除本地 Keychain 项。 - **Cross-device settings conflicts**: changing different settings on two devices no longer lets one device's full blob overwrite the other's unrelated fields. / **跨设备设置冲突**：两台设备分别修改不同设置项时，不再因整包覆盖而冲掉对方未改动的字段。 - **Usage statistics under-counting**: offline usage on multiple devices now sums correctly instead of taking per-field `max()`. / **使用统计少计**：多设备离线各自累计后合并为求和，不再对总量取 `max()`。 - **Dictionary/history resurrection**: deletes and "clear all" on one device propagate via tombstones so older remote entries cannot come back. / **词库/历史复活**：单设备删除或清空会通过墓碑传播，远端旧条目无法复活。 - **Flow false-ready mic state**: the keyboard mic now stays orange until the host app publishes a real ready contract (capture engine live + polling idle), not merely a fresh heartbeat; green tap-to-talk and jump-to-host behavior share the same `MicVoiceAvailability` gate, and orphaned `stopped` signals self-heal instead of hanging until timeout. / **Flow 伪就绪麦克风状态**：键盘麦克风在主 App 发布真实就绪合约（音频引擎在跑且轮询空闲）之前保持橙色，不再仅凭心跳误判；绿色「点按说话」与跳转主 App 共用同一 `MicVoiceAvailability` 闸门，孤立的 `stopped` 信号会自愈而不再长时间卡住。 - **Flow mic stuck orange after ready**: a single stale cross-process heartbeat read no longer flips a healthy session into a sticky "session ended" error that forced the mic orange. The "session ended" hint now fires only when the (heartbeat-independent) session contract truly drops; a brief read jitter is smoothed by a ready grace window, and a lingering expired hint auto-recovers to green once the host is ready again. / **就绪后麦克风卡橙色**：单次跨进程心跳读数抖动不再把健康会话打成粘滞的「会话已结束」错误、强制麦克风变橙。「会话已结束」提示现仅在（不依赖心跳的）会话合约真正失效时触发；短暂读数抖动由就绪宽限期平滑，遗留的过期提示会在宿主重新就绪后自动恢复为绿色。 +- **Orphaned Live Activity after force-quit**: force-quitting the app no longer leaves a stale OSGKeyboard status stuck on the Lock Screen / Dynamic Island. The `staleDate` is now ~45s (refreshed by the heartbeat while the session is alive) so the system reclaims a dead session's island on its own, and every app foreground now sweeps leftover Live Activities *before* trying to (re)start a session — so even a start that later fails (e.g. mic proof timeout) still clears the zombie island. / **强杀后遗留 Live Activity**：强制退出 App 不再在锁屏 / 灵动岛留下无法消失的 OSGKeyboard 状态。`staleDate` 缩短为约 45 秒（会话存活期间由心跳持续刷新），系统会自动回收已死会话的灵动岛；且每次 App 回到前台都会**先**清扫遗留的 Live Activity 再尝试（重新）启动会话——即便本次启动随后失败（如麦克风就绪超时），也不会留下僵尸灵动岛。 ## [0.5.0] - 2026-07-07 diff --git a/OSGKeyboard/Assets.xcassets/OSGStatusMark.imageset/Contents.json b/OSGKeyboard/Assets.xcassets/OSGStatusMark.imageset/Contents.json new file mode 100644 index 0000000..f008e7a --- /dev/null +++ b/OSGKeyboard/Assets.xcassets/OSGStatusMark.imageset/Contents.json @@ -0,0 +1,25 @@ +{ + "images" : [ + { + "filename" : "OSGStatusMark.png", + "idiom" : "universal", + "scale" : "1x" + }, + { + "idiom" : "universal", + "scale" : "2x" + }, + { + "idiom" : "universal", + "scale" : "3x" + } + ], + "info" : { + "author" : "xcode", + "version" : 1 + }, + "properties" : { + "preserves-vector-representation" : true, + "template-rendering-intent" : "template" + } +} diff --git a/OSGKeyboard/Assets.xcassets/OSGStatusMark.imageset/OSGStatusMark.png b/OSGKeyboard/Assets.xcassets/OSGStatusMark.imageset/OSGStatusMark.png new file mode 100644 index 0000000..b99bb95 Binary files /dev/null and b/OSGKeyboard/Assets.xcassets/OSGStatusMark.imageset/OSGStatusMark.png differ diff --git a/OSGKeyboard/Services/FlowLiveActivityController.swift b/OSGKeyboard/Services/FlowLiveActivityController.swift index 8c204ca..a173f26 100644 --- a/OSGKeyboard/Services/FlowLiveActivityController.swift +++ b/OSGKeyboard/Services/FlowLiveActivityController.swift @@ -10,12 +10,16 @@ import OSGKeyboardShared enum FlowLiveActivityController { nonisolated(unsafe) private static var currentActivity: Activity? + /// Last phase pushed to the Live Activity so `keepAlive()` can refresh the + /// `staleDate` without changing what the user sees. + nonisolated(unsafe) private static var currentPhase: FlowActivityAttributes.ContentState.Phase = .idle /// If the host app is force-quit its `endSession()` never runs, orphaning - /// the Live Activity. A `staleDate` lets the system grey it out and become - /// willing to reclaim it without our process — refreshed on every update - /// so a genuinely active, in-use session never looks stale. - private static let staleWindow: TimeInterval = 60 * 60 + /// the Live Activity. A short `staleDate` lets the system grey it out and + /// reclaim it on its own within ~45s of the process dying. While the host is + /// alive the heartbeat calls `keepAlive()` well inside this window, so a + /// genuinely active session never looks stale. + private static let staleWindow: TimeInterval = 45 private static func freshContent( phase: FlowActivityAttributes.ContentState.Phase @@ -41,6 +45,7 @@ enum FlowLiveActivityController { } do { + currentPhase = .idle currentActivity = try Activity.request( attributes: FlowActivityAttributes(), content: freshContent(phase: .idle), @@ -54,14 +59,28 @@ enum FlowLiveActivityController { static func update(phase: FlowActivityAttributes.ContentState.Phase) { guard let activity = currentActivity else { return } + currentPhase = phase let content = freshContent(phase: phase) Task { await activity.update(content) } } + /// Push a fresh `staleDate` without changing the visible phase. The host + /// heartbeat calls this well inside `staleWindow` so an in-use session + /// never looks stale; once the process dies the refreshes stop and the + /// system reclaims the orphaned Live Activity on its own. + static func keepAlive() { + guard let activity = currentActivity else { return } + let content = freshContent(phase: currentPhase) + Task { + await activity.update(content) + } + } + /// Dismiss the island presentation when the Flow session ends. static func endSession() { + currentPhase = .idle guard let activity = currentActivity else { endStaleActivities() return @@ -74,6 +93,18 @@ enum FlowLiveActivityController { } } + /// Clear Live Activities orphaned by a previous (force-quit) host process. + /// + /// Safe to call on every app foreground: when this process already owns a + /// Live Activity (`currentActivity != nil`) we leave it alone so a healthy + /// running session is never torn down; we only sweep leftovers that belong + /// to a dead process. Call this *before* attempting to (re)start a session + /// so a failed start (e.g. mic timeout) still clears the stale island. + static func clearOrphanedActivities() { + guard currentActivity == nil else { return } + endStaleActivities() + } + /// Host relaunch can leave orphan activities; clear them before starting anew. private static func endStaleActivities() { let staleActivities = Activity.activities diff --git a/OSGKeyboard/Services/FlowSessionManager.swift b/OSGKeyboard/Services/FlowSessionManager.swift index 269d424..cc7539d 100644 --- a/OSGKeyboard/Services/FlowSessionManager.swift +++ b/OSGKeyboard/Services/FlowSessionManager.swift @@ -25,9 +25,7 @@ final class FlowSessionManager: ObservableObject { private let capture = FlowContinuousCapture() private let store = AppGroupStore() /// Cloud-engine polish; local engine runs through built-in DeepSeek polish. - private var polisher: PolishingService { - PolishingService() - } + private let polisher = PolishingService() /// Cached ASR instance. v0.2.0: the only on-device backend is iOS /// `SpeechAnalyzer`, which has no warm-up step — we can hand the /// factory-built service straight back without going through the @@ -49,17 +47,26 @@ final class FlowSessionManager: ObservableObject { private var expiryTask: Task? private var levelTask: Task? private var startTask: Task? + private var commandObserver: FlowSessionDarwinObserver? /// Last recording state the poll loop observed — logs only on transition. private var lastObservedRecordingState: FlowSessionKeys.RecordingState = .idle + private var activeSessionId: UUID? + private var currentUtteranceId: UUID? + private var currentCommandSeq: Int64 = 0 + private var lastHandledCommandSeq: Int64 = 0 private var isUtteranceRecording = false /// True from `stopped` until the result/error is written back to App Group. private var isUtteranceProcessing = false private var finalizeTask: Task? private var asrTask: Task? + private var utteranceSafetyTask: Task? private var chunkedPipeline: ChunkedUtterancePipeline? private var currentPartial = "" private var lastFinal = "" private var chunkWarnings: [String] = [] + private var lastReadyTraceSignature = "" + private var lastCommandFingerprint = "" + private var lastIgnoredCommandSignature = "" /// Wall-clock span of the current mic-open utterance (excludes LLM polish). private var utteranceRecordingStartedAt: Date? /// True while the host app scene is `.active` — drives foreground renewal. @@ -67,7 +74,11 @@ final class FlowSessionManager: ObservableObject { private var backgroundTaskID: UIBackgroundTaskIdentifier = .invalid /// True while handling a keyboard-initiated `startflow` cold start. private var isColdStartHandoff = false - private static let coldStartAudioProofTimeout: TimeInterval = 2.5 + private var coldStartRecoveryTask: Task? + /// Initial proof window — cold mic sessions often need >2.5s after app switch. + private static let coldStartAudioProofTimeout: TimeInterval = 6 + /// Extra window after the first timeout while the overlay shows a failure hint. + private static let coldStartRecoveryProofTimeout: TimeInterval = 12 init() { // Sessions are (re)started explicitly on app foreground via @@ -83,6 +94,10 @@ final class FlowSessionManager: ObservableObject { /// Starts a Flow session: permissions → continuous capture → App Group active. func startSession(duration: TimeInterval? = nil, coldStart: Bool = false) { + traceState( + "startSession.request", + extra: "coldStart=\(coldStart) duration=\(Int(duration ?? FlowSessionPolicy.sessionDuration()))" + ) guard AppGroup.isAvailable else { debug("cannot start flow session: App Group unavailable") return @@ -106,7 +121,10 @@ final class FlowSessionManager: ObservableObject { return } - guard !isStarting else { return } + guard !isStarting else { + traceState("startSession.ignored", extra: "reason=alreadyStarting") + return + } startTask?.cancel() startTask = Task { @MainActor [weak self] in @@ -148,6 +166,14 @@ final class FlowSessionManager: ObservableObject { /// behind (its `endSession()` could not run at kill time). func activateOnForeground() { guard AppGroup.isAvailable else { return } + // Sweep any Live Activity a previous (force-quit) process left behind + // *before* we try to (re)start a session. Doing it here — rather than + // only inside `startSession()`'s success path — means a start that + // later fails (e.g. mic proof timeout) still clears the stale island + // instead of leaving a zombie on the lock screen / Dynamic Island. + // No-op when this process already owns a healthy Live Activity. + FlowLiveActivityController.clearOrphanedActivities() + guard AppPermissions.flowRequirementsMet else { sessionWarning = permissionWarningMessage() FlowSessionBridge.setHostReady(false) @@ -161,7 +187,8 @@ final class FlowSessionManager: ObservableObject { } func dismissColdStartOverlay() { - guard coldStartContext?.state != .preparing else { return } + coldStartRecoveryTask?.cancel() + coldStartRecoveryTask = nil if isActive { refreshHostReady() } @@ -189,8 +216,11 @@ final class FlowSessionManager: ObservableObject { coldStartContext = nil isColdStartHandoff = false + coldStartRecoveryTask?.cancel() + coldStartRecoveryTask = nil startTask?.cancel() startTask = nil + commandObserver = nil pollingTask?.cancel() pollingTask = nil heartbeatTask?.cancel() @@ -201,6 +231,8 @@ final class FlowSessionManager: ObservableObject { levelTask = nil finalizeTask?.cancel() finalizeTask = nil + utteranceSafetyTask?.cancel() + utteranceSafetyTask = nil if isUtteranceRecording || isUtteranceProcessing { capture.cancelUtterance() @@ -210,6 +242,10 @@ final class FlowSessionManager: ObservableObject { } asrTask = nil chunkedPipeline = nil + activeSessionId = nil + currentUtteranceId = nil + currentCommandSeq = 0 + lastHandledCommandSeq = 0 isUtteranceRecording = false isUtteranceProcessing = false @@ -305,9 +341,11 @@ final class FlowSessionManager: ObservableObject { guard isActive else { return } if capture.running { - capture.reassertIfRunning() - if capture.engineHasRecentAudio() { + let reasserted = capture.reassertIfRunning() + if reasserted, capture.engineHasRecentAudio() { sessionWarning = nil + } else if !reasserted { + sessionWarning = AppL10n.string("flow.error.audioUnavailable") } refreshHostReady() return @@ -329,22 +367,94 @@ final class FlowSessionManager: ObservableObject { /// Publish whether the keyboard can start a new utterance without jumping to the host app. private func refreshHostReady() { guard isActive else { - FlowSessionBridge.setHostReady(false) + FlowSessionBridge.writeReadySnapshot( + FlowReadySnapshot( + sessionId: activeSessionId, + ready: false, + reason: .noSession, + engineMode: store.engineMode, + localeId: store.localeId, + sessionExpiresAt: FlowSessionBridge.sessionExpiresAt() + ) + ) return } let pollingAlive = pollingTask != nil && pollingTask?.isCancelled != true - // Steady-state ready uses structural engine liveness. The stricter - // "recent audio frame" proof is reserved for cold-start handoff only - // (`waitForAudioProof`) so brief UI-driven session hiccups do not - // drop the keyboard back to orange while the host app is foreground. + let hasRecentAudio = capture.engineHasRecentAudio(maxAge: 2) let canAcceptUtterance = capture.engineIsLive && pollingAlive + && hasRecentAudio && !isUtteranceRecording && !isUtteranceProcessing && sessionWarning == nil - FlowSessionBridge.setHostReady(canAcceptUtterance) + let reason: FlowReadySnapshot.Reason + if canAcceptUtterance { + reason = .ready + } else if sessionWarning != nil { + reason = .error + } else if isUtteranceRecording { + reason = .recording + } else if isUtteranceProcessing { + reason = .processing + } else if !capture.engineIsLive { + reason = .audioEngineNotLive + } else if !hasRecentAudio { + reason = .waitingForAudioProof + } else { + reason = .starting + } + + let now = Date().timeIntervalSince1970 + FlowSessionBridge.writeReadySnapshot( + FlowReadySnapshot( + sessionId: activeSessionId, + ready: canAcceptUtterance, + reason: reason, + heartbeatAt: now, + readyAt: canAcceptUtterance ? now : nil, + audioProofAt: hasRecentAudio ? now : nil, + engineMode: store.engineMode, + localeId: store.localeId, + busyUtteranceId: isUtteranceRecording || isUtteranceProcessing ? currentUtteranceId : nil, + sessionExpiresAt: FlowSessionBridge.sessionExpiresAt() + ) + ) + let signature = [ + canAcceptUtterance ? "ready=1" : "ready=0", + "reason=\(reason.rawValue)", + capture.engineIsLive ? "engine=live" : "engine=dead", + hasRecentAudio ? "audio=fresh" : "audio=stale", + isUtteranceRecording ? "recording=1" : "recording=0", + isUtteranceProcessing ? "processing=1" : "processing=0", + sessionWarning == nil ? "warning=0" : "warning=1" + ].joined(separator: "|") + if signature != lastReadyTraceSignature { + lastReadyTraceSignature = signature + traceState("hostReady.update", extra: signature) + } + reconcileColdStartOverlayIfRecovered() + } + + /// When the host contract turns green while the cold-start overlay still + /// shows a stale preparing/failed snapshot, heal automatically. + private func reconcileColdStartOverlayIfRecovered() { + guard isColdStartHandoff, isActive else { return } + guard FlowSessionBridge.isHostReady() else { return } + guard let context = coldStartContext else { return } + + switch context.state { + case .preparing: + presentColdStartReadyOverlay() + case .failed: + sessionWarning = nil + coldStartRecoveryTask?.cancel() + coldStartRecoveryTask = nil + dismissColdStartOverlay() + case .ready: + break + } } /// Home preview field gained focus while this app is the Flow host. @@ -378,12 +488,14 @@ final class FlowSessionManager: ObservableObject { // MARK: - Session start private func startSessionAsync(duration: TimeInterval?) async { + traceState("startSessionAsync.begin") isStarting = true sessionWarning = nil defer { isStarting = false } guard AppPermissions.flowRequirementsMet else { sessionWarning = permissionWarningMessage() + traceState("startSessionAsync.blocked", extra: "reason=permissions") FlowSessionBridge.setHostReady(false) if isColdStartHandoff { showColdStartPermissionFailure() @@ -397,6 +509,7 @@ final class FlowSessionManager: ObservableObject { } catch { let message = (error as? LocalizedError)?.errorDescription ?? error.localizedDescription sessionWarning = message + traceState("startSessionAsync.failed", extra: "reason=captureStart error=\(message)") FlowSessionBridge.setHostReady(false) if isColdStartHandoff { showColdStartAudioFailure(message: message) @@ -408,23 +521,36 @@ final class FlowSessionManager: ObservableObject { guard await waitForAudioProof() else { let message = AppL10n.string("flow.coldStart.error.audioTimeout") sessionWarning = message - capture.stop() + traceState("startSessionAsync.failed", extra: "reason=audioProofTimeout") FlowSessionBridge.setHostReady(false) if isColdStartHandoff { showColdStartAudioFailure(message: message) + scheduleColdStartRecovery(duration: duration) + } else { + capture.stop() } debug("continuous capture did not produce audio frames before timeout") return } + activateFlowSessionAfterAudioProof(duration: duration) + traceState("startSessionAsync.ready") + debug("Flow session started (\(Int(duration ?? FlowSessionPolicy.sessionDuration()))s inactivity window), continuous capture running") + } + + private func activateFlowSessionAfterAudioProof(duration: TimeInterval?) { let resolvedDuration = duration ?? FlowSessionPolicy.sessionDuration() - FlowSessionBridge.markSessionActive(duration: resolvedDuration) + let sessionId = activeSessionId ?? UUID() + activeSessionId = sessionId + lastHandledCommandSeq = 0 + FlowSessionBridge.markSessionActive(duration: resolvedDuration, sessionId: sessionId) FlowSessionDarwin.postSessionChanged() isActive = true ScreenWakeLock.acquire() sessionExpiresAt = Date().addingTimeInterval(resolvedDuration) startHeartbeat() + startCommandObserver() startPolling() startLevelPublishing() scheduleExpiry(after: resolvedDuration) @@ -434,7 +560,7 @@ final class FlowSessionManager: ObservableObject { FlowLiveActivityController.startSession() refreshHostReady() - debug("Flow session started (\(Int(resolvedDuration))s inactivity window), continuous capture running") + traceState("activateFlowSessionAfterAudioProof.done") } private func prepareExistingSessionForColdStartReturn() async { @@ -445,6 +571,7 @@ final class FlowSessionManager: ObservableObject { sessionWarning = message FlowSessionBridge.setHostReady(false) showColdStartAudioFailure(message: message) + scheduleColdStartRecovery(duration: nil) debug("existing session failed cold-start audio proof") return } @@ -466,25 +593,54 @@ final class FlowSessionManager: ObservableObject { let message = AppL10n.string("flow.coldStart.error.audioTimeout") sessionWarning = message showColdStartAudioFailure(message: message) + scheduleColdStartRecovery(duration: nil) debug("cold-start blocked: host ready contract not published") return } - let hostEntry = HostReturnService.pendingHostEntry() - let skipSwitch = FlowSessionPolicy.skipAppSwitch() - coldStartContext = FlowColdStartContext(hostEntry: hostEntry, state: .ready) + presentColdStartReadyOverlay() + } - if skipSwitch, hostEntry != nil { - Task { @MainActor [weak self] in - try? await Task.sleep(nanoseconds: 450_000_000) - guard let self, self.coldStartContext?.state == .ready else { return } - if HostReturnService.openPendingHostIfPossible() { - self.dismissColdStartOverlay() - } + private func presentColdStartReadyOverlay() { + let hostEntry = HostReturnService.pendingHostEntry() + coldStartContext = FlowColdStartContext(hostEntry: hostEntry, state: .ready) + scheduleAutoReturnToHostIfNeeded(hostEntry: hostEntry) + } + + private func scheduleAutoReturnToHostIfNeeded(hostEntry: HostAppEntry?) { + let skipSwitch = FlowSessionPolicy.skipAppSwitch() + guard skipSwitch, hostEntry != nil else { return } + Task { @MainActor [weak self] in + try? await Task.sleep(nanoseconds: 450_000_000) + guard let self, self.coldStartContext?.state == .ready else { return } + if HostReturnService.openPendingHostIfPossible() { + self.dismissColdStartOverlay() } } } + /// Keeps proving mic readiness after the first timeout instead of tearing + /// capture down — many handoffs become ready a few seconds later. + private func scheduleColdStartRecovery(duration: TimeInterval?) { + coldStartRecoveryTask?.cancel() + coldStartRecoveryTask = Task { @MainActor [weak self] in + guard let self else { return } + let recovered = await self.capture.awaitAudioFlowing( + timeout: Self.coldStartRecoveryProofTimeout + ) + guard !Task.isCancelled else { return } + guard self.isColdStartHandoff else { return } + guard recovered else { return } + + self.sessionWarning = nil + self.traceState("coldStartRecovery.recovered") + if !self.isActive { + self.activateFlowSessionAfterAudioProof(duration: duration) + } + self.refreshHostReady() + } + } + private func showColdStartPreparing() { coldStartContext = FlowColdStartContext( hostEntry: HostReturnService.pendingHostEntry(), @@ -546,6 +702,14 @@ final class FlowSessionManager: ObservableObject { // MARK: - Polling + private func startCommandObserver() { + commandObserver = FlowSessionDarwinObserver( + notificationName: FlowSessionDarwin.commandNotificationName + ) { [weak self] in + self?.handleKeyboardSignal() + } + } + private func startPolling() { pollingTask?.cancel() lastObservedRecordingState = FlowSessionBridge.recordingState() @@ -556,46 +720,122 @@ final class FlowSessionManager: ObservableObject { pollingTask = Task { @MainActor [weak self] in while !Task.isCancelled { self?.handleKeyboardSignal() - try? await Task.sleep(nanoseconds: 50_000_000) + try? await Task.sleep(nanoseconds: 500_000_000) } } } private func handleKeyboardSignal() { - let signal = FlowSessionBridge.recordingState() - if signal != lastObservedRecordingState { - // The single most important cross-process signal: proves whether the - // host actually SEES the keyboard's recording state writes. - FlowDiagnostics.log( - "poll observed recordingState \(lastObservedRecordingState.rawValue) → \(signal.rawValue) " + - "[rec=\(isUtteranceRecording) proc=\(isUtteranceProcessing) fg=\(isAppForeground)]" - ) - lastObservedRecordingState = signal + guard let command = FlowSessionBridge.latestCommand() else { + lastCommandFingerprint = "" + return } - switch signal { - case .recording: + let fingerprint = "\(command.sessionId.uuidString)|\(command.utteranceId.uuidString)|\(command.action.rawValue)|\(command.commandSeq)" + guard fingerprint != lastCommandFingerprint else { return } + lastCommandFingerprint = fingerprint + handleFlowCommand(command) + } + + private func handleFlowCommand(_ command: FlowCommand) { + guard let activeSessionId, command.sessionId == activeSessionId else { + traceIgnoredCommand( + reason: "staleSession", + command: command, + detail: "commandSession=\(command.sessionId)" + ) + return + } + guard command.commandSeq > lastHandledCommandSeq else { + traceIgnoredCommand( + reason: "seqNotIncreasing", + command: command, + detail: "last=\(lastHandledCommandSeq)" + ) + return + } + lastHandledCommandSeq = command.commandSeq + lastIgnoredCommandSignature = "" + + FlowDiagnostics.log( + "command \(command.action.rawValue) seq=\(command.commandSeq) utterance=\(command.utteranceId)" + ) + + switch command.action { + case .startRecording: guard !isUtteranceRecording, !isUtteranceProcessing else { return } - beginUtterance() - case .stopped: + beginUtterance(utteranceId: command.utteranceId, commandSeq: command.commandSeq) + case .stopRecording: + guard currentUtteranceId == command.utteranceId else { return } if isUtteranceRecording { endUtterance() } else if !isUtteranceProcessing { - FlowSessionBridge.setRecordingState(.idle) - FlowSessionBridge.storeTranscriptionError( + storeCurrentError( AppL10n.string("flow.error.recognitionInterrupted"), kind: .recognitionInterrupted ) - debug("stopped without active utterance — notified keyboard") + debug("stop command without active utterance — notified keyboard") } - case .aborted: + case .abort: + guard currentUtteranceId == command.utteranceId else { return } abortUtterance() - case .idle, .processing: - break } } - private func beginUtterance() { - guard capture.engineIsLive else { + private func storeCurrentPartial(_ text: String) { + guard let activeSessionId, let currentUtteranceId else { return } + let trimmed = text.trimmingCharacters(in: .whitespacesAndNewlines) + guard !trimmed.isEmpty else { return } + FlowSessionBridge.writeResult( + FlowResult( + sessionId: activeSessionId, + utteranceId: currentUtteranceId, + commandSeq: currentCommandSeq, + status: .partial, + text: trimmed + ) + ) + } + + private func storeCurrentFinal(_ text: String, warning: String? = nil) { + let trimmed = text.trimmingCharacters(in: .whitespacesAndNewlines) + guard !trimmed.isEmpty else { + storeCurrentError(AppL10n.string("flow.error.noSpeech"), kind: .noSpeech) + return + } + guard let activeSessionId, let currentUtteranceId else { return } + FlowSessionBridge.writeResult( + FlowResult( + sessionId: activeSessionId, + utteranceId: currentUtteranceId, + commandSeq: currentCommandSeq, + status: .final, + text: trimmed, + warning: warning + ) + ) + } + + private func storeCurrentError( + _ message: String, + kind: FlowSessionKeys.TranscriptionErrorKind = .generic, + status: FlowResult.Status = .error + ) { + guard let activeSessionId, let currentUtteranceId else { return } + FlowSessionBridge.writeResult( + FlowResult( + sessionId: activeSessionId, + utteranceId: currentUtteranceId, + commandSeq: currentCommandSeq, + status: status, + text: message, + errorKind: kind + ) + ) + } + + private func beginUtterance(utteranceId: UUID? = nil, commandSeq: Int64 = 0) { + guard capture.engineHasRecentAudio(maxAge: 2) else { + traceState("beginUtterance.blocked", extra: "reason=audioNotRecent") failUtterance( message: AppL10n.string("flow.error.audioUnavailable"), kind: .audioUnavailable @@ -603,13 +843,25 @@ final class FlowSessionManager: ObservableObject { return } guard !isUtteranceProcessing else { + traceState("beginUtterance.ignored", extra: "reason=processing") debug("beginUtterance ignored — previous utterance still processing") return } + bindSessionASRIfNeeded() + let expectedEngine = store.engineMode + if sessionASREngineMode != expectedEngine { + traceState( + "beginUtterance.rebindMismatch", + extra: "expectedEngine=\(expectedEngine) boundEngine=\(sessionASREngineMode ?? "nil")" + ) + bindSessionASRIfNeeded(force: true) + } // Usually already warm from session start; refresh without blocking the mic gate. scheduleASRWarmup() + currentUtteranceId = utteranceId ?? UUID() + currentCommandSeq = commandSeq currentPartial = "" lastFinal = "" chunkWarnings = [] @@ -617,6 +869,7 @@ final class FlowSessionManager: ObservableObject { let localeId = store.localeId FlowSessionBridge.setTranscriptionLanguage(localeId) FlowSessionBridge.clearPendingTranscription() + FlowSessionBridge.clearResult() let locale = SpeechLocaleResolver.resolve(localeId) let stream = capture.beginUtterance() @@ -625,6 +878,7 @@ final class FlowSessionManager: ObservableObject { isUtteranceRecording = true utteranceRecordingStartedAt = Date() + startUtteranceSafetyTimer() refreshHostReady() FlowLiveActivityController.update(phase: .recording) FlowDiagnostics.log( @@ -637,8 +891,9 @@ final class FlowSessionManager: ObservableObject { asrTask = Task.detached(priority: .userInitiated) { [weak manager = self] in let outcome = await pipeline.transcribe(stream: stream) { partial in Task { @MainActor in - manager?.currentPartial = partial - FlowSessionBridge.storeTranscriptionPartial(partial) + guard let manager else { return } + manager.currentPartial = partial + manager.storeCurrentPartial(partial) } } // Re-bind `manager` inside the `@MainActor` block so the @@ -672,14 +927,33 @@ final class FlowSessionManager: ObservableObject { debug("utterance recording started") } + private func startUtteranceSafetyTimer() { + utteranceSafetyTask?.cancel() + let utteranceId = currentUtteranceId + utteranceSafetyTask = Task { @MainActor [weak self] in + let timeout = FlowSessionKeys.maxUtteranceDuration + 10 + try? await Task.sleep(nanoseconds: UInt64(timeout * 1_000_000_000)) + guard let self, !Task.isCancelled else { return } + guard self.isUtteranceRecording, self.currentUtteranceId == utteranceId else { return } + self.storeCurrentError( + AppL10n.string("flow.error.recognitionInterrupted"), + kind: .recognitionInterrupted, + status: .timeout + ) + self.abortUtterance() + self.debug("utterance safety timer aborted stale recording") + } + } + private func endUtterance() { guard isUtteranceRecording else { return } // Close the mic gate first, then mark processing before dropping the // recording flag so the poll loop cannot start a second utterance. - FlowSessionBridge.setRecordingState(.processing) isUtteranceRecording = false isUtteranceProcessing = true + utteranceSafetyTask?.cancel() + utteranceSafetyTask = nil refreshHostReady() FlowLiveActivityController.update(phase: .processing) @@ -699,6 +973,8 @@ final class FlowSessionManager: ObservableObject { isUtteranceRecording = false isUtteranceProcessing = false utteranceRecordingStartedAt = nil + utteranceSafetyTask?.cancel() + utteranceSafetyTask = nil finalizeTask?.cancel() finalizeTask = nil asrTask?.cancel() @@ -709,8 +985,8 @@ final class FlowSessionManager: ObservableObject { currentPartial = "" lastFinal = "" chunkWarnings = [] - FlowSessionBridge.storeTranscriptionPartial("") - FlowSessionBridge.setRecordingState(.idle) + currentUtteranceId = nil + currentCommandSeq = 0 FlowLiveActivityController.update(phase: .idle) refreshHostReady() debug("utterance aborted") @@ -723,6 +999,8 @@ final class FlowSessionManager: ObservableObject { isUtteranceRecording = false isUtteranceProcessing = false utteranceRecordingStartedAt = nil + utteranceSafetyTask?.cancel() + utteranceSafetyTask = nil finalizeTask?.cancel() finalizeTask = nil asrTask?.cancel() @@ -733,9 +1011,9 @@ final class FlowSessionManager: ObservableObject { currentPartial = "" lastFinal = "" chunkWarnings = [] - FlowSessionBridge.storeTranscriptionPartial("") - FlowSessionBridge.storeTranscriptionError(message, kind: kind) - FlowSessionBridge.setRecordingState(.idle) + storeCurrentError(message, kind: kind) + currentUtteranceId = nil + currentCommandSeq = 0 FlowLiveActivityController.update(phase: .idle) refreshHostReady() debug("utterance failed: \(message)") @@ -747,28 +1025,36 @@ final class FlowSessionManager: ObservableObject { ) { isUtteranceProcessing = false utteranceRecordingStartedAt = nil + utteranceSafetyTask?.cancel() + utteranceSafetyTask = nil finalizeTask?.cancel() finalizeTask = nil chunkedPipeline = nil currentPartial = "" lastFinal = "" chunkWarnings = [] - FlowSessionBridge.storeTranscriptionPartial("") - FlowSessionBridge.storeTranscriptionError(message, kind: kind) - FlowSessionBridge.setRecordingState(.idle) + storeCurrentError(message, kind: kind) + currentUtteranceId = nil + currentCommandSeq = 0 FlowLiveActivityController.update(phase: .idle) refreshHostReady() debug("utterance processing failed: \(message)") } private func finalizeUtterance() async { + let finalizeSessionId = activeSessionId + let finalizeUtteranceId = currentUtteranceId let pipelineStarted = Date() defer { - isUtteranceProcessing = false - FlowSessionBridge.setRecordingState(.idle) - FlowLiveActivityController.update(phase: .idle) - touchSessionActivity() - refreshHostReady() + if activeSessionId == finalizeSessionId, + currentUtteranceId == finalizeUtteranceId { + isUtteranceProcessing = false + FlowLiveActivityController.update(phase: .idle) + touchSessionActivity() + currentUtteranceId = nil + currentCommandSeq = 0 + refreshHostReady() + } } let asrWait = asrWaitTimeout() @@ -783,9 +1069,10 @@ final class FlowSessionManager: ObservableObject { try? await Task.sleep(nanoseconds: 100_000_000) } - if lastFinal.isEmpty, let asrTask { - FlowDiagnostics.log("ASR wait elapsed — awaiting asrTask completion") - _ = await asrTask.value + if lastFinal.isEmpty { + FlowDiagnostics.log("ASR wait elapsed — cancelling ASR task and using best available transcript") + asrTask?.cancel() + Task { await chunkedPipeline?.cancel() } } let asrElapsed = Date().timeIntervalSince(pipelineStarted) @@ -803,10 +1090,7 @@ final class FlowSessionManager: ObservableObject { (asrTask?.isCancelled == true) ? .recognitionInterrupted : .noSpeech FlowDiagnostics.log("finalize failed: empty transcript after \(String(format: "%.1f", asrElapsed))s") utteranceRecordingStartedAt = nil - FlowSessionBridge.storeTranscriptionError( - AppL10n.string(key), - kind: kind - ) + storeCurrentError(AppL10n.string(key), kind: kind) return } @@ -832,7 +1116,7 @@ final class FlowSessionManager: ObservableObject { providerIdOverride: pipelineStore.polishProviderIdOverride ) delivered = polished - FlowSessionBridge.storeTranscriptionResult(polished, polishWarning: chunkNote) + storeCurrentFinal(polished, warning: chunkNote) FlowDiagnostics.log( "polish done in \(String(format: "%.1f", Date().timeIntervalSince(polishStarted)))s " + "total=\(String(format: "%.1f", Date().timeIntervalSince(pipelineStarted)))s" @@ -854,7 +1138,7 @@ final class FlowSessionManager: ObservableObject { "\(error.localizedDescription)" ) delivered = fallback.text - FlowSessionBridge.storeTranscriptionResult(fallback.text, polishWarning: fallback.polishWarning) + storeCurrentFinal(fallback.text, warning: fallback.polishWarning) } SpeechHistoryStore.shared.recordUtterance( @@ -867,7 +1151,6 @@ final class FlowSessionManager: ObservableObject { currentPartial = "" lastFinal = "" chunkWarnings = [] - FlowSessionBridge.storeTranscriptionPartial("") chunkedPipeline = nil debug("utterance finalized length=\(text.count)") } @@ -964,6 +1247,12 @@ final class FlowSessionManager: ObservableObject { heartbeatTask?.cancel() FlowSessionBridge.writeHeartbeat() heartbeatTask = Task { @MainActor [weak self] in + // Refresh the Live Activity `staleDate` every N heartbeat ticks + // (1 Hz) — well inside `FlowLiveActivityController.staleWindow` so a + // live session never looks stale, while a force-quit stops these + // refreshes and lets the system reclaim the orphaned island. + let liveActivityKeepAliveEveryTicks = 15 + var tick = 0 while !Task.isCancelled { guard let self else { break } if self.isActive, !self.capture.engineIsLive { @@ -971,6 +1260,10 @@ final class FlowSessionManager: ObservableObject { } FlowSessionBridge.writeHeartbeat() self.refreshHostReady() + tick += 1 + if tick % liveActivityKeepAliveEveryTicks == 0 { + FlowLiveActivityController.keepAlive() + } try? await Task.sleep(nanoseconds: 1_000_000_000) guard self.isActive else { break } } @@ -989,4 +1282,43 @@ final class FlowSessionManager: ObservableObject { private func debug(_ message: String) { FlowDiagnostics.log(message) } + + private func traceIgnoredCommand(reason: String, command: FlowCommand, detail: String) { + let signature = "\(reason)|\(command.action.rawValue)|\(command.commandSeq)|\(command.sessionId.uuidString)|\(command.utteranceId.uuidString)|\(detail)" + guard signature != lastIgnoredCommandSignature else { return } + lastIgnoredCommandSignature = signature + traceState( + "command.ignored", + extra: "reason=\(reason) action=\(command.action.rawValue) seq=\(command.commandSeq) \(detail)" + ) + } + + private func traceState(_ event: String, extra: String? = nil) { + let staleness = FlowSessionBridge.heartbeatStaleness().map { String(format: "%.1f", $0) } ?? "nil" + let sessionId = activeSessionId?.uuidString ?? "nil" + let utteranceId = currentUtteranceId?.uuidString ?? "nil" + let summary = [ + "event=\(event)", + "active=\(isActive)", + "starting=\(isStarting)", + "coldStart=\(isColdStartHandoff)", + "sessionId=\(sessionId)", + "utteranceId=\(utteranceId)", + "cmdSeq=\(currentCommandSeq)", + "lastCmd=\(lastHandledCommandSeq)", + "recording=\(isUtteranceRecording)", + "processing=\(isUtteranceProcessing)", + "storeEngine=\(store.engineMode)", + "boundEngine=\(sessionASREngineMode ?? "nil")", + "engineLive=\(capture.engineIsLive)", + "hostReady=\(FlowSessionBridge.isHostReady())", + "sessionActive=\(FlowSessionBridge.isSessionActive())", + "heartbeatStaleness=\(staleness)" + ].joined(separator: " ") + if let extra, !extra.isEmpty { + debug("[trace] \(summary) \(extra)") + } else { + debug("[trace] \(summary)") + } + } } diff --git a/OSGKeyboard/Views/FlowColdStartOverlay.swift b/OSGKeyboard/Views/FlowColdStartOverlay.swift index 30f997c..7e09a04 100644 --- a/OSGKeyboard/Views/FlowColdStartOverlay.swift +++ b/OSGKeyboard/Views/FlowColdStartOverlay.swift @@ -38,6 +38,17 @@ struct FlowColdStartOverlay: View { /// Fraction of the screen height the bottom gradient occupies. private let gradientHeightFraction: CGFloat = 0.50 + /// Ready and failure states dismiss on blank tap; preparing stays + /// informational only (no accidental dismiss while proving audio). + private var allowsBlankTapDismiss: Bool { + switch context.state { + case .ready, .failed: + return true + case .preparing: + return false + } + } + var body: some View { GeometryReader { geo in ZStack(alignment: .bottom) { @@ -56,6 +67,16 @@ struct FlowColdStartOverlay: View { .frame(maxWidth: .infinity, alignment: .bottom) .allowsHitTesting(false) + if allowsBlankTapDismiss { + // Captures taps on empty overlay space (dismiss) and blocks + // pass-through to the host shell underneath. Action buttons in + // `content` sit above this layer and remain tappable. + Color.clear + .contentShape(Rectangle()) + .onTapGesture(perform: onDismiss) + .ignoresSafeArea() + } + VStack(spacing: Spacing.lg) { content .padding(.horizontal, Spacing.xl) @@ -63,14 +84,6 @@ struct FlowColdStartOverlay: View { homeIndicator .padding(.bottom, max(geo.safeAreaInsets.bottom, Spacing.sm)) } - .allowsHitTesting(false) - - if context.state == .ready { - Color.clear - .contentShape(Rectangle()) - .onTapGesture(perform: onDismiss) - .ignoresSafeArea() - } } .frame(maxWidth: .infinity, maxHeight: .infinity, alignment: .bottom) .ignoresSafeArea() diff --git a/OSGKeyboardExt/Services/KeyboardFlowCoordinator.swift b/OSGKeyboardExt/Services/KeyboardFlowCoordinator.swift index 3470fb6..1286a5e 100644 --- a/OSGKeyboardExt/Services/KeyboardFlowCoordinator.swift +++ b/OSGKeyboardExt/Services/KeyboardFlowCoordinator.swift @@ -42,6 +42,10 @@ final class KeyboardFlowCoordinator { private static let hostReadyGrace: TimeInterval = 4 private var flowSessionMonitorTask: Task? private var isAwaitingFlowResult = false + private var activeSessionId: UUID? + private var currentUtteranceId: UUID? + private var currentCommandSeq: Int64 = 0 + private var lastAvailabilityTraceSignature = "" init( state: KeyboardState, @@ -133,7 +137,9 @@ final class KeyboardFlowCoordinator { private func recomputeMicVoiceAvailability() { FlowSessionBridge.reloadFromDisk() - let hostReady = FlowSessionBridge.isHostReady() + let readySnapshot = FlowSessionBridge.readySnapshot() + activeSessionId = readySnapshot?.sessionId ?? activeSessionId + let hostReady = readySnapshot?.ready == true && FlowSessionBridge.isHostReady() let now = Date().timeIntervalSince1970 if hostReady { lastHostReadyAt = now } // Grace window: the host was ready very recently, so treat a momentary @@ -145,7 +151,7 @@ final class KeyboardFlowCoordinator { let hostWarming = !hostReady && FlowSessionBridge.isSessionActive() && (FlowSessionBridge.isHostReachable() || isPendingFlowStart || withinReadyGrace) - state.flowSessionActive = hostReady + state.flowSessionActive = FlowSessionBridge.isSessionActive() state.micVoiceAvailability = MicVoiceAvailabilityResolver.resolve( phase: state.phase, micDisabled: state.micDisabled, @@ -154,6 +160,20 @@ final class KeyboardFlowCoordinator { hostReady: hostReady, isPreparingSession: isPendingFlowStart || hostWarming ) + let signature = [ + "phase=\(String(describing: state.phase))", + "availability=\(String(describing: state.micVoiceAvailability))", + hostReady ? "hostReady=1" : "hostReady=0", + state.flowSessionActive ? "sessionActive=1" : "sessionActive=0", + isPendingFlowStart ? "pending=1" : "pending=0", + isFlowRecording ? "recording=1" : "recording=0", + isAwaitingFlowResult ? "awaiting=1" : "awaiting=0", + readySnapshot?.reason.rawValue ?? "snapshot=nil" + ].joined(separator: "|") + if signature != lastAvailabilityTraceSignature { + lastAvailabilityTraceSignature = signature + traceState("availability.update", extra: signature) + } } /// Session is live but the ready contract has not landed yet — poll @@ -228,7 +248,8 @@ final class KeyboardFlowCoordinator { scheduleAutoClearError() recomputeMicVoiceAvailability() case .unavailable(.preparingSession): - return + detectAndStoreAppContext() + beginFlowStart() case .unavailable(.hostNotReady): detectAndStoreAppContext() beginFlowStart() @@ -247,8 +268,8 @@ final class KeyboardFlowCoordinator { isFlowRecording = false stopUtteranceCountdown() ExtensionScreenWakeLock.release() - FlowSessionBridge.setRecordingState(.stopped) - debug("pressEnded wrote .stopped (readback=\(FlowSessionBridge.recordingState().rawValue))") + writeCommand(.stopRecording) + debug("pressEnded wrote stop command") state.phase = .processing state.lastTranscript = ExtL10n.string("keyboard.flow.transcribing") recomputeMicVoiceAvailability() @@ -256,7 +277,10 @@ final class KeyboardFlowCoordinator { } func beginFlowStart() { - guard !isPendingFlowStart else { return } + guard !isPendingFlowStart else { + traceState("beginFlowStart.ignored", extra: "reason=pendingAlreadyTrue") + return + } isPendingFlowStart = true isFlowRecording = false flowStartDeadline = Date().timeIntervalSince1970 + FlowWatchdog.startTimeout @@ -264,11 +288,11 @@ final class KeyboardFlowCoordinator { recomputeMicVoiceAvailability() openHostApp("startflow") startFlowStartWatchdog() - debug("beginFlowStart") + traceState("beginFlowStart.started") } func handleHostAppOpenResult(path: String, success: Bool) { - debug("openHostApp path=\(path) success=\(success)") + traceState("openHostApp.result", extra: "path=\(path) success=\(success)") guard !success else { return } // The open genuinely failed (iOS blocked it / no Full Access). Don't @@ -278,6 +302,7 @@ final class KeyboardFlowCoordinator { isPendingFlowStart = false flowStartDeadline = 0 stopFlowWatchdog() + traceState("openHostApp.failed", extra: "path=startflow cancelPending=1") showManualOpenHint(path: "startflow") recomputeMicVoiceAvailability() return @@ -290,9 +315,10 @@ final class KeyboardFlowCoordinator { guard !isAwaitingFlowResult else { return } if isFlowRecording || isPendingFlowStart { if isFlowRecording { - FlowSessionBridge.setRecordingState(.aborted) + writeCommand(.abort) ExtensionScreenWakeLock.release() } + currentUtteranceId = nil isFlowRecording = false isPendingFlowStart = false stopUtteranceCountdown() @@ -304,17 +330,56 @@ final class KeyboardFlowCoordinator { // MARK: - Private + private func nextCommandSeq() -> Int64 { + let millis = Int64(Date().timeIntervalSince1970 * 1_000) + currentCommandSeq = max(currentCommandSeq + 1, millis) + return currentCommandSeq + } + + private func writeCommand(_ action: FlowCommand.Action) { + guard let activeSessionId, let currentUtteranceId else { return } + let command = FlowCommand( + sessionId: activeSessionId, + utteranceId: currentUtteranceId, + commandSeq: nextCommandSeq(), + action: action, + localeId: state.localeId + ) + FlowSessionBridge.writeCommand(command) + debug( + "command \(action.rawValue) seq=\(command.commandSeq) " + + "utterance=\(currentUtteranceId.uuidString)" + ) + } + private func consumePendingFlowDeliveryIfNeeded() { if isAwaitingFlowResult { - if let delivery = FlowSessionBridge.consumeTranscriptionDelivery() { + if let result = matchingResult(), result.status == .final, let text = result.text, !text.isEmpty { isAwaitingFlowResult = false stopFlowWatchdog() - textInserter.handleFlowTranscript(delivery) + FlowSessionBridge.writeAck( + FlowAck( + sessionId: result.sessionId, + utteranceId: result.utteranceId, + commandSeq: result.commandSeq + ) + ) + FlowSessionBridge.clearResult() + currentUtteranceId = nil + textInserter.handleFlowTranscript( + TranscriptionDelivery(text: text, polishWarning: result.warning) + ) return } - if let error = FlowSessionBridge.consumeTranscriptionError() { + if let result = matchingResult(), isTerminalFailure(result) { isAwaitingFlowResult = false stopFlowWatchdog() + FlowSessionBridge.clearResult() + currentUtteranceId = nil + let error = FlowTranscriptionError( + message: result.text ?? ExtL10n.string("keyboard.flow.resultTimeout"), + kind: result.errorKind ?? .generic + ) state.phase = .error( .fromFlowTranscription(error), message: error.message @@ -330,6 +395,20 @@ final class KeyboardFlowCoordinator { } } + private func matchingResult() -> FlowResult? { + guard let result = FlowSessionBridge.latestResult() else { return nil } + guard let activeSessionId, let currentUtteranceId else { return nil } + guard result.sessionId == activeSessionId, + result.utteranceId == currentUtteranceId else { + return nil + } + return result + } + + private func isTerminalFailure(_ result: FlowResult) -> Bool { + result.status == .error || result.status == .timeout || result.status == .aborted + } + /// When the host process died mid-utterance, abort local recording / waiting /// so the user is not stuck until the long result watchdog fires. private func recoverFromDeadHostIfNeeded() { @@ -339,7 +418,8 @@ final class KeyboardFlowCoordinator { isFlowRecording = false stopUtteranceCountdown() ExtensionScreenWakeLock.release() - FlowSessionBridge.setRecordingState(.aborted) + writeCommand(.abort) + currentUtteranceId = nil stopFlowWatchdog() state.level = 0 state.phase = .idle @@ -355,12 +435,14 @@ final class KeyboardFlowCoordinator { } private func failHostDisconnected() { + traceState("hostDisconnected.fail") isAwaitingFlowResult = false isFlowRecording = false isPendingFlowStart = false stopUtteranceCountdown() ExtensionScreenWakeLock.release() - FlowSessionBridge.setRecordingState(.aborted) + writeCommand(.abort) + currentUtteranceId = nil stopFlowWatchdog() state.level = 0 let message = ExtL10n.string("keyboard.flow.hostDisconnected") @@ -396,6 +478,7 @@ final class KeyboardFlowCoordinator { private func startFlowRecording() { recomputeMicVoiceAvailability() guard state.micVoiceAvailability.isReady else { + traceState("startFlowRecording.blocked", extra: "availability=\(String(describing: state.micVoiceAvailability))") beginFlowStart() return } @@ -403,8 +486,14 @@ final class KeyboardFlowCoordinator { flowStartDeadline = 0 stopFlowWatchdog() - FlowSessionBridge.setTranscriptionLanguage(state.localeId) - FlowSessionBridge.setRecordingState(.recording) + guard let sessionId = FlowSessionBridge.readySnapshot()?.sessionId else { + traceState("startFlowRecording.blocked", extra: "reason=missingSessionIdInReadySnapshot") + beginFlowStart() + return + } + activeSessionId = sessionId + currentUtteranceId = UUID() + writeCommand(.startRecording) isFlowRecording = true state.lastTranscript = "" state.phase = .recording @@ -414,9 +503,7 @@ final class KeyboardFlowCoordinator { } startUtteranceCountdown() startFlowLevelWatchdog() - // Read back in-process to confirm the write landed before we rely on - // the host polling it out cross-process. - debug("startFlowRecording wrote .recording (readback=\(FlowSessionBridge.recordingState().rawValue))") + traceState("startFlowRecording.started") } private func startUtteranceCountdown() { @@ -450,6 +537,7 @@ final class KeyboardFlowCoordinator { state.phase = .idle state.lastTranscript = "" recomputeMicVoiceAvailability() + traceState("pendingStart.cancelledByUser") } private func startFlowStartWatchdog() { @@ -465,6 +553,7 @@ final class KeyboardFlowCoordinator { if self.flowStartDeadline > 0, now > self.flowStartDeadline { self.isPendingFlowStart = false self.flowStartDeadline = 0 + self.traceState("startWatchdog.timeout") self.showManualOpenHint(path: "startflow") return } @@ -480,7 +569,7 @@ final class KeyboardFlowCoordinator { state.lastTranscript = "" refreshSessionState() startFlowRecording() - debug("completeFlowStartHandoff → auto startFlowRecording") + traceState("completeFlowStartHandoff.done") } private func startFlowLevelWatchdog() { @@ -492,6 +581,12 @@ final class KeyboardFlowCoordinator { self.state.level = Double(peak) } self.refreshFlowPartialIfNeeded() + let staleness = FlowSessionBridge.heartbeatStaleness() ?? .infinity + if staleness > 5 { + self.debug("levelWatchdog: host heartbeat stale while recording") + self.failHostDisconnected() + return + } try? await Task.sleep(nanoseconds: FlowWatchdog.pollIntervalNs) } } @@ -501,7 +596,10 @@ final class KeyboardFlowCoordinator { guard isFlowRecording || isAwaitingFlowResult else { return } switch state.phase { case .recording, .processing: - if let partial = FlowSessionBridge.transcriptionPartial() { + if let result = matchingResult(), + result.status == .partial, + let partial = result.text, + !partial.isEmpty { state.lastTranscript = partial } default: @@ -517,16 +615,33 @@ final class KeyboardFlowCoordinator { debug("resultWatchdog started timeout=\(Int(resultTimeout))s engine=\(state.engineMode)") flowWatchdogTask = Task { @MainActor [weak self] in while let self, !Task.isCancelled { - if let delivery = FlowSessionBridge.consumeTranscriptionDelivery() { + if let result = self.matchingResult(), result.status == .final, let text = result.text, !text.isEmpty { self.isAwaitingFlowResult = false self.stopFlowWatchdog() - self.debug("resultWatchdog consumed delivery len=\(delivery.text.count)") - self.textInserter.handleFlowTranscript(delivery) + FlowSessionBridge.writeAck( + FlowAck( + sessionId: result.sessionId, + utteranceId: result.utteranceId, + commandSeq: result.commandSeq + ) + ) + FlowSessionBridge.clearResult() + self.currentUtteranceId = nil + self.debug("resultWatchdog consumed delivery len=\(text.count)") + self.textInserter.handleFlowTranscript( + TranscriptionDelivery(text: text, polishWarning: result.warning) + ) return } - if let error = FlowSessionBridge.consumeTranscriptionError() { + if let result = self.matchingResult(), self.isTerminalFailure(result) { self.isAwaitingFlowResult = false self.stopFlowWatchdog() + FlowSessionBridge.clearResult() + self.currentUtteranceId = nil + let error = FlowTranscriptionError( + message: result.text ?? ExtL10n.string("keyboard.flow.resultTimeout"), + kind: result.errorKind ?? .generic + ) self.debug("resultWatchdog consumed error kind=\(error.kind.rawValue)") self.state.phase = .error( .fromFlowTranscription(error), @@ -539,6 +654,11 @@ final class KeyboardFlowCoordinator { self.refreshFlowPartialIfNeeded() let now = Date().timeIntervalSince1970 let staleness = FlowSessionBridge.heartbeatStaleness() ?? .infinity + if self.isFlowRecording, staleness > 5 { + self.debug("level/result watchdog: host heartbeat stale while recording") + self.failHostDisconnected() + return + } if staleness > FlowSessionKeys.heartbeatZombieInterval { self.debug("resultWatchdog: host heartbeat zombie (staleness=\(String(format: "%.1f", staleness))s)") self.failHostDisconnected() @@ -553,6 +673,7 @@ final class KeyboardFlowCoordinator { if now - startedAt > resultTimeout { self.isAwaitingFlowResult = false self.stopFlowWatchdog() + self.currentUtteranceId = nil self.debug("resultWatchdog TIMEOUT after \(Int(resultTimeout))s — no result from host") let msg = ExtL10n.string("keyboard.flow.resultTimeout") self.state.phase = .error(.flowResultTimeout, message: msg) @@ -573,4 +694,29 @@ final class KeyboardFlowCoordinator { private func debug(_ message: String) { OSGLog.keyboardExt.info("\(message, privacy: .public)") } + + private func traceState(_ event: String, extra: String? = nil) { + let staleness = FlowSessionBridge.heartbeatStaleness().map { String(format: "%.1f", $0) } ?? "nil" + let sessionId = activeSessionId?.uuidString ?? "nil" + let utteranceId = currentUtteranceId?.uuidString ?? "nil" + let summary = [ + "event=\(event)", + "phase=\(String(describing: state.phase))", + "availability=\(String(describing: state.micVoiceAvailability))", + "pending=\(isPendingFlowStart)", + "recording=\(isFlowRecording)", + "awaiting=\(isAwaitingFlowResult)", + "sessionId=\(sessionId)", + "utteranceId=\(utteranceId)", + "cmdSeq=\(currentCommandSeq)", + "sessionActive=\(FlowSessionBridge.isSessionActive())", + "hostReady=\(FlowSessionBridge.isHostReady())", + "heartbeatStaleness=\(staleness)" + ].joined(separator: " ") + if let extra, !extra.isEmpty { + debug("[trace] \(summary) \(extra)") + } else { + debug("[trace] \(summary)") + } + } } diff --git a/OSGKeyboardLiveActivity/FlowLiveActivityWidget.swift b/OSGKeyboardLiveActivity/FlowLiveActivityWidget.swift index c3af7cb..f8a3f74 100644 --- a/OSGKeyboardLiveActivity/FlowLiveActivityWidget.swift +++ b/OSGKeyboardLiveActivity/FlowLiveActivityWidget.swift @@ -52,7 +52,7 @@ private struct FlowLiveActivityLockScreenView: View { var body: some View { HStack(spacing: 12) { - FlowLiveActivityBrandMark(height: 18) + FlowLiveActivityBrandMark(height: 13) VStack(alignment: .leading, spacing: 4) { Text("OSGKeyboard") .font(.headline) @@ -99,9 +99,10 @@ private struct FlowLiveActivityTrailingGlyph: View { .foregroundStyle(.red) .symbolEffect(.variableColor.iterative, options: .repeating) case .processing: - ProgressView() - .progressViewStyle(.circular) - .tint(.white) + Image(systemName: "ellipsis") + .font(.title3.weight(.semibold)) + .foregroundStyle(.white) + .symbolEffect(.variableColor.iterative, options: .repeating) case .idle: // Session ready but NOT listening — avoid a mic glyph so users // don't think the keyboard is recording in the background. diff --git a/OSGKeyboardMac/DashboardView.swift b/OSGKeyboardMac/DashboardView.swift index 0fe37c6..ab2f4b1 100644 --- a/OSGKeyboardMac/DashboardView.swift +++ b/OSGKeyboardMac/DashboardView.swift @@ -183,37 +183,48 @@ struct BottomDictationBar: View { .fixedSize() } + // 麦克风按钮始终居中固定：录音时的波形放进按钮内部， + // “按停止”提示作为浮层显示在按钮上方，二者均不参与布局， + // 因此按下 Option 触发录音时按钮位置不会发生偏移。 private var recordControl: some View { - HStack(spacing: Spacing.sm) { - if viewModel.isRecording { - MiniWaveform(level: viewModel.audioLevel) + recordButton + .overlay(alignment: .top) { + if viewModel.isRecording { + Text(MacL10n.string("mac.record.pressStop", language: lang)) + .font(TypeStyle.caption) + .foregroundStyle(palette.textTertiary) + .fixedSize() + .offset(y: -22) + } } - Button(action: viewModel.toggleRecording) { - ZStack { - Circle() - .fill(viewModel.isRecording ? palette.recordRed : palette.accent) - .frame(width: 52, height: 52) - .macGlassSurface(in: Circle(), fillOpacity: 0.2) - .shadow( - color: (viewModel.isRecording ? palette.recordRed : palette.accent).opacity(0.5), - radius: pulse ? 14 : 6 - ) - Image(systemName: viewModel.isRecording ? "stop.fill" : "mic.fill") + } + + private var recordButton: some View { + Button(action: viewModel.toggleRecording) { + ZStack { + Circle() + .fill(viewModel.isRecording ? palette.recordRed : palette.accent) + .frame(width: 52, height: 52) + .macGlassSurface(in: Circle(), fillOpacity: 0.2) + .shadow( + color: (viewModel.isRecording ? palette.recordRed : palette.accent).opacity(0.5), + radius: pulse ? 14 : 6 + ) + if viewModel.isRecording { + // 与 iOS 一致：录音时在红色按钮内部显示实时波形 + MiniWaveform(level: viewModel.audioLevel, barCount: 4, tint: palette.textOnAccent) + } else { + Image(systemName: "mic.fill") .font(.system(size: 20, weight: .bold)) .foregroundStyle(palette.textOnAccent) } } - .buttonStyle(.plain) - .disabled(viewModel.isProcessing) - .onAppear { - withAnimation(.easeInOut(duration: 1.1).repeatForever(autoreverses: true)) { - pulse = true - } - } - if viewModel.isRecording { - Text(MacL10n.string("mac.record.pressStop", language: lang)) - .font(TypeStyle.caption) - .foregroundStyle(palette.textTertiary) + } + .buttonStyle(.plain) + .disabled(viewModel.isProcessing) + .onAppear { + withAnimation(.easeInOut(duration: 1.1).repeatForever(autoreverses: true)) { + pulse = true } } } diff --git a/OSGKeyboardMac/MacDictationPipeline.swift b/OSGKeyboardMac/MacDictationPipeline.swift index 0125768..6693584 100644 --- a/OSGKeyboardMac/MacDictationPipeline.swift +++ b/OSGKeyboardMac/MacDictationPipeline.swift @@ -31,9 +31,30 @@ enum MacDictationPipeline { let locale = Locale(identifier: store.localeId.isEmpty ? "zh-CN" : store.localeId) let raw: String + var localBias: LocalASRBiasPayload? if store.engineMode == "local" { - raw = try await MacLocalASRService.transcribe(samples: samples, locale: locale) + MacAppContextService.captureAndPersist(to: store) + let capabilities = MacLocalASRService.currentCapabilities() + let bias = LocalASRBiasAdapter.adapt( + LocalASRBiasRequest( + dictionary: store.personalDictionary, + locale: locale, + frontAppBundleId: MacAppContextService.frontmostBundleIdentifier(), + capabilities: capabilities + ) + ) + localBias = bias + LocalASRBiasDiagnosticsStore.save( + payload: bias, + modelId: MacLocalASRService.selectedModelDefinition()?.id, + backendLabel: MacLocalASRService.currentBackendLabel() + ) + raw = try await MacLocalASRService.transcribe( + samples: samples, + locale: locale, + bias: bias + ) } else { let strategy = CloudASRModelCatalog.strategy(for: store.providerId) guard strategy != .localFallback else { throw MacDictationError.providerHasNoCloudASR } @@ -51,13 +72,33 @@ enum MacDictationPipeline { let trimmed = raw.trimmingCharacters(in: .whitespacesAndNewlines) guard !trimmed.isEmpty else { throw MacDictationError.emptyTranscript } + let postASR: String + if let localBias, !localBias.correctionPairs.isEmpty { + postASR = LocalASRTranscriptCorrector.apply(trimmed, pairs: localBias.correctionPairs) + } else { + postASR = trimmed + } + + let polishContext: PolishContext? + if let supplement = localBias?.polishFragment.trimmingCharacters(in: .whitespacesAndNewlines), + !supplement.isEmpty { + polishContext = PolishContext( + appContext: store.detectedAppContext?.context ?? .unknown, + intensity: store.polishIntensity, + dictionarySupplement: supplement + ) + } else { + polishContext = nil + } + if let polished = try? await PolishingService(store: store).polish( - trimmed, - mode: store.polishModeForPipeline + postASR, + mode: store.polishModeForPipeline, + context: polishContext ), !polished.isEmpty { return polished } - return trimmed + return postASR } } diff --git a/OSGKeyboardMac/MacDictationViewModel.swift b/OSGKeyboardMac/MacDictationViewModel.swift index c801711..09f3786 100644 --- a/OSGKeyboardMac/MacDictationViewModel.swift +++ b/OSGKeyboardMac/MacDictationViewModel.swift @@ -107,8 +107,9 @@ final class MacDictationViewModel: ObservableObject { /// Pre-load MLX weights + Metal shaders so the first dictation is fast. func warmUpQwen3IfNeeded() { guard config.engineMode == "local", - MacLocalASRPreferences.backend == .qwen3MLX, - MacLocalASRPreferences.qwen3ModelIsInstalled() else { return } + let model = MacLocalASRService.selectedModelDefinition(), + model.backend == .mlx, + MacLocalASRService.isModelInstalled(model) else { return } let path = MacLocalASRPreferences.qwen3ModelPath Task.detached(priority: .utility) { _ = try? await MacQwen3ASREngine.shared.prepareIfNeeded(modelPath: path) @@ -156,6 +157,39 @@ final class MacDictationViewModel: ObservableObject { return "\(seconds)s" } + var localModelReady: Bool { + _ = localModelRevision + if let model = MacLocalASRService.selectedModelDefinition() { + return MacLocalASRService.isModelInstalled(model) + } + return MacLocalASRPreferences.qwen3ModelIsInstalled() + } + + /// Context-aware warning when local engine is selected but the active model is not ready. + var localModelWarningMessage: String? { + _ = localModelRevision + guard config.engineMode == "local" else { return nil } + if localModelReady { return nil } + guard let model = MacLocalASRService.selectedModelDefinition() else { + return MacL10n.string("mac.settings.localModelFallbackApple", language: config.uiLanguage) + } + if model.installKind == .manual { + return MacL10n.string("mac.settings.mlxModelMissing", language: config.uiLanguage) + } + return MacL10n.format( + "mac.settings.selectedModelMissing", + language: config.uiLanguage, + model.displayName + ) + } + + @Published private(set) var localModelRevision = 0 + + func bumpLocalModelRevision() { + localModelRevision += 1 + objectWillChange.send() + } + var qwen3ModelInstalled: Bool { MacLocalASRPreferences.qwen3ModelIsInstalled() } diff --git a/OSGKeyboardMac/MacLocalASRModelSettingsView.swift b/OSGKeyboardMac/MacLocalASRModelSettingsView.swift new file mode 100644 index 0000000..fc10bfd --- /dev/null +++ b/OSGKeyboardMac/MacLocalASRModelSettingsView.swift @@ -0,0 +1,455 @@ +// MacLocalASRModelSettingsView.swift +// OSGKeyboard · Mac +// +// Local ASR model catalog, download progress, MLX path, and bias diagnostics. + +import AppKit +import SwiftUI + +@MainActor +final class MacLocalASRModelSettingsViewModel: ObservableObject { + @Published var catalog: LocalASRCatalogDocument? + @Published var selectedModelId: String = MacLocalASRPreferences.selectedModelId + @Published var installProgress = LocalASRModelInstallProgress.idle + @Published var diagnosticsSnapshot = LocalASRBiasDiagnosticsStore.load() + @Published var statusMessage = "" + @Published var isInstalling = false + @Published var isDownloadPaused = false + + var onLocalModelStateChanged: (() -> Void)? + + private let manager = LocalASRModelManager.shared + private var progressPollTask: Task? + + deinit { + progressPollTask?.cancel() + } + + func reload() { + catalog = try? LocalASRModelCatalog.loadBundled() + if let catalog { + let manifest = LocalASRInstalledManifestIO.load(defaultModelId: catalog.defaultModelId) + selectedModelId = manifest.selectedModelId.isEmpty + ? MacLocalASRPreferences.selectedModelId + : manifest.selectedModelId + } + diagnosticsSnapshot = LocalASRBiasDiagnosticsStore.load() + onLocalModelStateChanged?() + } + + func isInstalled(_ model: LocalASRModelDefinition) -> Bool { + MacLocalASRService.isModelInstalled(model) + } + + func isInstallingModel(_ model: LocalASRModelDefinition) -> Bool { + isInstalling && installProgress.activeItemId == model.id + } + + func installedDiskUsage(_ model: LocalASRModelDefinition) -> String? { + guard model.installKind == .archive, + let relative = model.installRelativePath, + isInstalled(model) else { return nil } + let dir = LocalASRModelInstallState.installDirectory(for: relative) + let bytes = LocalASRModelInstallState.directoryByteCount(at: dir) + guard bytes > 0 else { return nil } + return ByteCountFormatter.string(fromByteCount: bytes, countStyle: .file) + } + + func currentRuntime(in catalog: LocalASRCatalogDocument) -> LocalASRRuntimeDefinition? { + LocalASRModelCatalog.runtime(for: LocalASRModelCatalog.currentRuntimePlatform(), in: catalog) + } + + func isRuntimeInstalled(_ runtime: LocalASRRuntimeDefinition) -> Bool { + LocalASRModelInstallState.isRuntimeInstalled(runtime) + } + + func selectModel(_ modelId: String) { + guard let catalog, !isInstalling else { return } + selectedModelId = modelId + MacLocalASRPreferences.selectedModelId = modelId + var manifest = LocalASRInstalledManifestIO.load(defaultModelId: catalog.defaultModelId) + manifest.selectedModelId = modelId + manifest.updatedAt = Date() + try? LocalASRInstalledManifestIO.save(manifest) + onLocalModelStateChanged?() + } + + func installModel(_ model: LocalASRModelDefinition) { + guard let catalog, !isInstalling else { return } + statusMessage = "" + isInstalling = true + isDownloadPaused = false + startProgressPolling() + Task { + do { + try await manager.installModel(model, catalog: catalog) + installProgress = await manager.currentProgress() + selectModel(model.id) + statusMessage = MacL10n.string("mac.localASR.installDone") + } catch { + installProgress = await manager.currentProgress() + statusMessage = error.localizedDescription + } + isInstalling = false + isDownloadPaused = false + stopProgressPolling() + reload() + } + } + + func pauseDownload() { + Task { + do { + try await manager.pauseDownload() + isDownloadPaused = true + installProgress = await manager.currentProgress() + } catch { + statusMessage = error.localizedDescription + } + } + } + + func resumeDownload() { + Task { + do { + try await manager.resumeDownload() + isDownloadPaused = false + installProgress = await manager.currentProgress() + } catch { + statusMessage = error.localizedDescription + } + } + } + + func deleteModel(_ model: LocalASRModelDefinition) { + guard let catalog, !isInstalling else { return } + Task { + do { + try await manager.deleteModel(model, catalog: catalog) + statusMessage = MacL10n.string("mac.localASR.deleteDone") + reload() + } catch { + statusMessage = error.localizedDescription + } + } + } + + func revealModelInFinder(_ model: LocalASRModelDefinition) { + guard let relative = model.installRelativePath else { return } + let url = LocalASRModelInstallState.installDirectory(for: relative) + NSWorkspace.shared.activateFileViewerSelecting([url]) + } + + /// Opens (creating if needed) the model's shared subfolder so the user can + /// drop in manually-converted weights (used by the MLX model). + func revealModelFolder(_ model: LocalASRModelDefinition) { + guard let relative = model.installRelativePath else { return } + let url = LocalASRModelInstallState.installDirectory(for: relative) + try? FileManager.default.createDirectory(at: url, withIntermediateDirectories: true) + NSWorkspace.shared.open(url) + } + + func revealStorageRoot() { + let url = LocalASRModelInstallState.rootDirectory() + try? FileManager.default.createDirectory(at: url, withIntermediateDirectories: true) + NSWorkspace.shared.open(url) + } + + func progressLabel(for progress: LocalASRModelInstallProgress, language: AppUILanguage) -> String { + let phaseKey: String + switch progress.phase { + case .downloading: phaseKey = "mac.localASR.phase.downloading" + case .paused: phaseKey = "mac.localASR.phase.paused" + case .extracting: phaseKey = "mac.localASR.phase.extracting" + case .validating: phaseKey = "mac.localASR.phase.validating" + case .finalizing: phaseKey = "mac.localASR.phase.finalizing" + case .failed: phaseKey = "mac.localASR.phase.failed" + case .completed: phaseKey = "mac.localASR.phase.completed" + case .idle: return progress.message + } + let phase = MacL10n.string(phaseKey, language: language) + if let received = progress.bytesReceived, let total = progress.bytesTotal, total > 0 { + let recv = ByteCountFormatter.string(fromByteCount: received, countStyle: .file) + let tot = ByteCountFormatter.string(fromByteCount: total, countStyle: .file) + return "\(phase) · \(progress.message) (\(recv) / \(tot))" + } + return "\(phase) · \(progress.message)" + } + + private func startProgressPolling() { + progressPollTask?.cancel() + progressPollTask = Task { [weak self] in + while !Task.isCancelled { + guard let self else { return } + let current = await manager.currentProgress() + await MainActor.run { + self.installProgress = current + self.isDownloadPaused = current.phase == .paused + } + try? await Task.sleep(for: .milliseconds(120)) + } + } + } + + private func stopProgressPolling() { + progressPollTask?.cancel() + progressPollTask = nil + } + + func formattedSize(_ bytes: Int) -> String { + ByteCountFormatter.string(fromByteCount: Int64(bytes), countStyle: .file) + } +} + +struct MacLocalASRModelSettingsView: View { + @ObservedObject var viewModel: MacDictationViewModel + @StateObject private var modelVM = MacLocalASRModelSettingsViewModel() + @Environment(\.themePalette) private var palette + + private var lang: AppUILanguage { viewModel.config.uiLanguage } + + var body: some View { + Group { + if let catalog = modelVM.catalog { + modelPickerSection(catalog: catalog) + runtimeSection(catalog: catalog) + } else { + Text(MacL10n.string("mac.localASR.catalogMissing", language: lang)) + .foregroundStyle(palette.textSecondary) + } + } + .onAppear { + modelVM.onLocalModelStateChanged = { viewModel.bumpLocalModelRevision() } + modelVM.reload() + } + } + + private func modelPickerSection(catalog: LocalASRCatalogDocument) -> some View { + Section { + ForEach(catalog.models) { model in + modelRow(model) + } + + if modelVM.isInstalling, + modelVM.installProgress.phase == .extracting + || modelVM.installProgress.phase == .validating + || modelVM.installProgress.phase == .finalizing { + ProgressView(value: modelVM.installProgress.fraction) { + Text(modelVM.progressLabel(for: modelVM.installProgress, language: lang)) + .font(TypeStyle.caption) + } + } + + if !modelVM.statusMessage.isEmpty { + Text(modelVM.statusMessage) + .font(TypeStyle.caption) + .foregroundStyle(palette.textSecondary) + } + + Button(MacL10n.string("mac.localASR.openStorage", language: lang)) { + modelVM.revealStorageRoot() + } + } header: { + Text(MacL10n.string("mac.localASR.models", language: lang)) + } footer: { + Text(MacL10n.string("mac.localASR.modelsDesc", language: lang)) + .font(TypeStyle.caption) + .foregroundStyle(palette.textSecondary) + } + } + + private func runtimeSection(catalog: LocalASRCatalogDocument) -> some View { + Group { + if let runtime = modelVM.currentRuntime(in: catalog) { + Section { + LabeledContent(runtime.displayName) { + Text( + modelVM.isRuntimeInstalled(runtime) + ? MacL10n.string("mac.localASR.installed", language: lang) + : MacL10n.string("mac.localASR.notInstalled", language: lang) + ) + } + Text(MacL10n.string("mac.localASR.runtimeDesc", language: lang)) + .font(TypeStyle.caption) + .foregroundStyle(palette.textSecondary) + } header: { + Text(MacL10n.string("mac.localASR.runtime", language: lang)) + } + } + } + } + + @ViewBuilder + private func modelRow(_ model: LocalASRModelDefinition) -> some View { + let installed = modelVM.isInstalled(model) + let selected = modelVM.selectedModelId == model.id + let installing = modelVM.isInstallingModel(model) + + VStack(alignment: .leading, spacing: Spacing.xs) { + HStack(alignment: .top) { + Button { + modelVM.selectModel(model.id) + } label: { + HStack(spacing: Spacing.sm) { + Image(systemName: selected ? "largecircle.fill.circle" : "circle") + .foregroundStyle(selected ? palette.accent : palette.textTertiary) + VStack(alignment: .leading, spacing: 2) { + Text(model.displayName) + .foregroundStyle(palette.textPrimary) + Text(modelSubtitle(model, installed: installed)) + .font(TypeStyle.caption) + .foregroundStyle(palette.textSecondary) + } + } + } + .buttonStyle(.plain) + .disabled(modelVM.isInstalling) + + Spacer() + + modelRowActions(model: model, installed: installed, installing: installing) + } + } + .padding(.vertical, 2) + } + + @ViewBuilder + private func modelRowActions( + model: LocalASRModelDefinition, + installed: Bool, + installing: Bool + ) -> some View { + if installing { + HStack(spacing: Spacing.sm) { + circularInstallProgress(for: model) + if modelVM.installProgress.phase == .downloading + || modelVM.installProgress.phase == .paused { + Button { + if modelVM.isDownloadPaused { + modelVM.resumeDownload() + } else { + modelVM.pauseDownload() + } + } label: { + Image(systemName: modelVM.isDownloadPaused ? "play.fill" : "pause.fill") + .font(.system(size: 12, weight: .semibold)) + .frame(width: 28, height: 28) + } + .buttonStyle(.bordered) + .controlSize(.small) + .help( + modelVM.isDownloadPaused + ? MacL10n.string("mac.localASR.resume", language: lang) + : MacL10n.string("mac.localASR.pause", language: lang) + ) + } + } + } else if model.installKind == .manual { + Button(MacL10n.string("mac.localASR.openFolder", language: lang)) { + modelVM.revealModelFolder(model) + } + .buttonStyle(.bordered) + .controlSize(.small) + } else if installed { + Button(MacL10n.string("mac.localASR.delete", language: lang), role: .destructive) { + modelVM.deleteModel(model) + } + .buttonStyle(.bordered) + .controlSize(.small) + } else { + Button(MacL10n.string("mac.localASR.download", language: lang)) { + modelVM.installModel(model) + } + .buttonStyle(.borderedProminent) + .controlSize(.small) + } + } + + private func circularInstallProgress(for model: LocalASRModelDefinition) -> some View { + let fraction: Double = { + if modelVM.installProgress.phase == .downloading || modelVM.installProgress.phase == .paused, + let received = modelVM.installProgress.bytesReceived, + let total = modelVM.installProgress.bytesTotal, + total > 0 { + return min(1, max(0, Double(received) / Double(total))) + } + return modelVM.installProgress.fraction + }() + return ZStack { + Circle() + .stroke(palette.textTertiary.opacity(0.25), lineWidth: 3) + Circle() + .trim(from: 0, to: fraction) + .stroke(palette.accent, style: StrokeStyle(lineWidth: 3, lineCap: .round)) + .rotationEffect(.degrees(-90)) + .animation(.linear(duration: 0.15), value: fraction) + if modelVM.installProgress.phase == .paused { + Image(systemName: "pause.fill") + .font(.system(size: 10, weight: .bold)) + .foregroundStyle(palette.textSecondary) + } else { + Text("\(Int(fraction * 100))%") + .font(.system(size: 9, weight: .medium, design: .rounded)) + .foregroundStyle(palette.textSecondary) + } + } + .frame(width: 36, height: 36) + .accessibilityLabel(modelVM.progressLabel(for: modelVM.installProgress, language: lang)) + } + + private func modelSubtitle(_ model: LocalASRModelDefinition, installed: Bool) -> String { + let size = modelVM.formattedSize(model.sizeBytes) + let hotword = model.supportsHotwords + ? MacL10n.string("mac.localASR.hotwordsYes", language: lang) + : MacL10n.string("mac.localASR.hotwordsNo", language: lang) + let state = installed + ? MacL10n.string("mac.localASR.installed", language: lang) + : MacL10n.string("mac.localASR.notInstalled", language: lang) + if let usage = modelVM.installedDiskUsage(model) { + return "\(size) · \(hotword) · \(state) · \(usage)" + } + return "\(size) · \(hotword) · \(state)" + } + + private var diagnosticsSection: some View { + Section { + if let snapshot = modelVM.diagnosticsSnapshot { + LabeledContent(MacL10n.string("mac.localASR.diagBackend", language: lang)) { + Text(snapshot.backendLabel ?? "—") + } + LabeledContent(MacL10n.string("mac.localASR.diagUserTerms", language: lang)) { + Text("\(snapshot.diagnostics.userTermCount)") + } + LabeledContent(MacL10n.string("mac.localASR.diagBuiltinTerms", language: lang)) { + Text("\(snapshot.diagnostics.builtinTermCount)") + } + LabeledContent(MacL10n.string("mac.localASR.diagHotwords", language: lang)) { + Text("\(snapshot.hotwordCount)") + } + LabeledContent(MacL10n.string("mac.localASR.diagPrompt", language: lang)) { + Text("\(snapshot.promptBiasLength)") + } + if snapshot.diagnostics.truncated { + Label( + snapshot.diagnostics.truncationReason ?? MacL10n.string("mac.localASR.diagTruncated", language: lang), + systemImage: "exclamationmark.triangle" + ) + .font(TypeStyle.caption) + .foregroundStyle(palette.warning) + } + Text(snapshot.diagnostics.selectedSources.joined(separator: ", ")) + .font(TypeStyle.caption) + .foregroundStyle(palette.textSecondary) + } else { + Text(MacL10n.string("mac.localASR.diagEmpty", language: lang)) + .foregroundStyle(palette.textSecondary) + } + } header: { + Text(MacL10n.string("mac.localASR.diagnostics", language: lang)) + } footer: { + Text(MacL10n.string("mac.localASR.diagnosticsDesc", language: lang)) + .font(TypeStyle.caption) + .foregroundStyle(palette.textSecondary) + } + } +} diff --git a/OSGKeyboardMac/MacLocalASRService.swift b/OSGKeyboardMac/MacLocalASRService.swift index 0f7f3e9..6d185fc 100644 --- a/OSGKeyboardMac/MacLocalASRService.swift +++ b/OSGKeyboardMac/MacLocalASRService.swift @@ -1,13 +1,15 @@ // MacLocalASRService.swift // OSGKeyboard · Mac // -// On-device ASR for macOS. Primary: Qwen3-ASR-1.7B (MLX via mlx-swift-asr). -// Falls back to Apple Speech when Qwen3 weights are absent or backend is Apple Speech. +// On-device ASR for macOS. Routes through the bundled local ASR catalog: +// Qwen3 MLX (default), Sherpa Qwen3 hotwords POC, SenseVoice, Apple Speech fallback. import Foundation enum MacLocalASRBackend: String, Sendable, CaseIterable { case qwen3MLX + case sherpaQwen3 + case sherpaSenseVoice case appleSpeech } @@ -39,28 +41,33 @@ enum MacLocalASRError: Error, LocalizedError { enum MacLocalASRPreferences { static let backendKey = "mac.localASR.backend" - static let qwen3ModelPathKey = "mac.localASR.qwen3ModelPath" + static let selectedModelIdKey = LocalASRPreferenceKeys.selectedModelId + /// Shared managed subfolder for the manually-provided MLX weights. + static let qwen3ModelRelativePath = "models/qwen3-asr-1.7b-mlx" - static var backend: MacLocalASRBackend { + static var selectedModelId: String { get { - guard let raw = UserDefaults.standard.string(forKey: backendKey), - let value = MacLocalASRBackend(rawValue: raw) else { - return .qwen3MLX + if let raw = UserDefaults.standard.string(forKey: selectedModelIdKey), !raw.isEmpty { + return raw } - return value + return legacyBackend == .appleSpeech ? "apple-speech-fallback" : "qwen3-mlx-1.7b" } - set { UserDefaults.standard.set(newValue.rawValue, forKey: backendKey) } + set { UserDefaults.standard.set(newValue, forKey: selectedModelIdKey) } } + static var legacyBackend: MacLocalASRBackend { + guard let raw = UserDefaults.standard.string(forKey: backendKey), + let value = MacLocalASRBackend(rawValue: raw) else { + return .qwen3MLX + } + return value + } + + /// Fixed location inside the shared managed model storage root. All three + /// catalog models live under the same directory, so MLX no longer needs a + /// per-model folder picker — the user drops converted weights here. static var qwen3ModelPath: String { - get { UserDefaults.standard.string(forKey: qwen3ModelPathKey) ?? defaultQwen3ModelPath } - set { UserDefaults.standard.set(newValue, forKey: qwen3ModelPathKey) } - } - - /// Default install location for MLX-converted Qwen3-ASR weights. - static var defaultQwen3ModelPath: String { - let appSupport = FileManager.default.urls(for: .applicationSupportDirectory, in: .userDomainMask).first! - return appSupport.appendingPathComponent("OSGKeyboard/models/qwen3-asr-1.7b-mlx", isDirectory: true).path + LocalASRModelInstallState.installDirectory(for: qwen3ModelRelativePath).path } static func qwen3ModelIsInstalled(at path: String = qwen3ModelPath) -> Bool { @@ -80,24 +87,93 @@ enum MacLocalASRPreferences { } enum MacLocalASRService { - /// Transcribe using the user's preferred local backend with automatic - /// fallback to Apple Speech when Qwen3 weights are not present. - static func transcribe(samples: [Float], locale: Locale) async throws -> String { - let preferQwen3 = MacLocalASRPreferences.backend == .qwen3MLX - if preferQwen3, MacLocalASRPreferences.qwen3ModelIsInstalled() { + + static func loadCatalog() -> LocalASRCatalogDocument? { + try? LocalASRModelCatalog.loadBundled() + } + + static func selectedModelDefinition() -> LocalASRModelDefinition? { + guard let catalog = loadCatalog() else { return nil } + let manifest = LocalASRInstalledManifestIO.load(defaultModelId: catalog.defaultModelId) + let selectedId = manifest.selectedModelId.isEmpty + ? MacLocalASRPreferences.selectedModelId + : manifest.selectedModelId + if selectedId == "apple-speech-fallback" { return nil } + return LocalASRModelCatalog.model(selectedId, in: catalog) + ?? LocalASRModelCatalog.model(catalog.defaultModelId, in: catalog) + } + + static func currentCapabilities() -> LocalASRCapabilities { + guard let model = selectedModelDefinition() else { return .appleSpeech } + return LocalASRModelCatalog.capabilities(for: model) + } + + static func currentBackendLabel() -> String { + guard let model = selectedModelDefinition() else { return "Apple Speech" } + return model.displayName + } + + static func isModelInstalled(_ model: LocalASRModelDefinition) -> Bool { + LocalASRModelInstallState.isInstalled( + model, + manualMLXPath: MacLocalASRPreferences.qwen3ModelPath + ) + } + + /// Transcribe using the selected catalog model, with MLX → Apple Speech fallback. + static func transcribe( + samples: [Float], + locale: Locale, + bias: LocalASRBiasPayload? = nil + ) async throws -> String { + if let model = selectedModelDefinition(), isModelInstalled(model) { do { - return try await MacQwen3LocalASR.transcribe( - samples: samples, - sampleRate: 16_000, - locale: locale, - modelPath: MacLocalASRPreferences.qwen3ModelPath - ) - } catch MacLocalASRError.qwen3ModelMissing { - // Fall through to Apple Speech when weights are absent. + return try await transcribeWithModel(model, samples: samples, locale: locale, bias: bias) } catch { - throw error + if model.backend != .mlx { + throw error + } } } + + if MacLocalASRPreferences.qwen3ModelIsInstalled() { + return try await MacQwen3LocalASR.transcribe( + samples: samples, + sampleRate: 16_000, + locale: locale, + modelPath: MacLocalASRPreferences.qwen3ModelPath, + bias: bias + ) + } + return try await MacSpeechLocalASR.transcribe(samples: samples, locale: locale) } + + private static func transcribeWithModel( + _ model: LocalASRModelDefinition, + samples: [Float], + locale: Locale, + bias: LocalASRBiasPayload? + ) async throws -> String { + switch model.backend { + case .mlx: + return try await MacQwen3LocalASR.transcribe( + samples: samples, + sampleRate: 16_000, + locale: locale, + modelPath: MacLocalASRPreferences.qwen3ModelPath, + bias: bias + ) + case .sherpaQwen3, .sherpaSenseVoice: + return try await MacSherpaLocalASR.transcribe( + samples: samples, + sampleRate: 16_000, + locale: locale, + model: model, + bias: bias + ) + case .appleSpeech: + return try await MacSpeechLocalASR.transcribe(samples: samples, locale: locale) + } + } } diff --git a/OSGKeyboardMac/MacQwen3ASREngine.swift b/OSGKeyboardMac/MacQwen3ASREngine.swift index a2eee4b..e06fc3b 100644 --- a/OSGKeyboardMac/MacQwen3ASREngine.swift +++ b/OSGKeyboardMac/MacQwen3ASREngine.swift @@ -50,14 +50,15 @@ actor MacQwen3ASREngine { func transcribe( samples: [Float], language: String?, - modelPath: String + modelPath: String, + context: String? = nil ) async throws -> String { try await prepareIfNeeded(modelPath: modelPath) guard let stt else { throw MacLocalASRError.qwen3LoadFailed("Engine not initialized") } - let result = try await stt.transcribe(audio: samples, language: language) + let result = try await stt.transcribe(audio: samples, language: language, context: context) let text = result.text.trimmingCharacters(in: .whitespacesAndNewlines) guard !text.isEmpty else { throw MacLocalASRError.emptyTranscript diff --git a/OSGKeyboardMac/MacQwen3LocalASR.swift b/OSGKeyboardMac/MacQwen3LocalASR.swift index 377609e..cd46822 100644 --- a/OSGKeyboardMac/MacQwen3LocalASR.swift +++ b/OSGKeyboardMac/MacQwen3LocalASR.swift @@ -12,7 +12,8 @@ enum MacQwen3LocalASR { samples: [Float], sampleRate: Int, locale: Locale, - modelPath: String + modelPath: String, + bias: LocalASRBiasPayload? = nil ) async throws -> String { guard MacLocalASRPreferences.qwen3ModelIsInstalled(at: modelPath) else { throw MacLocalASRError.qwen3ModelMissing @@ -24,11 +25,14 @@ enum MacQwen3LocalASR { } let language = MacQwen3LanguageHint.from(locale: locale) + let context = bias?.promptBias?.trimmingCharacters(in: .whitespacesAndNewlines) + let promptContext = (context?.isEmpty == false) ? context : nil do { return try await MacQwen3ASREngine.shared.transcribe( samples: samples, language: language, - modelPath: modelPath + modelPath: modelPath, + context: promptContext ) } catch let error as MacLocalASRError { throw error diff --git a/OSGKeyboardMac/MacRootView.swift b/OSGKeyboardMac/MacRootView.swift index dc904e4..d4552d9 100644 --- a/OSGKeyboardMac/MacRootView.swift +++ b/OSGKeyboardMac/MacRootView.swift @@ -56,7 +56,6 @@ struct MacRootView: View { Spacer() devicesFooter } - .background(palette.surfaceMuted) } private func sidebarRow(_ section: MacSection) -> some View { diff --git a/OSGKeyboardMac/MacSettingsView.swift b/OSGKeyboardMac/MacSettingsView.swift index 6648178..a4b607a 100644 --- a/OSGKeyboardMac/MacSettingsView.swift +++ b/OSGKeyboardMac/MacSettingsView.swift @@ -37,7 +37,7 @@ struct MacSettingsView: View { providerSection } if viewModel.config.engineMode == "local" { - qwen3Section + MacLocalASRModelSettingsView(viewModel: viewModel) } inputSection syncSection @@ -147,12 +147,6 @@ struct MacSettingsView: View { systemImage: "cpu", selected: viewModel.config.engineMode == "local" ) { viewModel.setEngineMode("local") } - - if viewModel.config.engineMode == "local", !viewModel.qwen3ModelInstalled { - Label(MacL10n.string("mac.settings.qwen3Missing", language: lang), systemImage: "exclamationmark.triangle") - .font(TypeStyle.caption) - .foregroundStyle(palette.warning) - } } } @@ -196,25 +190,7 @@ struct MacSettingsView: View { } } - // MARK: - Qwen3 model path - - private var qwen3Section: some View { - Section { - HStack(spacing: Spacing.sm) { - TextField("", text: qwen3PathBinding, prompt: Text(verbatim: "~/Models/Qwen3-ASR")) - .macFieldStyle() - Button(MacL10n.string("mac.settings.qwen3Browse", language: lang)) { - pickQwen3Folder() - } - } - } header: { - Text(MacL10n.string("mac.settings.qwen3Model", language: lang)) - } footer: { - Text(MacL10n.string("mac.settings.qwen3ModelDesc", language: lang)) - .font(TypeStyle.caption) - .foregroundStyle(palette.textSecondary) - } - } + // MARK: - Qwen3 model path (legacy — see MacLocalASRModelSettingsView) // MARK: - Row helpers @@ -366,17 +342,6 @@ struct MacSettingsView: View { ) } - private var qwen3PathBinding: Binding { - Binding( - get: { MacLocalASRPreferences.qwen3ModelPath }, - set: { newPath in - MacLocalASRPreferences.qwen3ModelPath = newPath - Task { await MacQwen3ASREngine.shared.unload() } - viewModel.warmUpQwen3IfNeeded() - } - ) - } - // MARK: - AppKit actions (macOS only) private func openAccessibilitySettings() { @@ -413,19 +378,4 @@ struct MacSettingsView: View { private var accessibilityStatusNeeded: String { lang.resolvedLanguageCode().hasPrefix("zh") ? "未授权" : "Needed" } - - private func pickQwen3Folder() { - #if os(macOS) - let panel = NSOpenPanel() - panel.canChooseDirectories = true - panel.canChooseFiles = false - panel.allowsMultipleSelection = false - panel.begin { response in - guard response == .OK, let url = panel.url else { return } - MacLocalASRPreferences.qwen3ModelPath = url.path - Task { await MacQwen3ASREngine.shared.unload() } - viewModel.warmUpQwen3IfNeeded() - } - #endif - } } diff --git a/OSGKeyboardMac/MacSherpaLocalASR.swift b/OSGKeyboardMac/MacSherpaLocalASR.swift new file mode 100644 index 0000000..94b632e --- /dev/null +++ b/OSGKeyboardMac/MacSherpaLocalASR.swift @@ -0,0 +1,56 @@ +// MacSherpaLocalASR.swift +// OSGKeyboard · Mac +// +// Sherpa-onnx backed local ASR (Qwen3 hotwords POC + SenseVoice baseline). + +import Foundation + +enum MacSherpaLocalASR { + + static func transcribe( + samples: [Float], + sampleRate: Int, + locale: Locale, + model: LocalASRModelDefinition, + bias: LocalASRBiasPayload? + ) async throws -> String { + let catalog = try LocalASRModelCatalog.loadBundled() + let manager = LocalASRModelManager.shared + guard let layout = model.layout, + let modelRoot = LocalASRModelInstallState.modelRootURL(model) else { + throw MacLocalASRError.qwen3ModelMissing + } + + try await manager.ensureRuntimeInstalled(catalog: catalog) + guard let runtime = LocalASRModelCatalog.runtime( + for: LocalASRModelCatalog.currentRuntimePlatform(), + in: catalog + ), + let binary = LocalASRModelInstallState.resolveRuntimeBinary(runtime: runtime) else { + throw MacLocalASRError.qwen3LoadFailed("Sherpa runtime binary missing") + } + + switch model.backend { + case .sherpaQwen3: + return try await MacSherpaONNXRunner.transcribeQwen3( + samples: samples, + sampleRate: sampleRate, + locale: locale, + modelRoot: modelRoot, + layout: layout, + runtimeBinary: binary, + bias: bias + ) + case .sherpaSenseVoice: + return try await MacSherpaONNXRunner.transcribeSenseVoice( + samples: samples, + sampleRate: sampleRate, + modelRoot: modelRoot, + layout: layout, + runtimeBinary: binary + ) + default: + throw MacLocalASRError.qwen3InferenceFailed("Unsupported Sherpa backend") + } + } +} diff --git a/OSGKeyboardMac/MacSherpaONNXRunner.swift b/OSGKeyboardMac/MacSherpaONNXRunner.swift new file mode 100644 index 0000000..d4116b8 --- /dev/null +++ b/OSGKeyboardMac/MacSherpaONNXRunner.swift @@ -0,0 +1,153 @@ +// MacSherpaONNXRunner.swift +// OSGKeyboard · Mac +// +// Invokes the downloaded `sherpa-onnx-offline` binary for Sherpa-backed POC models. + +import Foundation + +enum MacSherpaONNXRunner { + + static func transcribeQwen3( + samples: [Float], + sampleRate: Int, + locale: Locale, + modelRoot: URL, + layout: LocalASRModelLayout, + runtimeBinary: URL, + bias: LocalASRBiasPayload? + ) async throws -> String { + guard sampleRate == 16_000 else { + throw MacLocalASRError.qwen3InferenceFailed("Sherpa expects 16 kHz audio") + } + guard let conv = layout.convFrontend, + let encoder = layout.encoder, + let decoder = layout.decoder, + let tokenizer = layout.tokenizer else { + throw MacLocalASRError.qwen3InferenceFailed("Incomplete Sherpa Qwen3 layout") + } + + let wavURL = try writeTemporaryWAV(samples: samples, sampleRate: sampleRate) + defer { try? FileManager.default.removeItem(at: wavURL) } + + var arguments = [ + "--qwen3-asr-conv-frontend=\(modelRoot.appendingPathComponent(conv).path)", + "--qwen3-asr-encoder=\(modelRoot.appendingPathComponent(encoder).path)", + "--qwen3-asr-decoder=\(modelRoot.appendingPathComponent(decoder).path)", + "--qwen3-asr-tokenizer=\(modelRoot.appendingPathComponent(tokenizer).path)", + "--qwen3-asr-max-new-tokens=512", + "--num-threads=2", + ] + + if let language = MacQwen3LanguageHint.from(locale: locale) { + arguments.append("--qwen3-asr-language=\(language)") + } + + if let hotwords = bias?.hardHotwords, !hotwords.isEmpty { + arguments.append("--qwen3-asr-hotwords=\(hotwords.joined(separator: ","))") + } + + arguments.append(wavURL.path) + return try await run(binary: runtimeBinary, arguments: arguments) + } + + static func transcribeSenseVoice( + samples: [Float], + sampleRate: Int, + modelRoot: URL, + layout: LocalASRModelLayout, + runtimeBinary: URL + ) async throws -> String { + guard sampleRate == 16_000 else { + throw MacLocalASRError.qwen3InferenceFailed("Sherpa expects 16 kHz audio") + } + guard let model = layout.senseVoiceModel, + let tokens = layout.tokens else { + throw MacLocalASRError.qwen3InferenceFailed("Incomplete SenseVoice layout") + } + + let wavURL = try writeTemporaryWAV(samples: samples, sampleRate: sampleRate) + defer { try? FileManager.default.removeItem(at: wavURL) } + + let arguments = [ + "--tokens=\(modelRoot.appendingPathComponent(tokens).path)", + "--sense-voice-model=\(modelRoot.appendingPathComponent(model).path)", + "--num-threads=2", + wavURL.path, + ] + return try await run(binary: runtimeBinary, arguments: arguments) + } + + // MARK: - Private + + private static func writeTemporaryWAV(samples: [Float], sampleRate: Int) throws -> URL { + let data = PCMSampleWavEncoder.encode(samples: samples, sampleRate: sampleRate) + let url = FileManager.default.temporaryDirectory + .appendingPathComponent("osg-sherpa-\(UUID().uuidString).wav") + try data.write(to: url, options: .atomic) + return url + } + + private static func run(binary: URL, arguments: [String]) async throws -> String { + try await withCheckedThrowingContinuation { continuation in + let process = Process() + process.executableURL = binary + process.arguments = arguments + process.currentDirectoryURL = binary.deletingLastPathComponent() + + let outputPipe = Pipe() + let errorPipe = Pipe() + process.standardOutput = outputPipe + process.standardError = errorPipe + + process.terminationHandler = { proc in + let outputData = outputPipe.fileHandleForReading.readDataToEndOfFile() + let errorData = errorPipe.fileHandleForReading.readDataToEndOfFile() + let stdout = String(data: outputData, encoding: .utf8) ?? "" + let stderr = String(data: errorData, encoding: .utf8) ?? "" + + guard proc.terminationStatus == 0 else { + let detail = stderr.isEmpty ? stdout : stderr + continuation.resume( + throwing: MacLocalASRError.qwen3InferenceFailed( + detail.trimmingCharacters(in: .whitespacesAndNewlines) + ) + ) + return + } + + let text = parseTranscript(stdout: stdout) + if text.isEmpty { + continuation.resume(throwing: MacLocalASRError.emptyTranscript) + } else { + continuation.resume(returning: text) + } + } + + do { + try process.run() + } catch { + continuation.resume(throwing: MacLocalASRError.qwen3InferenceFailed(error.localizedDescription)) + } + } + } + + private static func parseTranscript(stdout: String) -> String { + let lines = stdout + .split(whereSeparator: \.isNewline) + .map { String($0).trimmingCharacters(in: .whitespacesAndNewlines) } + .filter { !$0.isEmpty } + + for line in lines.reversed() { + if line.hasPrefix("{"), let data = line.data(using: .utf8), + let object = try? JSONSerialization.jsonObject(with: data) as? [String: Any], + let text = object["text"] as? String { + let trimmed = text.trimmingCharacters(in: .whitespacesAndNewlines) + if !trimmed.isEmpty { return trimmed } + } + if !line.hasPrefix("/"), !line.hasPrefix("--"), line.count > 1 { + return line + } + } + return "" + } +} diff --git a/OSGKeyboardMac/OSGKeyboardMac.entitlements b/OSGKeyboardMac/OSGKeyboardMac.entitlements index 422ac05..fa4ef6b 100644 --- a/OSGKeyboardMac/OSGKeyboardMac.entitlements +++ b/OSGKeyboardMac/OSGKeyboardMac.entitlements @@ -3,7 +3,7 @@ com.apple.security.app-sandbox - + com.apple.security.device.audio-input com.apple.security.network.client diff --git a/OSGKeyboardMac/OSGKeyboardMacApp.swift b/OSGKeyboardMac/OSGKeyboardMacApp.swift index 0895bf8..a87c048 100644 --- a/OSGKeyboardMac/OSGKeyboardMacApp.swift +++ b/OSGKeyboardMac/OSGKeyboardMacApp.swift @@ -86,6 +86,20 @@ final class MacAppDelegate: NSObject, NSApplicationDelegate { MacAppearancePreference.applyToApp(.current) configurePopover() configureStatusItem() + + // The menu bar always follows the *system* appearance, so the status + // item must ignore the app's forced light/dark override. Re-pin the + // button appearance whenever the system theme flips. + DistributedNotificationCenter.default.addObserver( + self, + selector: #selector(systemAppearanceDidChange), + name: NSNotification.Name("AppleInterfaceThemeChangedNotification"), + object: nil + ) + } + + deinit { + DistributedNotificationCenter.default.removeObserver(self) } /// Keep the app alive after the last window closes — it lives in the menu bar. @@ -96,18 +110,47 @@ final class MacAppDelegate: NSObject, NSApplicationDelegate { private func configureStatusItem() { let item = NSStatusBar.system.statusItem(withLength: NSStatusItem.variableLength) if let button = item.button { - // Prefer the brand mark; fall back to an SF Symbol so the item is - // never invisible even if the asset fails to resolve. - let image = NSImage(named: "OSGBrandMark") - ?? NSImage(systemSymbolName: "mic.circle.fill", accessibilityDescription: "OSGKeyboard") - image?.isTemplate = true - image?.size = NSSize(width: 18, height: 18) - button.image = image + button.image = Self.makeStatusBarImage() button.image?.accessibilityDescription = "OSGKeyboard" button.action = #selector(togglePopover(_:)) button.target = self } statusItem = item + applyStatusItemAppearance() + } + + /// Builds the menu-bar glyph from the dedicated horizontal status mark. + /// Height is pinned to the tallest practical menu-bar slot so the logo reads + /// clearly; width follows the asset's aspect ratio. + private static func makeStatusBarImage() -> NSImage? { + guard let image = NSImage(named: "OSGStatusMark") + ?? NSImage(named: "OSGBrandMark") + ?? NSImage(systemSymbolName: "mic.circle.fill", accessibilityDescription: "OSGKeyboard") else { + return nil + } + let height: CGFloat = 9 + let aspect = max(image.size.width / max(image.size.height, 1), 1) + image.size = NSSize(width: height * aspect, height: height) + image.isTemplate = true + return image + } + + /// Pins the status-bar button to the current *system* appearance so its + /// template image tint matches the real menu-bar background — regardless of + /// the in-app light/dark preference forced on `NSApp.appearance`. + private func applyStatusItemAppearance() { + guard let button = statusItem?.button else { return } + let isDark = UserDefaults.standard.string(forKey: "AppleInterfaceStyle")? + .lowercased().contains("dark") ?? false + button.appearance = NSAppearance(named: isDark ? .darkAqua : .aqua) + } + + @objc private func systemAppearanceDidChange() { + // The global-domain default lags the notification by a hair; hop to the + // next runloop tick so `AppleInterfaceStyle` reflects the new value. + DispatchQueue.main.async { [weak self] in + self?.applyStatusItemAppearance() + } } private func configurePopover() { diff --git a/OSGKeyboardShared/Models/LocalASRBiasPayload.swift b/OSGKeyboardShared/Models/LocalASRBiasPayload.swift new file mode 100644 index 0000000..392fa78 --- /dev/null +++ b/OSGKeyboardShared/Models/LocalASRBiasPayload.swift @@ -0,0 +1,99 @@ +// LocalASRBiasPayload.swift +// OSGKeyboard · Shared +// +// Output of `LocalASRBiasAdapter` — vocabulary signals for each pipeline layer. + +import Foundation + +public struct LocalASRCorrectionPair: Sendable, Equatable { + public let alias: String + public let term: String + + public init(alias: String, term: String) { + self.alias = alias + self.term = term + } +} + +public struct LocalASRBiasDiagnostics: Sendable, Equatable, Codable { + public var userTermCount: Int + public var builtinTermCount: Int + public var truncated: Bool + public var truncationReason: String? + public var selectedSources: [String] + + public init( + userTermCount: Int = 0, + builtinTermCount: Int = 0, + truncated: Bool = false, + truncationReason: String? = nil, + selectedSources: [String] = [] + ) { + self.userTermCount = userTermCount + self.builtinTermCount = builtinTermCount + self.truncated = truncated + self.truncationReason = truncationReason + self.selectedSources = selectedSources + } +} + +public struct LocalASRBiasPayload: Sendable, Equatable { + public var hardHotwords: [String] + public var promptBias: String? + public var corpusContext: String? + public var polishFragment: String + public var correctionPairs: [LocalASRCorrectionPair] + public var diagnostics: LocalASRBiasDiagnostics + + public static let empty = LocalASRBiasPayload( + hardHotwords: [], + promptBias: nil, + corpusContext: nil, + polishFragment: "", + correctionPairs: [], + diagnostics: LocalASRBiasDiagnostics() + ) + + public init( + hardHotwords: [String], + promptBias: String?, + corpusContext: String?, + polishFragment: String, + correctionPairs: [LocalASRCorrectionPair], + diagnostics: LocalASRBiasDiagnostics + ) { + self.hardHotwords = hardHotwords + self.promptBias = promptBias + self.corpusContext = corpusContext + self.polishFragment = polishFragment + self.correctionPairs = correctionPairs + self.diagnostics = diagnostics + } +} + +public struct LocalASRBiasRequest: Sendable { + public var dictionary: PersonalDictionary + public var locale: Locale + public var frontAppBundleId: String? + public var capabilities: LocalASRCapabilities + /// Max builtin `phrases.tsv` terms considered for ASR bias (not polish-only). + public var builtinASRLimit: Int + /// Max builtin terms referenced in the polish supplement block. + public var builtinPolishLimit: Int + + public init( + dictionary: PersonalDictionary, + locale: Locale, + frontAppBundleId: String? = nil, + capabilities: LocalASRCapabilities, + builtinASRLimit: Int = 300, + builtinPolishLimit: Int = 40 + ) { + self.dictionary = dictionary + self.locale = locale + self.frontAppBundleId = frontAppBundleId + self.capabilities = capabilities + self.builtinASRLimit = builtinASRLimit + self.builtinPolishLimit = builtinPolishLimit + } +} diff --git a/OSGKeyboardShared/Models/LocalASRCapabilities.swift b/OSGKeyboardShared/Models/LocalASRCapabilities.swift new file mode 100644 index 0000000..c5e33d5 --- /dev/null +++ b/OSGKeyboardShared/Models/LocalASRCapabilities.swift @@ -0,0 +1,82 @@ +// LocalASRCapabilities.swift +// OSGKeyboard · Shared +// +// Declares what each on-device ASR backend can accept for vocabulary bias. +// Callers must consult capabilities before building a `LocalASRBiasPayload`. + +import Foundation + +/// How a backend accepts vocabulary hints (honest matrix — not every model +/// supports hard hotwords). +public enum LocalASRHotwordMode: String, Sendable, Codable, Equatable { + case none + case promptOnly + case perRequest + case recognizerScoped + case cloudVocabulary +} + +/// Cost of refreshing hotwords on a backend (e.g. Sherpa Qwen3 reloads recognizer). +public enum LocalASRHotwordReloadCost: String, Sendable, Codable, Equatable { + case none + case recognizerReload + case modelReload +} + +public struct LocalASRCapabilities: Sendable, Equatable { + public let hotwordMode: LocalASRHotwordMode + public let maxHotwordCount: Int + public let maxPromptCharacters: Int + public let supportsStreaming: Bool + public let hotwordReloadCost: LocalASRHotwordReloadCost + + public init( + hotwordMode: LocalASRHotwordMode, + maxHotwordCount: Int, + maxPromptCharacters: Int, + supportsStreaming: Bool, + hotwordReloadCost: LocalASRHotwordReloadCost + ) { + self.hotwordMode = hotwordMode + self.maxHotwordCount = maxHotwordCount + self.maxPromptCharacters = maxPromptCharacters + self.supportsStreaming = supportsStreaming + self.hotwordReloadCost = hotwordReloadCost + } + + /// Qwen3 MLX via mlx-swift-asr — `context` soft prompt on `transcribe`. + public static let qwen3MLX = LocalASRCapabilities( + hotwordMode: .promptOnly, + maxHotwordCount: 0, + maxPromptCharacters: 800, + supportsStreaming: false, + hotwordReloadCost: .none + ) + + /// Apple Speech on macOS — no project-controlled hotword API today. + public static let appleSpeech = LocalASRCapabilities( + hotwordMode: .none, + maxHotwordCount: 0, + maxPromptCharacters: 0, + supportsStreaming: false, + hotwordReloadCost: .none + ) + + /// Sherpa Qwen3 — hard hotwords via `--qwen3-asr-hotwords`. + public static let sherpaQwen3 = LocalASRCapabilities( + hotwordMode: .recognizerScoped, + maxHotwordCount: 100, + maxPromptCharacters: 0, + supportsStreaming: false, + hotwordReloadCost: .recognizerReload + ) + + /// Sherpa SenseVoice — fast Chinese baseline without hotwords. + public static let sherpaSenseVoice = LocalASRCapabilities( + hotwordMode: .none, + maxHotwordCount: 0, + maxPromptCharacters: 0, + supportsStreaming: false, + hotwordReloadCost: .none + ) +} diff --git a/OSGKeyboardShared/Models/LocalASRModelCatalog.swift b/OSGKeyboardShared/Models/LocalASRModelCatalog.swift new file mode 100644 index 0000000..d4f2f12 --- /dev/null +++ b/OSGKeyboardShared/Models/LocalASRModelCatalog.swift @@ -0,0 +1,134 @@ +// LocalASRModelCatalog.swift +// OSGKeyboard · Shared +// +// Bundled catalog of downloadable / manual local ASR models and Sherpa runtimes. + +import Foundation + +public enum LocalASRModelBackend: String, Codable, Sendable, Equatable { + case mlx + case sherpaQwen3 + case sherpaSenseVoice + case appleSpeech +} + +public enum LocalASRInstallKind: String, Codable, Sendable, Equatable { + case manual + case archive + case runtime +} + +public struct LocalASRDownloadSource: Codable, Sendable, Equatable { + public let type: String + public let priority: Int + public let url: String +} + +public struct LocalASRModelLayout: Codable, Sendable, Equatable { + public var convFrontend: String? + public var encoder: String? + public var decoder: String? + public var tokenizer: String? + public var senseVoiceModel: String? + public var tokens: String? +} + +public struct LocalASRRuntimeDefinition: Codable, Sendable, Equatable, Identifiable { + public let id: String + public let displayName: String + public let installRelativePath: String + public let binaryCandidates: [String] + public let archiveFileName: String + public let sizeBytes: Int + public let platform: String + public let sources: [LocalASRDownloadSource] +} + +public struct LocalASRModelDefinition: Codable, Sendable, Equatable, Identifiable { + public let id: String + public let displayName: String + public let backend: LocalASRModelBackend + public let sizeBytes: Int + public let recommendedLocales: [String] + public let supportsHotwords: Bool + public let hotwordMode: LocalASRHotwordMode + public let installKind: LocalASRInstallKind + public let installRelativePath: String? + public let archiveBaseName: String? + public let layout: LocalASRModelLayout? + public let requiredRelativeFiles: [String]? + public let runtimePlatform: String? + public let sources: [LocalASRDownloadSource]? +} + +public struct LocalASRCatalogDocument: Codable, Sendable, Equatable { + public let schemaVersion: Int + public let defaultModelId: String + public let runtimes: [LocalASRRuntimeDefinition] + public let models: [LocalASRModelDefinition] +} + +public enum LocalASRModelCatalog { + + public static func loadBundled() throws -> LocalASRCatalogDocument { + let bundle = Bundle(for: LocalASRCatalogBundleToken.self) + guard let url = bundle.url(forResource: "local-asr-catalog", withExtension: "json") else { + throw LocalASRModelCatalogError.missingBundledCatalog + } + let data = try Data(contentsOf: url) + return try JSONDecoder().decode(LocalASRCatalogDocument.self, from: data) + } + + public static func model(_ id: String, in catalog: LocalASRCatalogDocument) -> LocalASRModelDefinition? { + catalog.models.first { $0.id == id } + } + + public static func capabilities(for model: LocalASRModelDefinition) -> LocalASRCapabilities { + switch model.backend { + case .mlx: + return .qwen3MLX + case .sherpaQwen3: + return .sherpaQwen3 + case .sherpaSenseVoice: + return .sherpaSenseVoice + case .appleSpeech: + return .appleSpeech + } + } + + #if os(macOS) + public static func runtime(for platform: String, in catalog: LocalASRCatalogDocument) -> LocalASRRuntimeDefinition? { + if platform == "macos-arm64" { + return catalog.runtimes.first { $0.platform == "macos-arm64" } + } + if platform == "macos-x64" { + return catalog.runtimes.first { $0.platform == "macos-x64" } + } + return catalog.runtimes.first + } + + public static func currentRuntimePlatform() -> String { + #if arch(arm64) + return "macos-arm64" + #else + return "macos-x64" + #endif + } + #endif +} + +public enum LocalASRModelCatalogError: Error, LocalizedError { + case missingBundledCatalog + case modelNotFound(String) + + public var errorDescription: String? { + switch self { + case .missingBundledCatalog: + return "Missing bundled local ASR catalog." + case .modelNotFound(let id): + return "Local ASR model not found: \(id)" + } + } +} + +private final class LocalASRCatalogBundleToken {} diff --git a/OSGKeyboardShared/Models/PersonalDictionary+ASRBias.swift b/OSGKeyboardShared/Models/PersonalDictionary+ASRBias.swift index 1a2341d..1215edb 100644 --- a/OSGKeyboardShared/Models/PersonalDictionary+ASRBias.swift +++ b/OSGKeyboardShared/Models/PersonalDictionary+ASRBias.swift @@ -111,4 +111,29 @@ extension PersonalDictionary { if hasNonASCII { return "zh" } return "en" } + + /// Alias → canonical term pairs for deterministic post-ASR correction. + /// Sorted longest-alias-first by the caller (`LocalASRTranscriptCorrector`). + public func localCorrectionPairs() -> [LocalASRCorrectionPair] { + var seen = Set() + var pairs: [LocalASRCorrectionPair] = [] + for entry in effectiveEntries { + let term = entry.term.trimmingCharacters(in: .whitespacesAndNewlines) + guard !term.isEmpty else { continue } + for alias in entry.aliases { + let trimmed = alias.trimmingCharacters(in: .whitespacesAndNewlines) + guard !trimmed.isEmpty else { continue } + guard trimmed.caseInsensitiveCompare(term) != .orderedSame else { continue } + let key = "\(trimmed.lowercased())|\(term.lowercased())" + guard seen.insert(key).inserted else { continue } + pairs.append(LocalASRCorrectionPair(alias: trimmed, term: term)) + } + } + return pairs.sorted { lhs, rhs in + if lhs.alias.count != rhs.alias.count { + return lhs.alias.count > rhs.alias.count + } + return lhs.alias.localizedCaseInsensitiveCompare(rhs.alias) == .orderedAscending + } + } } diff --git a/OSGKeyboardShared/Models/PolishContext.swift b/OSGKeyboardShared/Models/PolishContext.swift index b85a148..4977123 100644 --- a/OSGKeyboardShared/Models/PolishContext.swift +++ b/OSGKeyboardShared/Models/PolishContext.swift @@ -24,6 +24,10 @@ public struct PolishContext: Sendable { /// bias terminology choices. public let precedingText: String? + /// Extra dictionary block appended after `PersonalDictionary.promptFragment()` + /// (e.g. builtin `phrases.tsv` terms on macOS local ASR). + public let dictionarySupplement: String? + /// Cap on how many characters of `precedingText` we actually /// include in the prompt. The full preceding text is often /// hundreds of KB in a long note — we only need the tail. @@ -33,11 +37,13 @@ public struct PolishContext: Sendable { appContext: AppContext = .unknown, intensity: PolishIntensity = .default, precedingText: String? = nil, + dictionarySupplement: String? = nil, maxPrecedingChars: Int = 500 ) { self.appContext = appContext self.intensity = intensity self.precedingText = precedingText + self.dictionarySupplement = dictionarySupplement self.maxPrecedingChars = maxPrecedingChars } diff --git a/OSGKeyboardShared/Models/ProviderConfig.swift b/OSGKeyboardShared/Models/ProviderConfig.swift index 2259dc3..6bf4839 100644 --- a/OSGKeyboardShared/Models/ProviderConfig.swift +++ b/OSGKeyboardShared/Models/ProviderConfig.swift @@ -74,7 +74,7 @@ public final class ProviderConfig: ObservableObject, @unchecked Sendable { guard !isApplyingConfiguration, engineMode != configuration.engineMode else { return } configuration.engineMode = engineMode applyEngineModeSideEffects() - persistConfiguration() + persistConfiguration(postConfigChanged: true) } } @Published public var hasCompletedOnboarding: Bool { diff --git a/OSGKeyboardShared/Resources/LocalASR/local-asr-catalog.json b/OSGKeyboardShared/Resources/LocalASR/local-asr-catalog.json new file mode 100644 index 0000000..610986b --- /dev/null +++ b/OSGKeyboardShared/Resources/LocalASR/local-asr-catalog.json @@ -0,0 +1,102 @@ +{ + "schemaVersion": 1, + "defaultModelId": "qwen3-mlx-1.7b", + "runtimes": [ + { + "id": "sherpa-onnx-1.13.4-macos-arm64", + "displayName": "sherpa-onnx 1.13.4 (Apple Silicon)", + "installRelativePath": "runtimes/sherpa-onnx-1.13.4-macos-arm64", + "binaryCandidates": ["bin/sherpa-onnx-offline", "sherpa-onnx-offline"], + "archiveFileName": "sherpa-onnx-v1.13.4-osx-arm64-static-no-tts.tar.bz2", + "sizeBytes": 120000000, + "platform": "macos-arm64", + "sources": [ + { + "type": "github", + "priority": 1, + "url": "https://github.com/k2-fsa/sherpa-onnx/releases/download/v1.13.4/sherpa-onnx-v1.13.4-osx-arm64-static-no-tts.tar.bz2" + } + ] + }, + { + "id": "sherpa-onnx-1.13.4-macos-x64", + "displayName": "sherpa-onnx 1.13.4 (Intel)", + "installRelativePath": "runtimes/sherpa-onnx-1.13.4-macos-x64", + "binaryCandidates": ["bin/sherpa-onnx-offline", "sherpa-onnx-offline"], + "archiveFileName": "sherpa-onnx-v1.13.4-osx-x64-static-no-tts.tar.bz2", + "sizeBytes": 130000000, + "platform": "macos-x64", + "sources": [ + { + "type": "github", + "priority": 1, + "url": "https://github.com/k2-fsa/sherpa-onnx/releases/download/v1.13.4/sherpa-onnx-v1.13.4-osx-x64-static-no-tts.tar.bz2" + } + ] + } + ], + "models": [ + { + "id": "qwen3-mlx-1.7b", + "displayName": "Qwen3-ASR 1.7B (MLX)", + "backend": "mlx", + "sizeBytes": 1400000000, + "recommendedLocales": ["zh-CN", "en-US"], + "supportsHotwords": true, + "hotwordMode": "promptOnly", + "installKind": "manual", + "installRelativePath": "models/qwen3-asr-1.7b-mlx", + "requiredRelativeFiles": ["config.json", "model.safetensors", "vocab.json", "merges.txt"] + }, + { + "id": "sherpa-qwen3-0.6b-int8", + "displayName": "Qwen3-ASR 0.6B (Sherpa · hotwords)", + "backend": "sherpaQwen3", + "runtimePlatform": "macos", + "sizeBytes": 650000000, + "recommendedLocales": ["zh-CN", "en-US"], + "supportsHotwords": true, + "hotwordMode": "recognizerScoped", + "installKind": "archive", + "installRelativePath": "models/sherpa-qwen3-0.6b-int8", + "archiveBaseName": "sherpa-onnx-qwen3-asr-0.6B-int8-2026-03-25", + "layout": { + "convFrontend": "conv_frontend.onnx", + "encoder": "encoder.int8.onnx", + "decoder": "decoder.int8.onnx", + "tokenizer": "tokenizer" + }, + "sources": [ + { + "type": "github", + "priority": 1, + "url": "https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-qwen3-asr-0.6B-int8-2026-03-25.tar.bz2" + } + ] + }, + { + "id": "sherpa-sensevoice-small-int8", + "displayName": "SenseVoice Small (Sherpa)", + "backend": "sherpaSenseVoice", + "runtimePlatform": "macos", + "sizeBytes": 250000000, + "recommendedLocales": ["zh-CN", "en-US", "ja-JP", "ko-KR"], + "supportsHotwords": false, + "hotwordMode": "none", + "installKind": "archive", + "installRelativePath": "models/sherpa-sensevoice-small-int8", + "archiveBaseName": "sherpa-onnx-sense-voice-zh-en-ja-ko-yue-int8-2024-07-17", + "layout": { + "senseVoiceModel": "model.int8.onnx", + "tokens": "tokens.txt" + }, + "sources": [ + { + "type": "github", + "priority": 1, + "url": "https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-sense-voice-zh-en-ja-ko-yue-int8-2024-07-17.tar.bz2" + } + ] + } + ] +} diff --git a/OSGKeyboardShared/Services/AppGroupStore.swift b/OSGKeyboardShared/Services/AppGroupStore.swift index f1e7bdf..a021bd9 100644 --- a/OSGKeyboardShared/Services/AppGroupStore.swift +++ b/OSGKeyboardShared/Services/AppGroupStore.swift @@ -92,6 +92,7 @@ public struct AppGroupStore: @unchecked Sendable { config.model = openAI.defaultModel } } + AppGroupConfigDarwin.postConfigChanged() } public func setUILanguage(_ language: AppUILanguage) { diff --git a/OSGKeyboardShared/Services/BuiltinLexiconIndex.swift b/OSGKeyboardShared/Services/BuiltinLexiconIndex.swift new file mode 100644 index 0000000..815e5a8 --- /dev/null +++ b/OSGKeyboardShared/Services/BuiltinLexiconIndex.swift @@ -0,0 +1,152 @@ +// BuiltinLexiconIndex.swift +// OSGKeyboard · Shared +// +// In-memory index over bundled `phrases.tsv` (~10k computer terms). +// macOS local ASR consumes a Top-N subset; the full index also backs +// polish supplements and future retrieval. + +import Foundation + +public final class BuiltinLexiconIndex: @unchecked Sendable { + + public struct Term: Sendable, Equatable { + public let word: String + public let pinyin: String + public let source: String + public let weight: Int + } + + public static let shared = BuiltinLexiconIndex() + + private let lock = NSLock() + private var cachedTerms: [Term]? + private let injectedURL: URL? + + /// Production singleton loads from the app bundle. + private init() { + injectedURL = nil + } + + /// Test / preview hook with an explicit TSV file or inline fixture. + init(fixtureURL: URL) { + injectedURL = fixtureURL + } + + /// Parse TSV content without touching the bundle (unit tests). + public static func parseTSV(_ content: String) -> [Term] { + var terms: [Term] = [] + terms.reserveCapacity(256) + + for (lineIndex, line) in content.split(whereSeparator: \.isNewline).enumerated() { + if lineIndex == 0, line.hasPrefix("word\t") { continue } + let columns = line.split(separator: "\t", omittingEmptySubsequences: false) + guard columns.count >= 4 else { continue } + let word = String(columns[0]).trimmingCharacters(in: .whitespacesAndNewlines) + guard !word.isEmpty else { continue } + let pinyin = String(columns[1]) + let source = String(columns[2]) + let weight = Int(columns[3]) ?? 1 + terms.append(Term(word: word, pinyin: pinyin, source: source, weight: weight)) + } + return terms + } + + public func termCount() -> Int { + lock.lock() + defer { lock.unlock() } + return loadTermsLocked().count + } + + /// Returns canonical words ranked for ASR bias injection. + public func topTerms( + limit: Int, + minimumWeight: Int = 4, + preferredSources: Set? = nil + ) -> [String] { + guard limit > 0 else { return [] } + + lock.lock() + let all = loadTermsLocked() + lock.unlock() + + let filtered = all.filter { term in + guard term.weight >= minimumWeight else { return false } + if let preferredSources, !preferredSources.isEmpty { + return preferredSources.contains(term.source) + } + return true + } + + let ranked = filtered.sorted { lhs, rhs in + let leftScore = Self.rankingScore(lhs) + let rightScore = Self.rankingScore(rhs) + if leftScore != rightScore { return leftScore > rightScore } + return lhs.word.localizedCaseInsensitiveCompare(rhs.word) == .orderedAscending + } + + var seen = Set() + var words: [String] = [] + words.reserveCapacity(min(limit, ranked.count)) + for term in ranked { + let key = term.word.lowercased() + guard seen.insert(key).inserted else { continue } + words.append(term.word) + if words.count >= limit { break } + } + return words + } + + // MARK: - Private + + private func loadTermsLocked() -> [Term] { + if let cachedTerms { return cachedTerms } + let loaded: [Term] + if let injectedURL { + loaded = Self.load(from: injectedURL) + } else if let url = Self.locateBundledPhrasesURL() { + loaded = Self.load(from: url) + } else { + loaded = [] + } + cachedTerms = loaded + return loaded + } + + private static func load(from url: URL) -> [Term] { + guard let data = try? Data(contentsOf: url), + let content = String(data: data, encoding: .utf8) else { + return [] + } + return parseTSV(content) + } + + private static func locateBundledPhrasesURL() -> URL? { + let candidates: [Bundle] = [Bundle.main, Bundle(for: BuiltinLexiconIndex.self)] + for bundle in candidates { + if let url = bundle.url( + forResource: "phrases", + withExtension: "tsv", + subdirectory: "CustomLanguageModel/v1" + ) { + return url + } + if let url = bundle.url(forResource: "phrases", withExtension: "tsv") { + return url + } + } + return nil + } + + private static func rankingScore(_ term: Term) -> Int { + var score = term.weight * 100 + if containsLatinLetters(term.word) { score += 50 } + if term.word.count <= 8 { score += 10 } + return score + } + + private static func containsLatinLetters(_ text: String) -> Bool { + text.unicodeScalars.contains { scalar in + scalar.isASCII && CharacterSet.letters.contains(scalar) + } + } +} diff --git a/OSGKeyboardShared/Services/FlowContinuousCapture.swift b/OSGKeyboardShared/Services/FlowContinuousCapture.swift index 9e7e4d9..3e835bc 100644 --- a/OSGKeyboardShared/Services/FlowContinuousCapture.swift +++ b/OSGKeyboardShared/Services/FlowContinuousCapture.swift @@ -384,19 +384,26 @@ public final class FlowContinuousCapture { /// Re-activate capture after returning from background without /// reinstalling the tap (iOS may deactivate the audio session). - public func reassertIfRunning() { - guard isRunning else { return } + @discardableResult + public func reassertIfRunning() -> Bool { + guard isRunning else { return false } let session = AVAudioSession.sharedInstance() - try? session.setCategory( - .playAndRecord, - mode: .measurement, - options: [.defaultToSpeaker, .allowBluetoothHFP, .mixWithOthers] - ) - try? session.setActive(true, options: .notifyOthersOnDeactivation) - if !audioEngine.isRunning { - try? audioEngine.start() + do { + try session.setCategory( + .playAndRecord, + mode: .measurement, + options: [.defaultToSpeaker, .allowBluetoothHFP, .mixWithOthers] + ) + try session.setActive(true, options: .notifyOthersOnDeactivation) + if !audioEngine.isRunning { + try audioEngine.start() + } + notifyEngineLiveChanged() + return engineIsLive + } catch { + notifyEngineLiveChanged() + return false } - notifyEngineLiveChanged() } public func awaitAudioFlowing( diff --git a/OSGKeyboardShared/Services/FlowSessionBridge.swift b/OSGKeyboardShared/Services/FlowSessionBridge.swift index e963cda..353e7e7 100644 --- a/OSGKeyboardShared/Services/FlowSessionBridge.swift +++ b/OSGKeyboardShared/Services/FlowSessionBridge.swift @@ -6,6 +6,158 @@ import Foundation +public struct FlowCommand: Codable, Equatable, Sendable { + public enum Action: String, Codable, Sendable { + case startRecording + case stopRecording + case abort + } + + public let protocolVersion: Int + public let sessionId: UUID + public let utteranceId: UUID + public let commandSeq: Int64 + public let action: Action + public let localeId: String + public let createdAt: TimeInterval + + public init( + protocolVersion: Int = 1, + sessionId: UUID, + utteranceId: UUID, + commandSeq: Int64, + action: Action, + localeId: String, + createdAt: TimeInterval = Date().timeIntervalSince1970 + ) { + self.protocolVersion = protocolVersion + self.sessionId = sessionId + self.utteranceId = utteranceId + self.commandSeq = commandSeq + self.action = action + self.localeId = localeId + self.createdAt = createdAt + } +} + +public struct FlowResult: Codable, Equatable, Sendable { + public enum Status: String, Codable, Sendable { + case partial + case final + case error + case aborted + case timeout + } + + public let protocolVersion: Int + public let sessionId: UUID + public let utteranceId: UUID + public let commandSeq: Int64 + public let status: Status + public let text: String? + public let warning: String? + public let errorKind: FlowSessionKeys.TranscriptionErrorKind? + public let createdAt: TimeInterval + + public init( + protocolVersion: Int = 1, + sessionId: UUID, + utteranceId: UUID, + commandSeq: Int64, + status: Status, + text: String? = nil, + warning: String? = nil, + errorKind: FlowSessionKeys.TranscriptionErrorKind? = nil, + createdAt: TimeInterval = Date().timeIntervalSince1970 + ) { + self.protocolVersion = protocolVersion + self.sessionId = sessionId + self.utteranceId = utteranceId + self.commandSeq = commandSeq + self.status = status + self.text = text + self.warning = warning + self.errorKind = errorKind + self.createdAt = createdAt + } +} + +public struct FlowAck: Codable, Equatable, Sendable { + public let protocolVersion: Int + public let sessionId: UUID + public let utteranceId: UUID + public let commandSeq: Int64 + public let consumedAt: TimeInterval + + public init( + protocolVersion: Int = 1, + sessionId: UUID, + utteranceId: UUID, + commandSeq: Int64, + consumedAt: TimeInterval = Date().timeIntervalSince1970 + ) { + self.protocolVersion = protocolVersion + self.sessionId = sessionId + self.utteranceId = utteranceId + self.commandSeq = commandSeq + self.consumedAt = consumedAt + } +} + +public struct FlowReadySnapshot: Codable, Equatable, Sendable { + public enum Reason: String, Codable, Sendable { + case ready + case noSession + case starting + case audioEngineNotLive + case waitingForAudioProof + case recording + case processing + case permissionMissing + case appGroupUnavailable + case hostLost + case error + } + + public let protocolVersion: Int + public let sessionId: UUID? + public let ready: Bool + public let reason: Reason + public let heartbeatAt: TimeInterval + public let readyAt: TimeInterval? + public let audioProofAt: TimeInterval? + public let engineMode: String + public let localeId: String + public let busyUtteranceId: UUID? + public let sessionExpiresAt: TimeInterval? + + public init( + protocolVersion: Int = 1, + sessionId: UUID?, + ready: Bool, + reason: Reason, + heartbeatAt: TimeInterval = Date().timeIntervalSince1970, + readyAt: TimeInterval? = nil, + audioProofAt: TimeInterval? = nil, + engineMode: String, + localeId: String, + busyUtteranceId: UUID? = nil, + sessionExpiresAt: TimeInterval? = nil + ) { + self.protocolVersion = protocolVersion + self.sessionId = sessionId + self.ready = ready + self.reason = reason + self.heartbeatAt = heartbeatAt + self.readyAt = readyAt + self.audioProofAt = audioProofAt + self.engineMode = engineMode + self.localeId = localeId + self.busyUtteranceId = busyUtteranceId + self.sessionExpiresAt = sessionExpiresAt + } +} + public struct FlowTranscriptionError: Equatable, Sendable { public let message: String public let kind: FlowSessionKeys.TranscriptionErrorKind @@ -36,6 +188,15 @@ public enum FlowSessionBridge { } } + private static func encode(_ value: T) -> Data? { + try? JSONEncoder().encode(value) + } + + private static func decode(_ type: T.Type, from data: Data?) -> T? { + guard let data else { return nil } + return try? JSONDecoder().decode(type, from: data) + } + /// Keyboard/read side: refresh App Group defaults after the extension was /// suspended so decisions are not based on stale in-process caches. public static func reloadFromDisk(defaults: UserDefaults? = nil) { @@ -45,10 +206,87 @@ public enum FlowSessionBridge { } } + // MARK: - Typed Flow protocol + + public static func writeCommand(_ command: FlowCommand, defaults: UserDefaults? = nil) { + let store = resolvedDefaults(defaults) + if let data = encode(command) { + store.set(data, forKey: FlowSessionKeys.flowCommandPayload) + } + flush(store) + FlowSessionDarwin.postCommandChanged() + } + + public static func latestCommand(defaults: UserDefaults? = nil) -> FlowCommand? { + let store = resolvedDefaults(defaults) + return decode(FlowCommand.self, from: store.data(forKey: FlowSessionKeys.flowCommandPayload)) + } + + public static func writeResult(_ result: FlowResult, defaults: UserDefaults? = nil) { + let store = resolvedDefaults(defaults) + if let data = encode(result) { + store.set(data, forKey: FlowSessionKeys.flowResultPayload) + } + flush(store) + FlowSessionDarwin.postTranscriptionChanged() + } + + public static func latestResult(defaults: UserDefaults? = nil) -> FlowResult? { + let store = resolvedDefaults(defaults) + return decode(FlowResult.self, from: store.data(forKey: FlowSessionKeys.flowResultPayload)) + } + + public static func clearResult(defaults: UserDefaults? = nil) { + let store = resolvedDefaults(defaults) + store.removeObject(forKey: FlowSessionKeys.flowResultPayload) + flush(store) + } + + public static func writeAck(_ ack: FlowAck, defaults: UserDefaults? = nil) { + let store = resolvedDefaults(defaults) + if let data = encode(ack) { + store.set(data, forKey: FlowSessionKeys.flowAckPayload) + } + flush(store) + } + + public static func latestAck(defaults: UserDefaults? = nil) -> FlowAck? { + let store = resolvedDefaults(defaults) + return decode(FlowAck.self, from: store.data(forKey: FlowSessionKeys.flowAckPayload)) + } + + public static func writeReadySnapshot(_ snapshot: FlowReadySnapshot, defaults: UserDefaults? = nil) { + let store = resolvedDefaults(defaults) + if let data = encode(snapshot) { + store.set(data, forKey: FlowSessionKeys.flowReadyPayload) + } + if snapshot.ready { + store.set(true, forKey: FlowSessionKeys.flowHostReady) + if let readyAt = snapshot.readyAt { + store.set(readyAt, forKey: FlowSessionKeys.flowHostReadyAt) + } + } else { + clearHostReady(defaults: store, notify: false) + store.removeObject(forKey: FlowSessionKeys.flowReadyPayload) + } + if let expires = snapshot.sessionExpiresAt { + store.set(expires, forKey: FlowSessionKeys.flowSessionExpires) + } + store.set(snapshot.heartbeatAt, forKey: FlowSessionKeys.flowHeartbeat) + flush(store) + FlowSessionDarwin.postHostReadyChanged() + } + + public static func readySnapshot(defaults: UserDefaults? = nil) -> FlowReadySnapshot? { + let store = resolvedDefaults(defaults) + return decode(FlowReadySnapshot.self, from: store.data(forKey: FlowSessionKeys.flowReadyPayload)) + } + // MARK: - Session lifecycle (host app) public static func markSessionActive( duration: TimeInterval? = nil, + sessionId: UUID? = nil, defaults: UserDefaults? = nil ) { let store = resolvedDefaults(defaults) @@ -59,8 +297,26 @@ public enum FlowSessionBridge { store.set(expires, forKey: FlowSessionKeys.flowSessionExpires) store.set(now, forKey: FlowSessionKeys.lastActivityAt) writeHeartbeat(defaults: store) - setRecordingState(.idle, defaults: store) clearTranscription(defaults: store) + store.removeObject(forKey: FlowSessionKeys.flowCommandPayload) + store.removeObject(forKey: FlowSessionKeys.flowResultPayload) + store.removeObject(forKey: FlowSessionKeys.flowAckPayload) + if let sessionId { + let snapshot = FlowReadySnapshot( + sessionId: sessionId, + ready: false, + reason: .starting, + heartbeatAt: now, + engineMode: AppGroupConfiguration.load(fromAvailable: store).engineMode, + localeId: AppGroupConfiguration.load(fromAvailable: store).localeId, + sessionExpiresAt: expires + ) + if let data = encode(snapshot) { + store.set(data, forKey: FlowSessionKeys.flowReadyPayload) + } + } else { + store.removeObject(forKey: FlowSessionKeys.flowReadyPayload) + } flush(store) } @@ -69,8 +325,11 @@ public enum FlowSessionBridge { store.set(false, forKey: FlowSessionKeys.flowSessionActive) store.removeObject(forKey: FlowSessionKeys.flowSessionExpires) store.removeObject(forKey: FlowSessionKeys.flowHeartbeat) - setRecordingState(.idle, defaults: store) clearTranscription(defaults: store) + store.removeObject(forKey: FlowSessionKeys.flowCommandPayload) + store.removeObject(forKey: FlowSessionKeys.flowResultPayload) + store.removeObject(forKey: FlowSessionKeys.flowAckPayload) + store.removeObject(forKey: FlowSessionKeys.flowReadyPayload) clearHostReady(defaults: store, notify: false) flush(store) } @@ -185,6 +444,15 @@ public enum FlowSessionBridge { /// True when the host has published a fresh ready contract (stricter than heartbeat alone). public static func isHostReady(defaults: UserDefaults? = nil) -> Bool { let store = resolvedDefaults(defaults) + if let snapshot = readySnapshot(defaults: store) { + guard snapshot.ready else { return false } + guard isHostReachable(defaults: store) else { return false } + if let readyAt = snapshot.readyAt { + let skew = abs(snapshot.heartbeatAt - readyAt) + guard skew <= FlowSessionKeys.hostReadyMaxHeartbeatSkew else { return false } + } + return true + } guard isHostReachable(defaults: store) else { return false } return store.bool(forKey: FlowSessionKeys.flowHostReady) } @@ -393,6 +661,10 @@ public enum FlowSessionBridge { store.removeObject(forKey: FlowSessionKeys.flowHeartbeat) store.removeObject(forKey: FlowSessionKeys.keyboardRecordingState) store.removeObject(forKey: FlowSessionKeys.transcriptionLanguage) + store.removeObject(forKey: FlowSessionKeys.flowCommandPayload) + store.removeObject(forKey: FlowSessionKeys.flowResultPayload) + store.removeObject(forKey: FlowSessionKeys.flowAckPayload) + store.removeObject(forKey: FlowSessionKeys.flowReadyPayload) clearTranscription(defaults: store) store.removeObject(forKey: FlowSessionKeys.audioLevels) store.removeObject(forKey: FlowSessionKeys.pendingHostBundleId) diff --git a/OSGKeyboardShared/Services/FlowSessionDarwin.swift b/OSGKeyboardShared/Services/FlowSessionDarwin.swift index 4c5d942..a80e063 100644 --- a/OSGKeyboardShared/Services/FlowSessionDarwin.swift +++ b/OSGKeyboardShared/Services/FlowSessionDarwin.swift @@ -8,6 +8,8 @@ import Foundation public enum FlowSessionDarwin { public static let notificationName = "com.osgkeyboard.flow.session.changed" + /// Posted when the keyboard writes a command for the host app. + public static let commandNotificationName = "com.osgkeyboard.flow.command.changed" /// Posted when the host app writes a transcription result or error. public static let transcriptionNotificationName = "com.osgkeyboard.flow.transcription.changed" /// Posted when the host app publishes or clears the ready contract. @@ -23,6 +25,16 @@ public enum FlowSessionDarwin { ) } + public static func postCommandChanged() { + CFNotificationCenterPostNotification( + CFNotificationCenterGetDarwinNotifyCenter(), + CFNotificationName(commandNotificationName as CFString), + nil, + nil, + true + ) + } + public static func postTranscriptionChanged() { CFNotificationCenterPostNotification( CFNotificationCenterGetDarwinNotifyCenter(), diff --git a/OSGKeyboardShared/Services/FlowSessionKeys.swift b/OSGKeyboardShared/Services/FlowSessionKeys.swift index 6d4403a..f7513a3 100644 --- a/OSGKeyboardShared/Services/FlowSessionKeys.swift +++ b/OSGKeyboardShared/Services/FlowSessionKeys.swift @@ -7,6 +7,10 @@ import Foundation public enum FlowSessionKeys { + public static let flowCommandPayload = "flow.commandPayload.v1" + public static let flowResultPayload = "flow.resultPayload.v1" + public static let flowAckPayload = "flow.ackPayload.v1" + public static let flowReadyPayload = "flow.readyPayload.v1" public static let flowSessionActive = "flow.flowSessionActive" public static let flowSessionExpires = "flow.flowSessionExpires" public static let flowHeartbeat = "flow.flowHeartbeat" @@ -74,7 +78,7 @@ public enum FlowSessionKeys { } /// Structured host → keyboard transcription failure kind. - public enum TranscriptionErrorKind: String, Sendable, Equatable { + public enum TranscriptionErrorKind: String, Sendable, Equatable, Codable { case noSpeech case recognitionInterrupted case audioUnavailable diff --git a/OSGKeyboardShared/Services/LocalASRBiasAdapter.swift b/OSGKeyboardShared/Services/LocalASRBiasAdapter.swift new file mode 100644 index 0000000..351ab49 --- /dev/null +++ b/OSGKeyboardShared/Services/LocalASRBiasAdapter.swift @@ -0,0 +1,159 @@ +// LocalASRBiasAdapter.swift +// OSGKeyboard · Shared +// +// Maps `PersonalDictionary` + builtin lexicon + runtime context into the +// layered bias outputs consumed by local ASR, correction, and polish. + +import Foundation + +public enum LocalASRBiasAdapter { + + /// Bundle IDs where computer-science vocabulary is especially likely. + private static let codeEditorBundleIDs: Set = [ + "com.apple.dt.Xcode", + "com.microsoft.VSCode", + "com.google.android.studio", + "com.jetbrains.intellij", + "com.jetbrains.AppCode", + "com.sublimetext.4", + "com.apple.Terminal", + "com.googlecode.iterm2", + "dev.warp.Warp-Stable", + ] + + public static func adapt( + _ request: LocalASRBiasRequest, + lexicon: BuiltinLexiconIndex = .shared + ) -> LocalASRBiasPayload { + let capabilities = request.capabilities + let dictionary = request.dictionary + + var selectedSources = ["user"] + let preferredSources = Self.preferredLexiconSources(for: request.frontAppBundleId) + if preferredSources != nil { + selectedSources.append("builtin-computer") + } else { + selectedSources.append("builtin-top") + } + + let userSorted = dictionary.effectiveEntries.sorted { $0.usageCount > $1.usageCount } + var mergedTerms: [String] = [] + var seen = Set() + func appendTerm(_ term: String) { + let trimmed = term.trimmingCharacters(in: .whitespacesAndNewlines) + guard !trimmed.isEmpty else { return } + let key = trimmed.lowercased() + guard seen.insert(key).inserted else { return } + mergedTerms.append(trimmed) + } + + for entry in userSorted { + appendTerm(entry.term) + } + let userTermCount = mergedTerms.count + + let builtinWords = lexicon.topTerms( + limit: request.builtinASRLimit, + minimumWeight: 4, + preferredSources: preferredSources + ) + let beforeBuiltin = mergedTerms.count + for word in builtinWords { + appendTerm(word) + } + let builtinTermCount = mergedTerms.count - beforeBuiltin + + var hardHotwords: [String] = [] + switch capabilities.hotwordMode { + case .perRequest, .recognizerScoped: + let cap = max(capabilities.maxHotwordCount, 1) + hardHotwords = Self.hardHotwordList(from: mergedTerms, maxCount: cap) + case .cloudVocabulary: + hardHotwords = dictionary.asrHotwords(maxCount: max(capabilities.maxHotwordCount, 1)) + case .none, .promptOnly: + break + } + + var promptBias: String? + var truncated = false + var truncationReason: String? + + if capabilities.hotwordMode == .promptOnly, capabilities.maxPromptCharacters > 0 { + let built = Self.buildPromptBias( + dictionary: dictionary, + builtinTerms: builtinWords, + maxCharacters: capabilities.maxPromptCharacters + ) + if built.count > capabilities.maxPromptCharacters { + truncated = true + truncationReason = "promptBias exceeded \(capabilities.maxPromptCharacters) characters" + } + promptBias = built.isEmpty ? nil : built + } + + let polishFragment = Self.buildPolishFragment( + dictionary: dictionary, + builtinTerms: builtinWords, + maxTerms: request.builtinPolishLimit + ) + + let correctionPairs = dictionary.localCorrectionPairs() + + return LocalASRBiasPayload( + hardHotwords: hardHotwords, + promptBias: promptBias, + corpusContext: promptBias, + polishFragment: polishFragment, + correctionPairs: correctionPairs, + diagnostics: LocalASRBiasDiagnostics( + userTermCount: userTermCount, + builtinTermCount: builtinTermCount, + truncated: truncated, + truncationReason: truncationReason, + selectedSources: selectedSources + ) + ) + } + + // MARK: - Private + + private static func preferredLexiconSources(for bundleId: String?) -> Set? { + guard let bundleId, codeEditorBundleIDs.contains(bundleId) else { return nil } + return ["computer_terms"] + } + + private static func hardHotwordList(from terms: [String], maxCount: Int) -> [String] { + Array(terms.prefix(maxCount)) + } + + private static func buildPromptBias( + dictionary: PersonalDictionary, + builtinTerms: [String], + maxCharacters: Int + ) -> String { + let userBias = dictionary.asrPromptBias(maxCharacters: maxCharacters) + let userTermsLower = Set(dictionary.effectiveEntries.map { $0.term.lowercased() }) + let extras = builtinTerms.filter { !userTermsLower.contains($0.lowercased()) } + guard !extras.isEmpty else { return userBias } + + let extraBlock = "常见技术词汇：\(extras.prefix(80).joined(separator: "、"))" + if userBias.isEmpty { + return String(extraBlock.prefix(maxCharacters)) + } + let combined = userBias + "；" + extraBlock + return String(combined.prefix(maxCharacters)) + } + + private static func buildPolishFragment( + dictionary: PersonalDictionary, + builtinTerms: [String], + maxTerms: Int + ) -> String { + let userTermsLower = Set(dictionary.effectiveEntries.map { $0.term.lowercased() }) + let extras = builtinTerms + .filter { !userTermsLower.contains($0.lowercased()) } + .prefix(maxTerms) + guard !extras.isEmpty else { return "" } + return "内置技术词汇参考（需原样保留）：\(extras.joined(separator: "、"))" + } +} diff --git a/OSGKeyboardShared/Services/LocalASRBiasDiagnosticsStore.swift b/OSGKeyboardShared/Services/LocalASRBiasDiagnosticsStore.swift new file mode 100644 index 0000000..8461b54 --- /dev/null +++ b/OSGKeyboardShared/Services/LocalASRBiasDiagnosticsStore.swift @@ -0,0 +1,56 @@ +// LocalASRBiasDiagnosticsStore.swift +// OSGKeyboard · Shared +// +// Persists the most recent local ASR bias diagnostics for settings / debug UI. + +import Foundation + +public struct LocalASRBiasDiagnosticsSnapshot: Codable, Sendable, Equatable { + public var capturedAt: Date + public var modelId: String? + public var backendLabel: String? + public var diagnostics: LocalASRBiasDiagnostics + public var hotwordCount: Int + public var promptBiasLength: Int + + public init( + capturedAt: Date = Date(), + modelId: String? = nil, + backendLabel: String? = nil, + diagnostics: LocalASRBiasDiagnostics, + hotwordCount: Int = 0, + promptBiasLength: Int = 0 + ) { + self.capturedAt = capturedAt + self.modelId = modelId + self.backendLabel = backendLabel + self.diagnostics = diagnostics + self.hotwordCount = hotwordCount + self.promptBiasLength = promptBiasLength + } +} + +public enum LocalASRBiasDiagnosticsStore { + private static let defaultsKey = "mac.localASR.lastBiasDiagnostics" + + public static func save(payload: LocalASRBiasPayload, modelId: String?, backendLabel: String?) { + let snapshot = LocalASRBiasDiagnosticsSnapshot( + modelId: modelId, + backendLabel: backendLabel, + diagnostics: payload.diagnostics, + hotwordCount: payload.hardHotwords.count, + promptBiasLength: payload.promptBias?.count ?? 0 + ) + guard let data = try? JSONEncoder().encode(snapshot) else { return } + UserDefaults.standard.set(data, forKey: defaultsKey) + } + + public static func load() -> LocalASRBiasDiagnosticsSnapshot? { + guard let data = UserDefaults.standard.data(forKey: defaultsKey) else { return nil } + return try? JSONDecoder().decode(LocalASRBiasDiagnosticsSnapshot.self, from: data) + } + + public static func clear() { + UserDefaults.standard.removeObject(forKey: defaultsKey) + } +} diff --git a/OSGKeyboardShared/Services/LocalASRInstalledManifestIO.swift b/OSGKeyboardShared/Services/LocalASRInstalledManifestIO.swift new file mode 100644 index 0000000..5aaff4f --- /dev/null +++ b/OSGKeyboardShared/Services/LocalASRInstalledManifestIO.swift @@ -0,0 +1,32 @@ +// LocalASRInstalledManifestIO.swift +// OSGKeyboard · Shared + +import Foundation + +public enum LocalASRInstalledManifestIO { + + public static func manifestURL(fileManager: FileManager = .default) -> URL { + let appSupport = fileManager.urls(for: .applicationSupportDirectory, in: .userDomainMask).first! + return appSupport + .appendingPathComponent("OSGKeyboard/LocalASRModels/installed-manifest.json") + } + + public static func load(defaultModelId: String, fileManager: FileManager = .default) -> LocalASRInstalledManifest { + let url = manifestURL(fileManager: fileManager) + guard let data = try? Data(contentsOf: url), + let manifest = try? JSONDecoder().decode(LocalASRInstalledManifest.self, from: data) else { + return LocalASRInstalledManifest(selectedModelId: defaultModelId) + } + return manifest + } + + public static func save(_ manifest: LocalASRInstalledManifest, fileManager: FileManager = .default) throws { + let url = manifestURL(fileManager: fileManager) + try fileManager.createDirectory(at: url.deletingLastPathComponent(), withIntermediateDirectories: true) + let encoder = JSONEncoder() + encoder.outputFormatting = [.prettyPrinted, .sortedKeys] + encoder.dateEncodingStrategy = .iso8601 + let data = try encoder.encode(manifest) + try data.write(to: url, options: .atomic) + } +} diff --git a/OSGKeyboardShared/Services/LocalASRModelDownloadClient.swift b/OSGKeyboardShared/Services/LocalASRModelDownloadClient.swift new file mode 100644 index 0000000..1d5e809 --- /dev/null +++ b/OSGKeyboardShared/Services/LocalASRModelDownloadClient.swift @@ -0,0 +1,157 @@ +// LocalASRModelDownloadClient.swift +// OSGKeyboard · Shared +// +// URLSession download with byte-level progress and pause/resume (macOS local model installs). + +import Foundation + +#if os(macOS) + +public struct LocalASRDownloadProgressUpdate: Sendable { + public let bytesReceived: Int64 + public let bytesTotal: Int64 + + public var fraction: Double { + guard bytesTotal > 0 else { return 0 } + return min(1, max(0, Double(bytesReceived) / Double(bytesTotal))) + } +} + +/// Controls an in-flight URLSession download; supports pause via resume data. +public final class LocalASRModelDownloadController: NSObject, URLSessionDownloadDelegate, @unchecked Sendable { + private let destinationURL: URL + private let onProgress: @Sendable (LocalASRDownloadProgressUpdate) -> Void + private lazy var delegateSession: URLSession = { + URLSession(configuration: .default, delegate: self, delegateQueue: nil) + }() + + private var remoteURL: URL? + private var task: URLSessionDownloadTask? + private var completionContinuation: CheckedContinuation? + private var isPausing = false + private var finished = false + + init( + destinationURL: URL, + onProgress: @escaping @Sendable (LocalASRDownloadProgressUpdate) -> Void + ) { + self.destinationURL = destinationURL + self.onProgress = onProgress + super.init() + } + + /// Runs until the archive is fully written to `destinationURL` (survives pause/resume). + public func download(from remoteURL: URL) async throws { + self.remoteURL = remoteURL + try await withCheckedThrowingContinuation { (continuation: CheckedContinuation) in + completionContinuation = continuation + startTask(resumeData: nil) + } + } + + public func pause() async throws -> Data { + guard task != nil, !finished else { + throw LocalASRModelManagerError.downloadFailed("No active download to pause.") + } + return try await withCheckedThrowingContinuation { (continuation: CheckedContinuation) in + isPausing = true + task?.cancel(byProducingResumeData: { [weak self] data in + guard let self else { return } + self.isPausing = false + if let data { + continuation.resume(returning: data) + } else { + continuation.resume(throwing: LocalASRModelManagerError.downloadFailed("Pause failed.")) + } + }) + } + } + + /// Continues a paused download; `download(from:)` must still be awaiting. + public func resumeFromPause(_ resumeData: Data) { + finished = false + startTask(resumeData: resumeData) + } + + public func cancel() { + finished = true + task?.cancel() + completionContinuation?.resume(throwing: CancellationError()) + completionContinuation = nil + delegateSession.invalidateAndCancel() + } + + private func startTask(resumeData: Data?) { + if let resumeData { + task = delegateSession.downloadTask(withResumeData: resumeData) + } else if let remoteURL { + task = delegateSession.downloadTask(with: remoteURL) + } + task?.resume() + } + + // MARK: - URLSessionDownloadDelegate + + public func urlSession( + _ session: URLSession, + downloadTask: URLSessionDownloadTask, + didWriteData bytesWritten: Int64, + totalBytesWritten: Int64, + totalBytesExpectedToWrite: Int64 + ) { + onProgress( + LocalASRDownloadProgressUpdate( + bytesReceived: totalBytesWritten, + bytesTotal: max(totalBytesExpectedToWrite, 1) + ) + ) + } + + public func urlSession( + _ session: URLSession, + downloadTask: URLSessionDownloadTask, + didFinishDownloadingTo location: URL + ) { + guard !finished else { return } + finished = true + do { + let fm = FileManager.default + if fm.fileExists(atPath: destinationURL.path) { + try fm.removeItem(at: destinationURL) + } + try fm.moveItem(at: location, to: destinationURL) + completionContinuation?.resume() + } catch { + completionContinuation?.resume( + throwing: LocalASRModelManagerError.downloadFailed(error.localizedDescription) + ) + } + completionContinuation = nil + session.finishTasksAndInvalidate() + } + + public func urlSession(_ session: URLSession, task: URLSessionTask, didCompleteWithError error: Error?) { + guard !finished else { return } + if isPausing { return } + if let error { + finished = true + completionContinuation?.resume( + throwing: LocalASRModelManagerError.downloadFailed(error.localizedDescription) + ) + completionContinuation = nil + session.finishTasksAndInvalidate() + } + } +} + +public enum LocalASRModelDownloadClient { + + public static func makeController( + destinationURL: URL, + onProgress: @escaping @Sendable (LocalASRDownloadProgressUpdate) -> Void + ) -> LocalASRModelDownloadController { + LocalASRModelDownloadController(destinationURL: destinationURL, onProgress: onProgress) + } +} + +#endif diff --git a/OSGKeyboardShared/Services/LocalASRModelInstallState.swift b/OSGKeyboardShared/Services/LocalASRModelInstallState.swift new file mode 100644 index 0000000..5545c26 --- /dev/null +++ b/OSGKeyboardShared/Services/LocalASRModelInstallState.swift @@ -0,0 +1,138 @@ +// LocalASRModelInstallState.swift +// OSGKeyboard · Shared + +import Foundation + +public enum LocalASRModelInstallState { + + public static func rootDirectory(fileManager: FileManager = .default) -> URL { + let appSupport = fileManager.urls(for: .applicationSupportDirectory, in: .userDomainMask).first! + return appSupport.appendingPathComponent("OSGKeyboard/LocalASRModels", isDirectory: true) + } + + public static func installDirectory(for relativePath: String, fileManager: FileManager = .default) -> URL { + rootDirectory(fileManager: fileManager).appendingPathComponent(relativePath, isDirectory: true) + } + + public static func isInstalled( + _ model: LocalASRModelDefinition, + manualMLXPath: String?, + fileManager: FileManager = .default + ) -> Bool { + switch model.installKind { + case .manual: + guard let required = model.requiredRelativeFiles, !required.isEmpty else { return false } + let base = URL(fileURLWithPath: manualMLXPath ?? "", isDirectory: true) + guard fileManager.fileExists(atPath: base.path) else { return false } + return required.allSatisfy { fileManager.fileExists(atPath: base.appendingPathComponent($0).path) } + case .archive: + guard let relative = model.installRelativePath, + let layout = model.layout, + let baseName = model.archiveBaseName else { return false } + let root = installDirectory(for: relative, fileManager: fileManager) + .appendingPathComponent(baseName, isDirectory: true) + return validateArchiveModel(at: root, model: model, layout: layout, fileManager: fileManager) + case .runtime: + return false + } + } + + public static func modelRootURL( + _ model: LocalASRModelDefinition, + fileManager: FileManager = .default + ) -> URL? { + guard model.installKind == .archive, + let relative = model.installRelativePath, + let baseName = model.archiveBaseName else { return nil } + return installDirectory(for: relative, fileManager: fileManager) + .appendingPathComponent(baseName, isDirectory: true) + } + + public static func resolveRuntimeBinary( + runtime: LocalASRRuntimeDefinition, + fileManager: FileManager = .default + ) -> URL? { + let root = installDirectory(for: runtime.installRelativePath, fileManager: fileManager) + for candidate in runtime.binaryCandidates { + let direct = root.appendingPathComponent(candidate) + if fileManager.isExecutableFile(atPath: direct.path) { + return direct + } + } + for candidate in runtime.binaryCandidates { + let name = (candidate as NSString).lastPathComponent + if let found = findExecutable(named: name, under: root, fileManager: fileManager) { + return found + } + } + return nil + } + + public static func isRuntimeInstalled( + _ runtime: LocalASRRuntimeDefinition, + fileManager: FileManager = .default + ) -> Bool { + resolveRuntimeBinary(runtime: runtime, fileManager: fileManager) != nil + } + + // MARK: - Private + + private static func validateArchiveModel( + at root: URL, + model: LocalASRModelDefinition, + layout: LocalASRModelLayout, + fileManager: FileManager + ) -> Bool { + switch model.backend { + case .sherpaQwen3: + guard let conv = layout.convFrontend, + let encoder = layout.encoder, + let decoder = layout.decoder, + let tokenizer = layout.tokenizer else { return false } + return fileManager.fileExists(atPath: root.appendingPathComponent(conv).path) + && fileManager.fileExists(atPath: root.appendingPathComponent(encoder).path) + && fileManager.fileExists(atPath: root.appendingPathComponent(decoder).path) + && fileManager.fileExists(atPath: root.appendingPathComponent(tokenizer, isDirectory: true).path) + case .sherpaSenseVoice: + guard let onnx = layout.senseVoiceModel, + let tokens = layout.tokens else { return false } + return fileManager.fileExists(atPath: root.appendingPathComponent(onnx).path) + && fileManager.fileExists(atPath: root.appendingPathComponent(tokens).path) + default: + return false + } + } + + private static func findExecutable( + named name: String, + under root: URL, + fileManager: FileManager + ) -> URL? { + guard let enumerator = fileManager.enumerator( + at: root, + includingPropertiesForKeys: [.isExecutableKey], + options: [.skipsHiddenFiles] + ) else { return nil } + for case let url as URL in enumerator { + guard url.lastPathComponent == name else { continue } + if fileManager.isExecutableFile(atPath: url.path) { + return url + } + } + return nil + } + + public static func directoryByteCount(at url: URL, fileManager: FileManager = .default) -> Int64 { + guard let enumerator = fileManager.enumerator( + at: url, + includingPropertiesForKeys: [.fileSizeKey], + options: [.skipsHiddenFiles] + ) else { return 0 } + var total: Int64 = 0 + for case let fileURL as URL in enumerator { + let size = (try? fileURL.resourceValues(forKeys: [.fileSizeKey]).fileSize) ?? 0 + total += Int64(size) + } + return total + } +} diff --git a/OSGKeyboardShared/Services/LocalASRModelManager.swift b/OSGKeyboardShared/Services/LocalASRModelManager.swift new file mode 100644 index 0000000..022fbe4 --- /dev/null +++ b/OSGKeyboardShared/Services/LocalASRModelManager.swift @@ -0,0 +1,492 @@ +// LocalASRModelManager.swift +// OSGKeyboard · Shared +// +// Installs local ASR model archives and Sherpa runtimes under Application Support. +// Catalog is bundled; installed state is persisted in `installed-manifest.json`. + +import Foundation + +public struct LocalASRInstalledManifest: Codable, Sendable, Equatable { + public var schemaVersion: Int + public var selectedModelId: String + public var installedModelIDs: [String] + public var installedRuntimeIDs: [String] + public var updatedAt: Date + + public init( + schemaVersion: Int = 1, + selectedModelId: String, + installedModelIDs: [String] = [], + installedRuntimeIDs: [String] = [], + updatedAt: Date = Date() + ) { + self.schemaVersion = schemaVersion + self.selectedModelId = selectedModelId + self.installedModelIDs = installedModelIDs + self.installedRuntimeIDs = installedRuntimeIDs + self.updatedAt = updatedAt + } +} + +public enum LocalASRModelInstallPhase: String, Sendable, Equatable { + case idle + case downloading + case paused + case extracting + case validating + case finalizing + case failed + case completed +} + +public struct LocalASRModelInstallProgress: Sendable, Equatable { + public var phase: LocalASRModelInstallPhase + public var fraction: Double + public var message: String + public var bytesReceived: Int64? + public var bytesTotal: Int64? + public var activeItemId: String? + + public init( + phase: LocalASRModelInstallPhase, + fraction: Double, + message: String, + bytesReceived: Int64? = nil, + bytesTotal: Int64? = nil, + activeItemId: String? = nil + ) { + self.phase = phase + self.fraction = fraction + self.message = message + self.bytesReceived = bytesReceived + self.bytesTotal = bytesTotal + self.activeItemId = activeItemId + } + + public static let idle = LocalASRModelInstallProgress(phase: .idle, fraction: 0, message: "") +} + +public enum LocalASRModelManagerError: Error, LocalizedError { + case downloadFailed(String) + case extractFailed(String) + case validationFailed(String) + case runtimeMissing + case binaryMissing + + public var errorDescription: String? { + switch self { + case .downloadFailed(let detail): return "Download failed: \(detail)" + case .extractFailed(let detail): return "Extract failed: \(detail)" + case .validationFailed(let detail): return "Validation failed: \(detail)" + case .runtimeMissing: return "Sherpa runtime is not installed." + case .binaryMissing: return "Sherpa binary not found in runtime bundle." + } + } +} + +public actor LocalASRModelManager { + + public static let shared = LocalASRModelManager() + + private let fileManager = FileManager.default + private var progress = LocalASRModelInstallProgress.idle + #if os(macOS) + private var activeDownloadController: LocalASRModelDownloadController? + private var pausedResumeData: Data? + #endif + + private init() {} + + public func currentProgress() -> LocalASRModelInstallProgress { + progress + } + + #if os(macOS) + public func pauseDownload() async throws { + guard progress.phase == .downloading, let controller = activeDownloadController else { return } + let resumeData = try await controller.pause() + pausedResumeData = resumeData + progress = LocalASRModelInstallProgress( + phase: .paused, + fraction: progress.fraction, + message: progress.message, + bytesReceived: progress.bytesReceived, + bytesTotal: progress.bytesTotal, + activeItemId: progress.activeItemId + ) + } + + public func resumeDownload() async throws { + guard progress.phase == .paused, + let resumeData = pausedResumeData, + let controller = activeDownloadController else { + throw LocalASRModelManagerError.downloadFailed("No paused download to resume.") + } + pausedResumeData = nil + progress = LocalASRModelInstallProgress( + phase: .downloading, + fraction: progress.fraction, + message: progress.message, + bytesReceived: progress.bytesReceived, + bytesTotal: progress.bytesTotal, + activeItemId: progress.activeItemId + ) + controller.resumeFromPause(resumeData) + } + + public func isDownloadPaused() -> Bool { + progress.phase == .paused + } + #endif + + public func rootDirectory() -> URL { + let appSupport = fileManager.urls(for: .applicationSupportDirectory, in: .userDomainMask).first! + return appSupport.appendingPathComponent("OSGKeyboard/LocalASRModels", isDirectory: true) + } + + public func manifestURL() -> URL { + LocalASRInstalledManifestIO.manifestURL(fileManager: fileManager) + } + + public func loadManifest(defaultModelId: String) -> LocalASRInstalledManifest { + LocalASRInstalledManifestIO.load(defaultModelId: defaultModelId, fileManager: fileManager) + } + + public func saveManifest(_ manifest: LocalASRInstalledManifest) throws { + try LocalASRInstalledManifestIO.save(manifest, fileManager: fileManager) + } + + public func setSelectedModelId(_ modelId: String, catalog: LocalASRCatalogDocument) throws { + var manifest = loadManifest(defaultModelId: catalog.defaultModelId) + manifest.selectedModelId = modelId + manifest.updatedAt = Date() + try saveManifest(manifest) + } + + public func installDirectory(for relativePath: String) -> URL { + rootDirectory().appendingPathComponent(relativePath, isDirectory: true) + } + + public func isModelInstalled(_ model: LocalASRModelDefinition, manualMLXPath: String?) -> Bool { + LocalASRModelInstallState.isInstalled(model, manualMLXPath: manualMLXPath, fileManager: fileManager) + } + + public func isRuntimeInstalled(_ runtime: LocalASRRuntimeDefinition) -> Bool { + LocalASRModelInstallState.isRuntimeInstalled(runtime, fileManager: fileManager) + } + + #if os(macOS) + public func resolveRuntimeBinary(runtime: LocalASRRuntimeDefinition) -> URL? { + LocalASRModelInstallState.resolveRuntimeBinary(runtime: runtime, fileManager: fileManager) + } + + public func installModel( + _ model: LocalASRModelDefinition, + catalog: LocalASRCatalogDocument + ) async throws { + guard model.installKind == .archive, + let relative = model.installRelativePath, + let baseName = model.archiveBaseName, + let sources = model.sources, + !sources.isEmpty else { + throw LocalASRModelManagerError.validationFailed("Model is not downloadable.") + } + + progress = LocalASRModelInstallProgress( + phase: .downloading, + fraction: 0.05, + message: model.displayName, + activeItemId: model.id + ) + if model.backend == .sherpaQwen3 || model.backend == .sherpaSenseVoice { + try await ensureRuntimeInstalled(catalog: catalog) + } + let sortedSources = sources.sorted { $0.priority < $1.priority } + var lastError: Error? + for source in sortedSources { + do { + try await installArchive( + from: source.url, + installRelativePath: relative, + archiveBaseName: baseName, + layoutModel: model, + itemId: model.id, + displayName: model.displayName + ) + var manifest = loadManifest(defaultModelId: catalog.defaultModelId) + if !manifest.installedModelIDs.contains(model.id) { + manifest.installedModelIDs.append(model.id) + } + manifest.updatedAt = Date() + try saveManifest(manifest) + progress = LocalASRModelInstallProgress( + phase: .completed, + fraction: 1, + message: model.displayName, + activeItemId: model.id + ) + return + } catch { + lastError = error + } + } + progress = LocalASRModelInstallProgress( + phase: .failed, + fraction: 0, + message: lastError?.localizedDescription ?? "Download failed" + ) + throw lastError ?? LocalASRModelManagerError.downloadFailed("All mirrors failed") + } + + public func installRuntime( + _ runtime: LocalASRRuntimeDefinition, + catalog: LocalASRCatalogDocument + ) async throws { + guard let source = runtime.sources.sorted(by: { $0.priority < $1.priority }).first else { + throw LocalASRModelManagerError.downloadFailed("No runtime source configured.") + } + progress = LocalASRModelInstallProgress( + phase: .downloading, + fraction: 0.05, + message: runtime.displayName, + activeItemId: runtime.id + ) + try await installArchive( + from: source.url, + installRelativePath: runtime.installRelativePath, + archiveBaseName: runtime.installRelativePath.split(separator: "/").last.map(String.init) ?? runtime.id, + layoutModel: nil, + expectedBinaryCandidates: runtime.binaryCandidates, + itemId: runtime.id, + displayName: runtime.displayName + ) + guard isRuntimeInstalled(runtime) else { + throw LocalASRModelManagerError.binaryMissing + } + var manifest = loadManifest(defaultModelId: catalog.defaultModelId) + if !manifest.installedRuntimeIDs.contains(runtime.id) { + manifest.installedRuntimeIDs.append(runtime.id) + } + manifest.updatedAt = Date() + try saveManifest(manifest) + progress = LocalASRModelInstallProgress(phase: .completed, fraction: 1, message: runtime.displayName) + } + + public func modelRootURL(_ model: LocalASRModelDefinition) -> URL? { + LocalASRModelInstallState.modelRootURL(model, fileManager: fileManager) + } + + public func installDirectoryURL(for model: LocalASRModelDefinition) -> URL? { + guard let relative = model.installRelativePath else { return nil } + return installDirectory(for: relative) + } + + public func deleteModel( + _ model: LocalASRModelDefinition, + catalog: LocalASRCatalogDocument + ) throws { + guard model.installKind == .archive, let relative = model.installRelativePath else { return } + let dir = installDirectory(for: relative) + if fileManager.fileExists(atPath: dir.path) { + try fileManager.removeItem(at: dir) + } + var manifest = loadManifest(defaultModelId: catalog.defaultModelId) + manifest.installedModelIDs.removeAll { $0 == model.id } + if manifest.selectedModelId == model.id { + manifest.selectedModelId = catalog.defaultModelId + UserDefaults.standard.set(catalog.defaultModelId, forKey: LocalASRPreferenceKeys.selectedModelId) + } + manifest.updatedAt = Date() + try saveManifest(manifest) + if progress.activeItemId == model.id { + progress = .idle + } + } + + public func deleteRuntime( + _ runtime: LocalASRRuntimeDefinition, + catalog: LocalASRCatalogDocument + ) throws { + let dir = installDirectory(for: runtime.installRelativePath) + if fileManager.fileExists(atPath: dir.path) { + try fileManager.removeItem(at: dir) + } + var manifest = loadManifest(defaultModelId: catalog.defaultModelId) + manifest.installedRuntimeIDs.removeAll { $0 == runtime.id } + manifest.updatedAt = Date() + try saveManifest(manifest) + } + + private func setProgress(_ update: LocalASRModelInstallProgress) { + progress = update + } + + func updateDownloadProgress( + itemId: String, + displayName: String, + update: LocalASRDownloadProgressUpdate + ) { + reportDownloadProgress(itemId: itemId, displayName: displayName, update: update) + } + + private func reportDownloadProgress( + itemId: String, + displayName: String, + update: LocalASRDownloadProgressUpdate + ) { + // Download phase occupies 10%–55% of the overall install bar. + let mapped = 0.10 + update.fraction * 0.45 + progress = LocalASRModelInstallProgress( + phase: .downloading, + fraction: mapped, + message: displayName, + bytesReceived: update.bytesReceived, + bytesTotal: update.bytesTotal, + activeItemId: itemId + ) + } + #endif + + // MARK: - Private + + #if os(macOS) + private func installArchive( + from urlString: String, + installRelativePath: String, + archiveBaseName: String, + layoutModel: LocalASRModelDefinition?, + expectedBinaryCandidates: [String]? = nil, + itemId: String, + displayName: String + ) async throws { + guard let remoteURL = URL(string: urlString) else { + throw LocalASRModelManagerError.downloadFailed("Invalid URL") + } + + let stagingRoot = rootDirectory().appendingPathComponent("staging/\(UUID().uuidString)", isDirectory: true) + let destinationParent = installDirectory(for: installRelativePath) + try fileManager.createDirectory(at: stagingRoot, withIntermediateDirectories: true) + defer { try? fileManager.removeItem(at: stagingRoot) } + + let archiveURL = stagingRoot.appendingPathComponent(remoteURL.lastPathComponent) + progress = LocalASRModelInstallProgress( + phase: .downloading, + fraction: 0.10, + message: displayName, + activeItemId: itemId + ) + + do { + let controller = LocalASRModelDownloadClient.makeController(destinationURL: archiveURL) { update in + Task { + await LocalASRModelManager.shared.updateDownloadProgress( + itemId: itemId, + displayName: displayName, + update: update + ) + } + } + activeDownloadController = controller + try await controller.download(from: remoteURL) + activeDownloadController = nil + pausedResumeData = nil + } catch { + activeDownloadController = nil + pausedResumeData = nil + throw LocalASRModelManagerError.downloadFailed(error.localizedDescription) + } + + progress = LocalASRModelInstallProgress( + phase: .extracting, + fraction: 0.58, + message: displayName, + activeItemId: itemId + ) + try fileManager.createDirectory(at: destinationParent, withIntermediateDirectories: true) + let extractOK = try await extractTarBz2(archiveURL: archiveURL, destination: destinationParent) + guard extractOK else { + throw LocalASRModelManagerError.extractFailed("tar extraction failed") + } + + progress = LocalASRModelInstallProgress( + phase: .validating, + fraction: 0.82, + message: displayName, + activeItemId: itemId + ) + if let layoutModel { + guard LocalASRModelInstallState.isInstalled( + layoutModel, + manualMLXPath: nil, + fileManager: fileManager + ) else { + throw LocalASRModelManagerError.validationFailed("Required model files missing after extract.") + } + } + if let expectedBinaryCandidates { + let runtimeRoot = destinationParent + let found = expectedBinaryCandidates.contains { candidate in + let direct = runtimeRoot.appendingPathComponent(candidate) + if fileManager.isExecutableFile(atPath: direct.path) { return true } + let name = (candidate as NSString).lastPathComponent + return findExecutable(named: name, under: runtimeRoot) != nil + } + guard found else { + throw LocalASRModelManagerError.binaryMissing + } + } + + progress = LocalASRModelInstallProgress( + phase: .finalizing, + fraction: 0.95, + message: displayName, + activeItemId: itemId + ) + try? fileManager.removeItem(at: archiveURL) + } + + public func ensureRuntimeInstalled(catalog: LocalASRCatalogDocument) async throws { + guard let runtime = LocalASRModelCatalog.runtime( + for: LocalASRModelCatalog.currentRuntimePlatform(), + in: catalog + ) else { + throw LocalASRModelManagerError.runtimeMissing + } + if isRuntimeInstalled(runtime) { return } + try await installRuntime(runtime, catalog: catalog) + } + + private func findExecutable(named name: String, under root: URL) -> URL? { + guard let enumerator = fileManager.enumerator( + at: root, + includingPropertiesForKeys: [.isExecutableKey], + options: [.skipsHiddenFiles] + ) else { return nil } + for case let url as URL in enumerator { + guard url.lastPathComponent == name else { continue } + if fileManager.isExecutableFile(atPath: url.path) { + return url + } + } + return nil + } + + private func extractTarBz2(archiveURL: URL, destination: URL) async throws -> Bool { + try await withCheckedThrowingContinuation { continuation in + let process = Process() + process.executableURL = URL(fileURLWithPath: "/usr/bin/tar") + process.arguments = ["-xjf", archiveURL.path, "-C", destination.path] + process.standardOutput = FileHandle.nullDevice + process.standardError = FileHandle.nullDevice + process.terminationHandler = { proc in + continuation.resume(returning: proc.terminationStatus == 0) + } + do { + try process.run() + } catch { + continuation.resume(throwing: LocalASRModelManagerError.extractFailed(error.localizedDescription)) + } + } + } + #endif +} diff --git a/OSGKeyboardShared/Services/LocalASRPreferenceKeys.swift b/OSGKeyboardShared/Services/LocalASRPreferenceKeys.swift new file mode 100644 index 0000000..12bef80 --- /dev/null +++ b/OSGKeyboardShared/Services/LocalASRPreferenceKeys.swift @@ -0,0 +1,8 @@ +// LocalASRPreferenceKeys.swift +// OSGKeyboard · Shared + +import Foundation + +enum LocalASRPreferenceKeys { + static let selectedModelId = "mac.localASR.selectedModelId" +} diff --git a/OSGKeyboardShared/Services/LocalASRTranscriptCorrector.swift b/OSGKeyboardShared/Services/LocalASRTranscriptCorrector.swift new file mode 100644 index 0000000..2905de7 --- /dev/null +++ b/OSGKeyboardShared/Services/LocalASRTranscriptCorrector.swift @@ -0,0 +1,68 @@ +// LocalASRTranscriptCorrector.swift +// OSGKeyboard · Shared +// +// Deterministic alias → canonical term replacement between raw ASR output +// and the LLM polish step. Only applies whole-phrase matches. + +import Foundation + +public enum LocalASRTranscriptCorrector { + + /// Applies high-confidence alias replacements (longest match first). + public static func apply( + _ text: String, + pairs: [LocalASRCorrectionPair] + ) -> String { + guard !text.isEmpty, !pairs.isEmpty else { return text } + + let sorted = pairs.sorted { lhs, rhs in + if lhs.alias.count != rhs.alias.count { + return lhs.alias.count > rhs.alias.count + } + return lhs.alias.localizedCaseInsensitiveCompare(rhs.alias) == .orderedAscending + } + + var result = text + for pair in sorted { + result = replaceWholeMatches( + in: result, + alias: pair.alias, + term: pair.term + ) + } + return result + } + + // MARK: - Private + + private static func replaceWholeMatches( + in text: String, + alias: String, + term: String + ) -> String { + guard !alias.isEmpty, alias != term else { return text } + + if alias.unicodeScalars.allSatisfy({ $0.isASCII }) { + return replaceASCIIWord(in: text, alias: alias, term: term) + } + return text.replacingOccurrences(of: alias, with: term) + } + + private static func replaceASCIIWord( + in text: String, + alias: String, + term: String + ) -> String { + let escaped = NSRegularExpression.escapedPattern(for: alias) + let pattern = "(?i)(? String { let dictionary = store.personalDictionary - let dictionaryBlock = dictionary.promptFragment() + let dictionaryBlock = Self.mergedDictionaryBlock( + dictionary: dictionary, + supplement: context.dictionarySupplement + ) let contextGuideline = context.appContext.polishGuideline let intensityGuideline = context.intensity.promptGuideline let contract = Self.globalOutputContract(useChinese: shouldUseChineseGuidance(providerId: providerId)) @@ -321,6 +324,17 @@ public actor PolishingService { } } + internal static func mergedDictionaryBlock( + dictionary: PersonalDictionary, + supplement: String? + ) -> String { + let base = dictionary.promptFragment() + let extra = supplement?.trimmingCharacters(in: .whitespacesAndNewlines) ?? "" + if base.isEmpty { return extra } + if extra.isEmpty { return base } + return base + "\n" + extra + } + private func shouldUseChineseGuidance(providerId: String) -> Bool { switch providerId { case "zhipu", "moonshot", "qwen", "deepseek": diff --git a/OSGKeyboardShared/en.lproj/Shared.strings b/OSGKeyboardShared/en.lproj/Shared.strings index 02b30f9..218bd11 100644 --- a/OSGKeyboardShared/en.lproj/Shared.strings +++ b/OSGKeyboardShared/en.lproj/Shared.strings @@ -157,8 +157,8 @@ "mac.settings.recognition" = "RECOGNITION METHOD"; "mac.settings.cloudEngine" = "Cloud Engine & AI Refinement"; "mac.settings.cloudEngineDesc" = "Premium transcription via your provider's API, plus AI grammar and style polishing."; -"mac.settings.localEngine" = "Local Recognition (Qwen3-ASR)"; -"mac.settings.localEngineDesc" = "On-device ASR with Qwen3-ASR 1.7B (MLX). High privacy, zero latency."; +"mac.settings.localEngine" = "Local Recognition"; +"mac.settings.localEngineDesc" = "On-device transcription with a local model. High privacy, zero latency."; "mac.settings.localSpeechFallback" = "Local Recognition (Apple Speech)"; "mac.settings.localSpeechFallbackDesc" = "On-device Apple Speech when the Qwen3 model is not installed."; "mac.settings.about" = "About"; @@ -174,6 +174,9 @@ "mac.settings.qwen3ModelDesc" = "Folder with config.json, model.safetensors, vocab.json, and merges.txt."; "mac.settings.qwen3Browse" = "Choose folder…"; "mac.settings.qwen3Missing" = "Qwen3 model not found — using Apple Speech for now."; +"mac.settings.mlxModelMissing" = "Select a Qwen3 MLX model folder below, or choose another installed model."; +"mac.settings.selectedModelMissing" = "%@ is not installed — using Apple Speech for now."; +"mac.settings.localModelFallbackApple" = "No local model is ready — using Apple Speech for now."; "mac.settings.accessibility" = "Accessibility"; "mac.settings.accessibilityDesc" = "Required for global shortcut and auto-paste."; "mac.settings.openAccessibility" = "Open System Settings"; @@ -188,6 +191,43 @@ "mac.error.qwen3ModelMissing" = "Qwen3-ASR model not installed"; "mac.error.qwen3LoadFailed" = "Failed to load Qwen3 model: %@"; "mac.error.qwen3InferenceFailed" = "Qwen3 transcription failed: %@"; +"mac.localASR.models" = "Local ASR Models"; +"mac.localASR.modelsDesc" = "Sherpa models download directly. For MLX Qwen3, drop your converted weights into the folder opened by “Open folder”. All three share one storage directory."; +"mac.localASR.download" = "Download"; +"mac.localASR.selectFolder" = "Choose folder"; +"mac.localASR.openFolder" = "Open folder"; +"mac.localASR.pause" = "Pause"; +"mac.localASR.resume" = "Resume"; +"mac.localASR.needsFolder" = "Folder required"; +"mac.localASR.installDone" = "Install completed."; +"mac.localASR.installed" = "Installed"; +"mac.localASR.notInstalled" = "Not installed"; +"mac.localASR.hotwordsYes" = "Hotwords"; +"mac.localASR.hotwordsNo" = "No hotwords"; +"mac.localASR.catalogMissing" = "Local ASR catalog is missing from the app bundle."; +"mac.localASR.diagnostics" = "Last Bias Diagnostics"; +"mac.localASR.diagnosticsDesc" = "Captured after your most recent local dictation."; +"mac.localASR.diagEmpty" = "No local dictation yet."; +"mac.localASR.diagBackend" = "Backend"; +"mac.localASR.diagUserTerms" = "User terms"; +"mac.localASR.diagBuiltinTerms" = "Builtin terms"; +"mac.localASR.diagHotwords" = "Hotwords sent"; +"mac.localASR.diagPrompt" = "Prompt chars"; +"mac.localASR.diagTruncated" = "Prompt truncated"; +"mac.localASR.delete" = "Delete"; +"mac.localASR.deleteDone" = "Model deleted."; +"mac.localASR.redownload" = "Re-download"; +"mac.localASR.revealInFinder" = "Reveal in Finder"; +"mac.localASR.openStorage" = "Open model storage folder"; +"mac.localASR.runtime" = "Sherpa Runtime"; +"mac.localASR.runtimeDesc" = "Required for Sherpa Qwen3 and SenseVoice models. Installed automatically with those models."; +"mac.localASR.phase.downloading" = "Downloading"; +"mac.localASR.phase.paused" = "Paused"; +"mac.localASR.phase.extracting" = "Extracting"; +"mac.localASR.phase.validating" = "Validating"; +"mac.localASR.phase.finalizing" = "Finalizing"; +"mac.localASR.phase.failed" = "Failed"; +"mac.localASR.phase.completed" = "Completed"; "mac.error.accessibilityRequired" = "Enable Accessibility for OSGKeyboard in System Settings"; "mac.foregroundApp" = "Front app: %@"; "mac.sync.settingsTitle" = "iCloud Sync"; diff --git a/OSGKeyboardShared/zh-Hans.lproj/Shared.strings b/OSGKeyboardShared/zh-Hans.lproj/Shared.strings index 0aa9f21..5562931 100644 --- a/OSGKeyboardShared/zh-Hans.lproj/Shared.strings +++ b/OSGKeyboardShared/zh-Hans.lproj/Shared.strings @@ -157,8 +157,8 @@ "mac.settings.recognition" = "识别方式"; "mac.settings.cloudEngine" = "云端引擎与 AI 润色"; "mac.settings.cloudEngineDesc" = "通过服务商 API 进行高质量转写，并自动润色语法与风格。"; -"mac.settings.localEngine" = "本地识别（Qwen3-ASR）"; -"mac.settings.localEngineDesc" = "使用 Qwen3-ASR 1.7B（MLX）本地转写，高隐私、低延迟。"; +"mac.settings.localEngine" = "本地识别"; +"mac.settings.localEngineDesc" = "在本机使用本地模型转写，高隐私、低延迟。"; "mac.settings.localSpeechFallback" = "本地识别（Apple Speech）"; "mac.settings.localSpeechFallbackDesc" = "未安装 Qwen3 模型时，使用 Apple 本地语音识别。"; "mac.settings.about" = "关于"; @@ -174,6 +174,9 @@ "mac.settings.qwen3ModelDesc" = "需包含 config.json、model.safetensors、vocab.json 与 merges.txt。"; "mac.settings.qwen3Browse" = "选择文件夹…"; "mac.settings.qwen3Missing" = "未找到 Qwen3 模型，暂时使用 Apple Speech。"; +"mac.settings.mlxModelMissing" = "请在下方选择 Qwen3 MLX 模型目录，或改用其他已安装的模型。"; +"mac.settings.selectedModelMissing" = "「%@」尚未安装，暂时使用 Apple Speech。"; +"mac.settings.localModelFallbackApple" = "没有可用的本地模型，暂时使用 Apple Speech。"; "mac.settings.accessibility" = "辅助功能"; "mac.settings.accessibilityDesc" = "全局快捷键与自动粘贴需要此权限。"; "mac.settings.openAccessibility" = "打开系统设置"; @@ -188,6 +191,43 @@ "mac.error.qwen3ModelMissing" = "未安装 Qwen3-ASR 模型"; "mac.error.qwen3LoadFailed" = "Qwen3 模型加载失败：%@"; "mac.error.qwen3InferenceFailed" = "Qwen3 转写失败：%@"; +"mac.localASR.models" = "本地 ASR 模型"; +"mac.localASR.modelsDesc" = "Sherpa 模型可直接下载；MLX Qwen3 请将转换好的权重放入「打开目录」指向的文件夹。三个模型共用同一存储目录。"; +"mac.localASR.download" = "下载"; +"mac.localASR.selectFolder" = "选择目录"; +"mac.localASR.openFolder" = "打开目录"; +"mac.localASR.pause" = "暂停"; +"mac.localASR.resume" = "继续"; +"mac.localASR.needsFolder" = "需选择目录"; +"mac.localASR.installDone" = "安装完成。"; +"mac.localASR.installed" = "已安装"; +"mac.localASR.notInstalled" = "未安装"; +"mac.localASR.hotwordsYes" = "支持热词"; +"mac.localASR.hotwordsNo" = "无热词"; +"mac.localASR.catalogMissing" = "应用包内缺少本地 ASR 模型目录。"; +"mac.localASR.diagnostics" = "最近一次词库诊断"; +"mac.localASR.diagnosticsDesc" = "在上一轮本地听写后记录。"; +"mac.localASR.diagEmpty" = "尚无本地听写记录。"; +"mac.localASR.diagBackend" = "后端"; +"mac.localASR.diagUserTerms" = "用户词条"; +"mac.localASR.diagBuiltinTerms" = "内置词条"; +"mac.localASR.diagHotwords" = "热词数量"; +"mac.localASR.diagPrompt" = "Prompt 字符"; +"mac.localASR.diagTruncated" = "Prompt 已截断"; +"mac.localASR.delete" = "删除"; +"mac.localASR.deleteDone" = "模型已删除。"; +"mac.localASR.redownload" = "重新下载"; +"mac.localASR.revealInFinder" = "在 Finder 中显示"; +"mac.localASR.openStorage" = "打开模型存储目录"; +"mac.localASR.runtime" = "Sherpa 运行时"; +"mac.localASR.runtimeDesc" = "Sherpa Qwen3 与 SenseVoice 模型需要此运行时；下载上述模型时会自动安装。"; +"mac.localASR.phase.downloading" = "下载中"; +"mac.localASR.phase.paused" = "已暂停"; +"mac.localASR.phase.extracting" = "解压中"; +"mac.localASR.phase.validating" = "校验中"; +"mac.localASR.phase.finalizing" = "完成安装"; +"mac.localASR.phase.failed" = "失败"; +"mac.localASR.phase.completed" = "已完成"; "mac.error.accessibilityRequired" = "请在系统设置中为 OSGKeyboard 启用辅助功能"; "mac.foregroundApp" = "前台应用：%@"; "mac.sync.settingsTitle" = "iCloud 同步"; diff --git a/OSGKeyboardTests/FlowSessionBridgeTests.swift b/OSGKeyboardTests/FlowSessionBridgeTests.swift index 272cf61..286c3f5 100644 --- a/OSGKeyboardTests/FlowSessionBridgeTests.swift +++ b/OSGKeyboardTests/FlowSessionBridgeTests.swift @@ -141,6 +141,7 @@ final class FlowSessionBridgeTests: XCTestCase { func testDarwinNotificationPostsWithoutCrashing() { FlowSessionDarwin.postSessionChanged() + FlowSessionDarwin.postCommandChanged() FlowSessionDarwin.postHostReadyChanged() } @@ -183,4 +184,112 @@ final class FlowSessionBridgeTests: XCTestCase { XCTAssertFalse(defaults.bool(forKey: FlowSessionKeys.flowHostReady)) XCTAssertFalse(FlowSessionBridge.isHostReady(defaults: defaults)) } + + func testFlowCommandRoundTrip() { + let defaults = makeDefaults() + let sessionId = UUID() + let utteranceId = UUID() + let command = FlowCommand( + sessionId: sessionId, + utteranceId: utteranceId, + commandSeq: 42, + action: .startRecording, + localeId: "zh-Hans", + createdAt: 123 + ) + + FlowSessionBridge.writeCommand(command, defaults: defaults) + + XCTAssertEqual(FlowSessionBridge.latestCommand(defaults: defaults), command) + } + + func testFlowResultRoundTripPreservesUtteranceIdentity() { + let defaults = makeDefaults() + let sessionId = UUID() + let utteranceId = UUID() + let result = FlowResult( + sessionId: sessionId, + utteranceId: utteranceId, + commandSeq: 43, + status: .final, + text: "hello", + warning: "raw fallback", + createdAt: 124 + ) + + FlowSessionBridge.writeResult(result, defaults: defaults) + + XCTAssertEqual(FlowSessionBridge.latestResult(defaults: defaults), result) + FlowSessionBridge.clearResult(defaults: defaults) + XCTAssertNil(FlowSessionBridge.latestResult(defaults: defaults)) + } + + func testFlowAckRoundTrip() { + let defaults = makeDefaults() + let ack = FlowAck( + sessionId: UUID(), + utteranceId: UUID(), + commandSeq: 44, + consumedAt: 125 + ) + + FlowSessionBridge.writeAck(ack, defaults: defaults) + + XCTAssertEqual(FlowSessionBridge.latestAck(defaults: defaults), ack) + } + + func testReadySnapshotDrivesHostReady() { + let defaults = makeDefaults() + let sessionId = UUID() + let now = Date().timeIntervalSince1970 + FlowSessionBridge.markSessionActive(duration: 60, sessionId: sessionId, defaults: defaults) + let snapshot = FlowReadySnapshot( + sessionId: sessionId, + ready: true, + reason: .ready, + heartbeatAt: now, + readyAt: now, + audioProofAt: now, + engineMode: "local", + localeId: "zh-Hans", + sessionExpiresAt: now + 60 + ) + + FlowSessionBridge.writeReadySnapshot(snapshot, defaults: defaults) + + XCTAssertEqual(FlowSessionBridge.readySnapshot(defaults: defaults), snapshot) + XCTAssertTrue(FlowSessionBridge.isHostReady(defaults: defaults)) + } + + func testClearFlowStateRemovesProtocolPayloads() { + let defaults = makeDefaults() + let sessionId = UUID() + let utteranceId = UUID() + FlowSessionBridge.writeCommand( + FlowCommand( + sessionId: sessionId, + utteranceId: utteranceId, + commandSeq: 1, + action: .startRecording, + localeId: "en-US" + ), + defaults: defaults + ) + FlowSessionBridge.writeResult( + FlowResult( + sessionId: sessionId, + utteranceId: utteranceId, + commandSeq: 1, + status: .partial, + text: "hello" + ), + defaults: defaults + ) + + FlowSessionBridge.clearFlowState(defaults: defaults) + + XCTAssertNil(FlowSessionBridge.latestCommand(defaults: defaults)) + XCTAssertNil(FlowSessionBridge.latestResult(defaults: defaults)) + XCTAssertNil(FlowSessionBridge.readySnapshot(defaults: defaults)) + } } diff --git a/OSGKeyboardTests/LocalASRBiasAdapterTests.swift b/OSGKeyboardTests/LocalASRBiasAdapterTests.swift new file mode 100644 index 0000000..8fbae04 --- /dev/null +++ b/OSGKeyboardTests/LocalASRBiasAdapterTests.swift @@ -0,0 +1,124 @@ +// LocalASRBiasAdapterTests.swift +// OSGKeyboardTests + +import XCTest +@testable import OSGKeyboardShared + +final class LocalASRBiasAdapterTests: XCTestCase { + + private func makeFixtureLexicon() throws -> BuiltinLexiconIndex { + let dir = FileManager.default.temporaryDirectory + .appendingPathComponent("osg-phrases-\(UUID().uuidString)", isDirectory: true) + try FileManager.default.createDirectory(at: dir, withIntermediateDirectories: true) + let url = dir.appendingPathComponent("phrases.tsv") + let tsv = """ + word\tpinyin\tsource\tweight + SwiftUI\tswift ui\tcomputer_terms\t5 + Kubernetes\tku bo ne si\tcomputer_terms\t5 + 一致性\tyi zhi xing\tcomputer_terms\t5 + """ + try tsv.write(to: url, atomically: true, encoding: .utf8) + addTeardownBlock { + try? FileManager.default.removeItem(at: dir) + } + return BuiltinLexiconIndex(fixtureURL: url) + } + + func testAdaptBuildsPromptBiasForQwen3MLX() throws { + let lexicon = try makeFixtureLexicon() + var dict = PersonalDictionary.empty + _ = dict.upsertManual(term: "Cursor") + dict.updateAliases(for: dict.entries[0].id, aliases: ["cursor"]) + + let payload = LocalASRBiasAdapter.adapt( + LocalASRBiasRequest( + dictionary: dict, + locale: Locale(identifier: "zh-CN"), + capabilities: .qwen3MLX + ), + lexicon: lexicon + ) + + XCTAssertNotNil(payload.promptBias) + XCTAssertTrue(payload.promptBias?.contains("Cursor") == true) + XCTAssertTrue(payload.promptBias?.contains("SwiftUI") == true) + XCTAssertEqual(payload.diagnostics.userTermCount, 2) // OSGKeyboard system + Cursor + XCTAssertGreaterThan(payload.diagnostics.builtinTermCount, 0) + } + + func testAdaptProducesPolishFragmentWithoutUserDuplicates() throws { + let lexicon = try makeFixtureLexicon() + var dict = PersonalDictionary.empty + _ = dict.upsertManual(term: "SwiftUI") + + let payload = LocalASRBiasAdapter.adapt( + LocalASRBiasRequest( + dictionary: dict, + locale: Locale(identifier: "zh-CN"), + capabilities: .qwen3MLX + ), + lexicon: lexicon + ) + + XCTAssertFalse(payload.polishFragment.contains("SwiftUI")) + XCTAssertTrue(payload.polishFragment.contains("Kubernetes")) + } + + func testCorrectionPairsFromAliases() { + var dict = PersonalDictionary.empty + _ = dict.upsertManual(term: "Kubernetes") + dict.updateAliases(for: dict.entries[0].id, aliases: ["k8s"]) + + let payload = LocalASRBiasAdapter.adapt( + LocalASRBiasRequest( + dictionary: dict, + locale: Locale(identifier: "zh-CN"), + capabilities: .qwen3MLX + ), + lexicon: BuiltinLexiconIndex.shared + ) + + XCTAssertEqual(payload.correctionPairs.count, 1) + XCTAssertEqual(payload.correctionPairs[0].alias, "k8s") + XCTAssertEqual(payload.correctionPairs[0].term, "Kubernetes") + } + + func testTranscriptCorrectorReplacesASCIIAlias() { + let pairs = [LocalASRCorrectionPair(alias: "k8s", term: "Kubernetes")] + let result = LocalASRTranscriptCorrector.apply( + "部署 k8s 集群", + pairs: pairs + ) + XCTAssertEqual(result, "部署 Kubernetes 集群") + } + + func testTranscriptCorrectorSkipsPartialASCIIMatch() { + let pairs = [LocalASRCorrectionPair(alias: "k8s", term: "Kubernetes")] + let result = LocalASRTranscriptCorrector.apply( + "xk8s集群", + pairs: pairs + ) + XCTAssertEqual(result, "xk8s集群") + } + + func testBuiltinLexiconParsesTSV() { + let terms = BuiltinLexiconIndex.parseTSV( + "word\tpinyin\tsource\tweight\nFoo\tfoo\tcomputer_terms\t5\n" + ) + XCTAssertEqual(terms.count, 1) + XCTAssertEqual(terms[0].word, "Foo") + XCTAssertEqual(terms[0].weight, 5) + } + + func testPolishingServiceMergesDictionarySupplement() { + let dict = PersonalDictionary(entries: [ + PersonalDictionary.Entry(term: "Cursor", category: .productName, source: .manual), + ]) + let merged = PolishingService.mergedDictionaryBlock( + dictionary: dict, + supplement: "内置技术词汇参考：SwiftUI" + ) + XCTAssertTrue(merged.contains("Cursor")) + XCTAssertTrue(merged.contains("SwiftUI")) + } +} diff --git a/OSGKeyboardTests/LocalASRModelCatalogTests.swift b/OSGKeyboardTests/LocalASRModelCatalogTests.swift new file mode 100644 index 0000000..4e13d7a --- /dev/null +++ b/OSGKeyboardTests/LocalASRModelCatalogTests.swift @@ -0,0 +1,88 @@ +// LocalASRModelCatalogTests.swift +// OSGKeyboardTests + +import XCTest +@testable import OSGKeyboardShared + +final class LocalASRModelCatalogTests: XCTestCase { + + func testBundledCatalogLoads() throws { + let catalog = try LocalASRModelCatalog.loadBundled() + XCTAssertEqual(catalog.schemaVersion, 1) + XCTAssertFalse(catalog.models.isEmpty) + XCTAssertTrue(catalog.models.contains { $0.id == "qwen3-mlx-1.7b" }) + XCTAssertTrue(catalog.models.contains { $0.id == "sherpa-qwen3-0.6b-int8" }) + } + + func testCapabilitiesForSherpaQwen3() throws { + let catalog = try LocalASRModelCatalog.loadBundled() + let model = try XCTUnwrap(LocalASRModelCatalog.model("sherpa-qwen3-0.6b-int8", in: catalog)) + let caps = LocalASRModelCatalog.capabilities(for: model) + XCTAssertEqual(caps.hotwordMode, .recognizerScoped) + XCTAssertTrue(model.supportsHotwords) + } + + func testManifestRoundTrip() throws { + let manifest = LocalASRInstalledManifest( + selectedModelId: "sherpa-qwen3-0.6b-int8", + installedModelIDs: ["sherpa-qwen3-0.6b-int8"] + ) + let url = FileManager.default.temporaryDirectory + .appendingPathComponent("manifest-\(UUID().uuidString).json") + defer { try? FileManager.default.removeItem(at: url) } + + let encoder = JSONEncoder() + encoder.dateEncodingStrategy = .iso8601 + try encoder.encode(manifest).write(to: url) + + let decoder = JSONDecoder() + decoder.dateDecodingStrategy = .iso8601 + let loaded = try decoder.decode(LocalASRInstalledManifest.self, from: Data(contentsOf: url)) + XCTAssertEqual(loaded.selectedModelId, manifest.selectedModelId) + XCTAssertEqual(loaded.installedModelIDs, manifest.installedModelIDs) + } + + func testBiasDiagnosticsStoreRoundTrip() { + LocalASRBiasDiagnosticsStore.clear() + let payload = LocalASRBiasPayload( + hardHotwords: ["Cursor"], + promptBias: "test", + corpusContext: nil, + polishFragment: "fragment", + correctionPairs: [], + diagnostics: LocalASRBiasDiagnostics(userTermCount: 2, builtinTermCount: 3) + ) + LocalASRBiasDiagnosticsStore.save( + payload: payload, + modelId: "qwen3-mlx-1.7b", + backendLabel: "MLX" + ) + let snapshot = LocalASRBiasDiagnosticsStore.load() + XCTAssertEqual(snapshot?.modelId, "qwen3-mlx-1.7b") + XCTAssertEqual(snapshot?.diagnostics.userTermCount, 2) + XCTAssertEqual(snapshot?.hotwordCount, 1) + LocalASRBiasDiagnosticsStore.clear() + } + + func testSherpaAdapterProducesHardHotwords() throws { + let fixtureURL = FileManager.default.temporaryDirectory + .appendingPathComponent("phrases-\(UUID().uuidString).tsv") + try "word\tpinyin\tsource\tweight\nSwiftUI\tswift ui\tcomputer_terms\t5\n" + .write(to: fixtureURL, atomically: true, encoding: .utf8) + defer { try? FileManager.default.removeItem(at: fixtureURL) } + + var dict = PersonalDictionary.empty + _ = dict.upsertManual(term: "Kubernetes") + + let payload = LocalASRBiasAdapter.adapt( + LocalASRBiasRequest( + dictionary: dict, + locale: Locale(identifier: "zh-CN"), + capabilities: .sherpaQwen3 + ), + lexicon: BuiltinLexiconIndex(fixtureURL: fixtureURL) + ) + XCTAssertFalse(payload.hardHotwords.isEmpty) + XCTAssertTrue(payload.hardHotwords.contains("Kubernetes")) + } +} diff --git a/docs/local-asr-architecture.md b/docs/local-asr-architecture.md new file mode 100644 index 0000000..d67bd8f --- /dev/null +++ b/docs/local-asr-architecture.md @@ -0,0 +1,585 @@ +# OSGKeyboard 本地 ASR 技术架构 + +> **文档状态**：架构规划（非实现规格） +> **适用范围**：macOS 本地听写；与 iOS 键盘扩展、云 ASR 路径的关系见各节说明。 +> **核心结论**：短期不换主模型，优先打通 **词库感知管道**；中期用 POC 验证 **Sherpa Qwen3 hard hotwords** 是否值得成为热词主线。 + +--- + +## 1. Executive Summary + +OSGKeyboard 的本地 ASR 竞争力不来自单一模型，而来自： + +1. **用户 PersonalDictionary**（term / aliases / iCloud） +2. **内置技术词库**（`phrases.tsv` ≈ 1 万词，iOS 已用于 Apple CLM） +3. **分层 bias**：ASR 偏置 → 后处理纠错 → Polish 保真 +4. **可替换的 Local ASR Provider**（Qwen3 MLX 主线，Sherpa / SenseVoice / Apple Speech 对照） + +当前最大缺口：**macOS 本地路径未消费任何词库**；云路径已通过 `PersonalDictionary+ASRBias` 完整接线。 + +推荐路线： + +| 阶段 | 动作 | +|------|------| +| **短期** | 保留 Qwen3 MLX；实现 `LocalASRBiasAdapter`；接 soft prompt + polish + aliases 后处理 | +| **中期** | ModelScope 优先的本地模型 catalog；Sherpa Qwen3 hotwords POC | +| **长期** | 按评测数据决定是否新增默认 provider 或保留 Qwen3 MLX | + +--- + +## 2. 背景与问题定义 + +### 2.1 为什么本地 ASR 不能只讨论模型 + +语音输入的「专有名词准确率」由多层共同决定： + +- **ASR 层**：听出 `Claude`、`SwiftUI`、`Qwen3-ASR` +- **后处理层**：`克劳德` → `Claude` +- **润色层**：保留品牌名、变量名，不擅自改写 + +闭源产品（Typeless 等）常把词典效果归因于云端 ASR；开源竞品（OpenLess、Typeflux、SayIt）表明：**词典必须按 backend 能力分层注入**，不能假设「一个 hotwords 数组走天下」。 + +### 2.2 OSG 相对竞品的结构性优势 + +| 能力 | OSGKeyboard | 典型开源竞品 | +|------|-------------|----------------| +| 用户词库 | `PersonalDictionary`（term + aliases + category + iCloud） | 多为 phrase-only | +| 内置领域词库 | ~10k `phrases.tsv` + iOS CLM | OpenLess preset ~20 词；SayIt server hotwords.txt ~30 词 | +| 云 ASR bias | 智谱 / 阿里 vocabulary / Whisper prompt | 单云或单 provider | +| iOS 本地 CLM | `SFCustomLanguageModelData` | macOS 路径未等价 | + +### 2.3 设计目标 + +- 离线、隐私友好的 macOS 本地听写 +- 复用 `PersonalDictionary` 与 `phrases.tsv` **源数据**（非 iOS `.bin` 直用） +- Provider 可替换；能力矩阵诚实声明（尤其热词模式） +- 模型下载可管理（**ModelScope 优先**，HF / GitHub 备用） +- 可评测、可灰度、可回退 + +### 2.4 非目标 + +- 不立即将主路径切到 FunASR Python server 或 Sherpa +- 不把 1 万词全量塞入 ASR prompt +- 不把 Polish 当作唯一纠错层 +- 不承诺未 POC 验证的模型效果 +- 第一期不强制实现 Typeflux 式「自动词库学习」（仅作可选实验设计） + +--- + +## 3. 当前架构（代码事实） + +### 3.1 端到端数据流 + +```mermaid +flowchart LR + subgraph macOS["macOS"] + Rec["MacAudioRecorder"] + Pipe["MacDictationPipeline"] + Local["MacLocalASRService"] + Cloud["CloudASRClient"] + Polish["PolishingService"] + Insert["MacTextInsertionService"] + end + Rec --> Pipe + Pipe -->|engineMode local| Local + Pipe -->|engineMode cloud| Cloud + Local --> Polish + Cloud --> Polish + Polish --> Insert +``` + +### 3.2 云路径 vs 本地路径 + +| 环节 | 云 ASR | 本地 ASR（当前） | +|------|--------|------------------| +| 入口 | `MacDictationPipeline.run` | 同左 | +| ASR | `CloudASRClientFactory` + `dictionary: store.personalDictionary` | `MacLocalASRService.transcribe(samples, locale)` **无 dictionary** | +| 词库 bias | `PersonalDictionary+ASRBias`（按 provider） | **无** | +| 润色 | `PolishingService` + `promptFragment()` | 同左（仅用词典做 polish，不经 ASR） | +| 默认模型 | 用户所选云 provider | Qwen3 MLX 1.7B；缺权重 → Apple Speech | + +关键代码： + +- [`OSGKeyboardMac/MacDictationPipeline.swift`](../OSGKeyboardMac/MacDictationPipeline.swift) — 本地分支未传 `personalDictionary` +- [`OSGKeyboardMac/MacLocalASRService.swift`](../OSGKeyboardMac/MacLocalASRService.swift) — Qwen3 / Apple Speech 二选一 +- [`OSGKeyboardShared/Models/PersonalDictionary+ASRBias.swift`](../OSGKeyboardShared/Models/PersonalDictionary+ASRBias.swift) — 云侧 `asrHotwords` / `asrPromptBias` / 阿里热词表 +- [`OSGKeyboardShared/Services/PolishingService.swift`](../OSGKeyboardShared/Services/PolishingService.swift) — 润色层消费 `dictionary.promptFragment()` + +### 3.3 iOS 词库资产（macOS 不可直接复用） + +- **内置词库**：[`OSGKeyboard/Resources/CustomLanguageModel/v1/phrases.tsv`](../OSGKeyboard/Resources/CustomLanguageModel/v1/phrases.tsv)（约 10,301 行，含 `word` / `pinyin` / `source` / `weight`） +- **Apple CLM**：[`OSGKeyboardShared/Services/CustomLanguageModelManager.swift`](../OSGKeyboardShared/Services/CustomLanguageModelManager.swift) → `SFSpeechLanguageModel.Configuration`，仅 iOS 26+ 中文本地 ASR 路径 +- **结论**：macOS 需从 TSV + `PersonalDictionary` **重新适配**为 Qwen3 prompt / Sherpa hotwords / polish fragment，不能加载 `OSGKeyboardCLM.bin` + +### 3.4 本地模型现状 + +- 默认路径：`~/Library/Application Support/OSGKeyboard/models/qwen3-asr-1.7b-mlx` +- 校验：`config.json`、`model.safetensors`、`vocab.json`、`merges.txt` +- Qwen3 引擎：[`MacQwen3LocalASR.swift`](../OSGKeyboardMac/MacQwen3LocalASR.swift) / [`MacQwen3ASREngine.swift`](../OSGKeyboardMac/MacQwen3ASREngine.swift) — **无 prompt / hotword 入参** + +--- + +## 4. 开源竞品源码观察 + +基于 GitHub topic `typeless-alternative` 及关联仓库（2026-03 快照）。 + +### 4.1 对比总表 + +| 项目 | 技术栈 | 本地 ASR | 热词 / 词库 | 可借鉴 | 不宜照搬 | +|------|--------|----------|-------------|--------|----------| +| **OpenLess** | Tauri/Rust | Qwen3 C 引擎、Apple Speech、Sherpa（Win） | 火山 `context.hotwords`；Whisper prompt；Polish hotword block；**本地 Qwen3 未接词典** | 多 provider + polish 双层 | 本地热词叙事过度乐观 | +| **Typeflux** | Swift/macOS | SenseVoice、FunASR、Qwen3 Sherpa CLI、WhisperKit | `VocabularyStore` cap 500；Doubao hotwords；Whisper prompt；**Sherpa 本地无热词**；自动项目词 + 编辑后学习 | Swift 原生、词库排序、自动学习（作实验） | Sherpa 仅 CLI 离线，无 hotwords 接线 | +| **SayIt** | Tauri + FastAPI | sherpa-onnx Rust：**Qwen3 recognizer 创建时写 hotwords** | 内置主题词 + 自定义；云 Qwen corpus；server `hotwords.txt` | **本地 Qwen3 hard hotwords 实证**；跨 provider `StartOptions.hotwords` | 服务端 vLLM 非 macOS 客户端主线 | +| **VoiceSnap** | Go/Wails | SenseVoice + sherpa-onnx | **无个性化词库** | 离线体验：静音截断、剪贴板保护、填充词过滤 | 无万级词库场景 | +| **OpenBroca** | Electron | Sherpa 等 | Dictionary hotword/replacement | **one-shot `recognize` first**；model catalog、sha256、selected model | Electron 栈 | + +### 4.2 对 OSG 的启示 + +1. **学架构，不学「本地已完整支持热词」的 README 叙事**（OpenLess 本地 Qwen3 缺口与 OSG 类似）。 +2. **SayIt 证明**：sherpa-onnx `OfflineQwen3ASRModelConfig.hotwords` 可在 recognizer 创建时注入；热词变化需 **重建 recognizer**（缓存 key 含 hotwords 字符串）。 +3. **Typeflux 证明**：`activeTerms()` 上限 500 + 动态排序；但 Sherpa Qwen3/SenseVoice 命令行路径**未传** vocabulary prompt。 +4. **OpenBroca 证明**：runtime 不得静默选「目录里第一个模型」；须 `selectedModelId` + manifest。 + +--- + +## 5. 目标架构 + +### 5.1 管道总览 + +```mermaid +flowchart TD + audioSamples["Audio Samples 16kHz"] --> macPipeline["MacDictationPipeline"] + personalDict["PersonalDictionary"] --> biasAdapter["LocalASRBiasAdapter"] + phrasesTSV["phrases.tsv Index"] --> biasAdapter + runtimeCtx["Runtime Context locale app recentHits"] --> biasAdapter + providerCap["Provider Capability"] --> biasAdapter + + biasAdapter --> hotwords["hardHotwords"] + biasAdapter --> promptBias["promptBias"] + biasAdapter --> corpusText["corpusContext"] + biasAdapter --> correctionPairs["correctionPairs"] + biasAdapter --> polishFrag["polishFragment"] + biasAdapter --> diag["diagnostics"] + + macPipeline --> recognize["LocalASRProvider.recognize"] + hotwords --> recognize + promptBias --> recognize + corpusText --> recognize + + recognize --> rawTranscript["Raw Transcript"] + rawTranscript --> correctionLayer["Correction Layer"] + correctionPairs --> correctionLayer + correctionLayer --> polishingService["PolishingService"] + polishFrag --> polishingService + polishingService --> finalText["Final Text"] +``` + +### 5.2 三层职责边界 + +| 层 | 职责 | 禁止 | +|----|------|------| +| **ASR bias** | 提高听写阶段专有名词概率 | 承担全文语法润色 | +| **Correction** | 高置信 `aliases → term` 替换 | 凭热词表改写普通句意 | +| **Polish** | 标点、口语转书面、热词保真 | 单独承担全部专名纠错 | + +--- + +## 6. Local ASR Provider 抽象 + +### 6.1 One-shot first + +借鉴 OpenBroca：macOS 听写主路径为 **录完后一次性 `recognize`**；流式预览（`transcribe` / partial）为可选能力，非第一期必做。 + +建议协议（概念层）： + +```swift +protocol LocalASRProvider { + var capabilities: LocalASRCapabilities { get } + func recognize( + samples: [Float], + sampleRate: Int, + locale: Locale, + bias: LocalASRBiasPayload?, + options: LocalASRRecognizeOptions? + ) async throws -> LocalASRResult +} +``` + +### 6.2 能力矩阵（须诚实声明） + +除 `supportsStreaming`、`maxHotwordCount` 外，**必须区分热词模式**： + +| 字段 | 含义 | +|------|------| +| `hotwordMode` | `none` / `promptOnly` / `perRequest` / `recognizerScoped` / `cloudVocabulary` | +| `hotwordStrength` | `weak` / `medium` / `strong`（产品文案用，非科学绝对值） | +| `hotwordReloadCost` | `none` / `recognizerReload` / `modelReload` | +| `maxPromptCharacters` | soft prompt 上限 | +| `maxHotwordCount` | hard hotwords 上限 | +| `supportsLanguageHint` | 是否接受 locale → language hint | + +### 6.3 各 Backend 定位（规划） + +| Provider | 角色 | hotwordMode（规划） | 备注 | +|----------|------|---------------------|------| +| **Qwen3 MLX** | 短期主线 | `promptOnly`（待接 `qwen_set_prompt` 等价 API） | 已有权重路径；改动面最小 | +| **Sherpa Qwen3** | 中期 POC | `recognizerScoped` | SayIt 同款；热词变更加载成本 | +| **SenseVoice** | 对照 | `none` 或弱 prompt | 速度/中文基线；非热词主线 | +| **Apple Speech** | Fallback | `none`（macOS CLM 待验证） | 系统稳定 | +| **Cloud ASR** | 质量上限 | 各云 `PersonalDictionary+ASRBias` | 非离线 | + +**暂不主推**：Sherpa Paraformer 作为热词主线（官方不支持 Paraformer hotwords，与 transducer/Qwen3 不同）。 + +--- + +## 7. LocalASRBiasAdapter 设计 + +### 7.1 输入 + +| 输入 | 说明 | +|------|------| +| `PersonalDictionary.effectiveEntries` | 用户词；最高优先级 | +| `BuiltinLexiconIndex` | 自 `phrases.tsv` 构建；按 weight / 场景筛选 | +| `locale` | `store.localeId` | +| `frontAppBundleId` | 可选；技术类 App 提升 IT 子集权重 | +| `recentHitTerms` | 历史命中统计（若已有） | +| `providerCapabilities` | 决定输出哪些字段、如何截断 | + +### 7.2 输出 `LocalASRBiasPayload` + +```swift +struct LocalASRBiasPayload { + var hardHotwords: [String] // Sherpa Qwen3、部分云 API + var promptBias: String? // Qwen3 MLX、Whisper 系 + var corpusContext: String? // Qwen 云 corpus 风格（若将来统一） + var polishFragment: String // PolishingService 追加块 + var correctionPairs: [(alias: String, term: String)] + var diagnostics: BiasDiagnostics // 供设置页 / 调试 +} + +struct BiasDiagnostics { + var userTermCount: Int + var builtinTermCount: Int + var truncated: Bool + var truncationReason: String? + var selectedSources: [String] // e.g. user, builtin-it, builtin-top +} +``` + +### 7.3 优先级与截断 + +``` +用户高频 / 最近命中 + > PersonalDictionary（全部有效 term） + > 当前 App 相关内置词（phrases 子集） + > 高 weight 内置技术词（Top-N） + > 其余内置词（仅 polish / 检索，不进 ASR） +``` + +默认建议（可 POC 调参）： + +| 输出 | 默认上限 | +|------|----------| +| `hardHotwords` | 100（Qwen3 Sherpa）；对齐 `asrHotwords(maxCount: 100)` | +| `promptBias` | 800 字符；复用 `asrPromptBias(maxCharacters:)` 逻辑 | +| ASR 层内置词 | 200–500；**不全量 1 万** | +| `correctionPairs` | aliases 全量可进后处理，但仅 **整词 / 高置信** 替换 | + +### 7.4 防污染规则 + +- 近静音、极短音频：减少或跳过内置词，保留用户词。 +- 用户词始终优先于内置词。 +- diagnostics 必须记录「为何丢弃」某批词（超 cap、provider 不支持、场景不匹配）。 + +### 7.5 与现有云代码复用 + +扩展 [`PersonalDictionary+ASRBias.swift`](../OSGKeyboardShared/Models/PersonalDictionary+ASRBias.swift) 为 **单一事实来源**，新增例如： + +- `localPromptBias(maxCharacters:builtinTerms:)` +- `correctionPairs()` +- `rankedTermsForASR(limit:builtinBoost:)` + +避免 macOS / iOS / Cloud 三套独立拼接逻辑。 + +--- + +## 8. 词库策略 + +### 8.1 PersonalDictionary + +| 字段 | ASR | Correction | Polish | +|------|-----|------------|--------| +| `term` | hotword / prompt | 标准写法 | 必须保留 | +| `aliases` | 可进 prompt 提示 | **主战场** | 语义纠错参考 | +| `category` | 排序权重 | — | 分组展示 | +| `usageCount` | 排序权重 | — | — | + +iCloud：[`PersonalDictionaryCloudSync`](../OSGKeyboardShared/Services/PersonalDictionaryCloudSync/) 保证 Mac / iOS / Extension 一致；本地 ASR 只读 `AppGroupStore.personalDictionary`。 + +### 8.2 phrases.tsv 分层 + +**不全量进入 ASR prompt。** + +| 层级 | 用途 | 规模建议 | +|------|------|----------| +| L1 ASR 高价值 | `weight >= 4` 或 curated IT 品牌缩写 | 200–500 | +| L2 场景相关 | 按 `frontApp` / 用户最近命中动态加入 | +0–100 | +| L3 全量索引 | 后处理模糊匹配、polish 检索 | ~10k | + +TSV 列：`word`、`pinyin`、`source`、`weight` — 构建索引时保留 weight 用于排序。 + +### 8.3 iOS CLM 与 macOS 关系 + +- iOS：TSV → export script → `.bin` → `CustomLanguageModelManager` +- macOS：TSV → `BuiltinLexiconIndex` → `LocalASRBiasAdapter` → Qwen3 / Sherpa / Polish +- **同一 TSV 源**，两种消费格式；不尝试把 `.bin` 喂给 Sherpa/MLX + +--- + +## 9. 自动词库学习（可选实验，非第一期） + +借鉴 Typeflux `WorkflowController+AutomaticVocabulary`： + +- 听写插入后，短时观察用户在前台可编辑框内的修改 +- LLM 或规则判断是否为「专名 / 品牌 / 大小写修正」 +- 候选进入 **待确认队列**，不直接写入 `PersonalDictionary` + +**约束（必须写进隐私说明）**： + +- 默认关闭 +- 不自动 iCloud 同步待确认项 +- 可一键清空、可审计来源 +- 拒绝：整句改写、纯语法修正、过短词条、编辑幅度过大 + +OSG 已有 `PersonalDictionary.Entry.Source.recentEdit` 与合并逻辑，可与之对齐而非新建平行存储。 + +--- + +## 10. 本地模型管理 + +### 10.1 原则 + +- **Catalog 与 Runtime 分离**：下载源只影响安装；推理只读本地 **已验证 manifest** +- **Selected model 显式**：禁止「扫描目录用第一个 onnx」 +- **完整性**：sha256 或 size 校验 + staging 目录原子发布 + +### 10.2 存储布局（建议） + +``` +~/Library/Application Support/OSGKeyboard/ + LocalASRModels/ + manifest.json # 已安装模型、版本、backend、capabilities + qwen3-mlx-1.7b/ # 当前 MLX 布局（可与现路径兼容） + sherpa-qwen3-0.6b/ + sherpa-sensevoice-small/ +``` + +### 10.3 Catalog 条目（概念） + +```json +{ + "modelId": "sherpa-qwen3-asr-0.6b-int8", + "displayName": "Qwen3-ASR 0.6B (Sherpa)", + "backend": "sherpaQwen3", + "sizeBytes": 1200000000, + "recommendedLocales": ["zh-CN", "en-US"], + "supportsHotwords": true, + "hotwordMode": "recognizerScoped", + "sources": [ + { + "type": "modelscope", + "url": "https://www.modelscope.cn/api/v1/models/.../repo?Revision=master&FilePath=...", + "sha256": "...", + "priority": 1 + }, + { + "type": "huggingface", + "url": "https://huggingface.co/...", + "priority": 2 + }, + { + "type": "github", + "url": "https://github.com/k2-fsa/sherpa-onnx/releases/download/...", + "priority": 3 + } + ] +} +``` + +### 10.4 ModelScope 策略 + +| 场景 | 策略 | +|------|------| +| 中国大陆用户默认 | **ModelScope 优先**（Qwen3-ASR、SenseVoice、FunASR 相关 ONNX） | +| 国际 / ModelScope 失败 | Hugging Face → GitHub Releases | +| 企业内网 | `custom` mirror URL（用户配置） | +| 安装流程 | download → verify → extract → validate required files → rename staging → update manifest | +| 失败 | 清理 staging / 临时文件；不留下半安装状态 | + +MLX Qwen3 权重：可继续支持用户自选目录（现状），逐步纳入统一 catalog 的 `type: mlx` 条目。 + +### 10.5 UI / 设置需求（规划） + +- 模型列表：体积、语言、安装状态、是否支持热词 +- 下载进度：phase（downloading / extracting / validating / finalizing） +- 切换模型：仅允许 **installed + manifest 合法** 的项为默认 +- 诊断：当前 provider、capability、上次 bias diagnostics + +--- + +## 11. 后端对比与决策矩阵 + +### 11.1 产品分层 + +```text +主线: Qwen3 MLX + LocalASRBiasAdapter + Polish/Correction +重点 POC: Sherpa Qwen3 hotwords +对照: SenseVoice（速度）、Apple Speech（fallback） +参考上限: Cloud ASR + PersonalDictionary +团队部署: Qwen3-ASR vLLM（SayIt 式，非客户端主线） +暂不主推: FunASR Paraformer hotwords、纯 OpenLess 本地词典叙事 +``` + +### 11.2 详细对比 + +| 维度 | Qwen3 MLX | Sherpa Qwen3 | SenseVoice | Apple Speech | Cloud | +|------|-----------|--------------|------------|--------------|-------| +| 离线 | ✅ | ✅ | ✅ | ✅ | ❌ | +| 中英混合技术词 | 强（经验性） | 待 POC | 中 | 中 | 强 | +| Hard hotwords | ❌→prompt | ✅ recognizerScoped | ❌ | ❌ | ✅ 因 provider 异 | +| 实现成本 | 低（已有） | 高（runtime 体积） | 中 | 低 | 已有 | +| 模型体积 | ~1.3GB+ | 类似 | ~350MB 级 | 0 | N/A | +| 隐私 | 本地 | 本地 | 本地 | 本地 | 依配置 | + +### 11.3 Sherpa POC 通过阈值（建议） + +相对 **当前 Qwen3 MLX + 仅 polish** 基线： + +| 指标 | 建议阈值 | +|------|----------| +| 用户热词召回率 | 提升 ≥ 20% | +| 误触发率（未说热词却被改成热词） | ≤ 2% | +| 30s 音频端到端延迟 | ≤ 基线 × 1.5 | +| 内存峰值（8GB Mac 目标机） | 可接受且无 OOM | +| 安装成功率 | 普通用户可完成 ModelScope/HF 下载 | + +未达阈值：**保留 Qwen3 MLX 主线**，Sherpa 仅作高级选项。 + +--- + +## 12. POC 评测计划 + +### 12.1 测试集 + +| 类别 | 内容 | 目的 | +|------|------|------| +| A 普通中文 | 日常口语 50 句 | 基线 WER / 误触发 | +| B 技术术语 | SwiftUI、Cursor、Qwen3-ASR 等 50 句 | 专名召回 | +| C 用户词典 | 模拟 PersonalDictionary 20 词 × 多句 | 热词核心场景 | +| D 长句润色 | 30s+ 口语 | polish 兜底 | +| E 噪声 / 短句 | 低 SNR、<2s | 防污染规则 | + +### 12.2 对照矩阵 + +| 配置 | 说明 | +|------|------| +| Baseline | Qwen3 MLX，无 bias | +| B1 | Qwen3 MLX + promptBias | +| B2 | B1 + polishFragment + correction | +| POC1 | Sherpa Qwen3 + hardHotwords | +| POC2 | SenseVoice，无 hotwords | +| Ref | 云 ASR + PersonalDictionary | + +### 12.3 指标 + +- Raw CER/WER（中文可用字错误率） +- **Hotword recall**（用户词是否出现在 raw 或 final） +- **False hotword rate** +- Final accuracy（用户主观或编辑距离） +- Latency：record end → text inserted +- Memory / CPU、模型加载时间 +- 离线可靠性（无网络完成全流程） + +--- + +## 13. 失败回退策略 + +```mermaid +flowchart TD + start["recognize 开始"] --> qwen{"Qwen3 MLX 可用?"} + qwen -->|是| qwenRun["Qwen3 + bias"] + qwen -->|否| apple["Apple Speech"] + qwenRun -->|失败| apple + qwenRun -->|成功| post["Correction + Polish"] + apple --> post + post -->|Polish 失败| raw["返回 raw transcript"] + post -->|成功| done["插入 final"] +``` + +| 条件 | 行为 | +|------|------| +| Qwen3 权重缺失 | Apple Speech（现状） | +| Qwen3 推理失败 | 可配置：重试一次 → Apple Speech | +| Sherpa 未安装 | 不回退云；提示下载 | +| 模型 manifest 损坏 | 标记 invalid，禁止设为默认 | +| Polish 失败 | 使用 raw（现状） | +| 用户禁用云 | 不静默切云 | + +--- + +## 14. 分阶段落地路线 + +| Phase | 内容 | 交付物 | +|-------|------|--------| +| **1** | 本文档定稿；`LocalASRCapabilities` + `LocalASRBiasPayload` 类型设计 | 架构文档 + ADR 可选 | +| **2** | `MacDictationPipeline` 接入 adapter；Qwen3 MLX `promptBias`；polish + correction | 实现 PR | +| **3** | `BuiltinLexiconIndex`；Top-N；diagnostics UI | 实现 PR | +| **4** | Model catalog + ModelScope 下载 + manifest | 实现 PR | +| **5** | Sherpa Qwen3 POC + 评测报告 | 决策是否默认切换 | +| **6** | 可选：自动词库学习实验（默认关） | 功能 flag | + +--- + +## 15. 风险与待确认问题 + +| 风险 | 缓解 | +|------|------| +| Qwen3 prompt bias 过弱 | POC 对比 Sherpa hard hotwords；保留 correction + polish | +| 热词过多污染识别 | cap + 场景筛选 + diagnostics | +| aliases 后处理误改 | 整词边界、低置信跳过 | +| Sherpa 分发体积 / 签名 / 公证 | 单独评估；可选按需下载 | +| Apple Speech macOS CLM | 调研 macOS 26+ 是否可接 CLM；否则仅 fallback | +| ModelScope API 变更 | 多 mirror；manifest 可更新 URL | +| 自动学习隐私 | 默认关、本地、待确认 | +| 低配 Mac 内存 | 单模型常驻策略；SenseVoice 作轻量选项 | + +**待确认**： + +1. MLX Swift API 是否暴露等价 `setPrompt`（对标 Open-Less/qwen-asr `qwen_set_prompt`） +2. Sherpa-ONNX Swift/SPM 与 App Store 公证路径 +3. `phrases.tsv` Top-N 是否按 `weight` 静态裁剪即可，或需按 App 动态检索 + +--- + +## 16. 相关代码索引 + +| 主题 | 路径 | +|------|------| +| Mac 听写管道 | `OSGKeyboardMac/MacDictationPipeline.swift` | +| 本地 ASR 入口 | `OSGKeyboardMac/MacLocalASRService.swift` | +| Qwen3 MLX | `OSGKeyboardMac/MacQwen3LocalASR.swift`, `MacQwen3ASREngine.swift` | +| Apple Speech fallback | `OSGKeyboardMac/MacSpeechLocalASR.swift` | +| 用户词库 | `OSGKeyboardShared/Models/PersonalDictionary.swift` | +| 云 bias | `OSGKeyboardShared/Models/PersonalDictionary+ASRBias.swift` | +| 内置 TSV | `OSGKeyboard/Resources/CustomLanguageModel/v1/phrases.tsv` | +| iOS CLM | `OSGKeyboardShared/Services/CustomLanguageModelManager.swift` | +| 润色 | `OSGKeyboardShared/Services/PolishingService.swift` | +| 文本插入 | `OSGKeyboardMac/MacTextInsertionService.swift` | + +--- + +## 17. 修订记录 + +| 日期 | 说明 | +|------|------| +| 2026-03-31 | 初版：基于 OSG 代码审计 + typeless-alternative 竞品源码 + 计划评审（ModelScope、hotwordMode、回退策略） | diff --git a/project.yml b/project.yml index d2b7d39..70b8ead 100644 --- a/project.yml +++ b/project.yml @@ -50,8 +50,8 @@ settings: ASSETCATALOG_COMPILER_GENERATE_SWIFT_ASSET_SYMBOL_EXTENSIONS: YES STRING_CATALOG_GENERATE_SYMBOLS: YES CLANG_CXX_LANGUAGE_STANDARD: c++17 - MARKETING_VERSION: "0.5.0" - CURRENT_PROJECT_VERSION: "18" + MARKETING_VERSION: "0.5.2" + CURRENT_PROJECT_VERSION: "19" # 签名配置来自 Signing.local.xcconfig（gitignored，不会被覆盖） # 项目级签名 xcconfig，适用于所有 target @@ -327,6 +327,7 @@ targets: resources: - path: OSGKeyboardShared/Resources/CustomLanguageModel/v1/OSGKeyboardCLM.bin - path: OSGKeyboardShared/Resources/CustomLanguageModel/v1/compiled-manifest.json + - path: OSGKeyboardShared/Resources/LocalASR/local-asr-catalog.json info: path: OSGKeyboardShared/Info.plist settings: @@ -429,10 +430,18 @@ targets: buildPhase: resources - path: OSGKeyboardShared/zh-Hans.lproj/Shared.strings buildPhase: resources + - path: OSGKeyboard/Resources/CustomLanguageModel/v1/phrases.tsv + buildPhase: resources + - path: OSGKeyboardShared/Resources/LocalASR/local-asr-catalog.json + buildPhase: resources entitlements: path: OSGKeyboardMac/OSGKeyboardMac.entitlements properties: - com.apple.security.app-sandbox: true + # Sandbox disabled on purpose: this menu-bar dictation utility uses the + # Accessibility API to monitor a global hotkey and inject ⌘V into other + # apps — both are forbidden by App Sandbox. Ship via Developer ID + + # notarization (not Mac App Store / TestFlight, which force the sandbox). + com.apple.security.app-sandbox: false com.apple.security.device.audio-input: true com.apple.security.network.client: true com.apple.developer.ubiquity-kvstore-identifier: $(TeamIdentifierPrefix)com.osgkeyboard.ios @@ -455,6 +464,9 @@ targets: ITSAppUsesNonExemptEncryption: false settings: base: + PRODUCT_NAME: OSGKeyboard + INFOPLIST_KEY_CFBundleDisplayName: OSGKeyboard + INFOPLIST_KEY_CFBundleName: OSGKeyboard PRODUCT_BUNDLE_IDENTIFIER: com.osgkeyboard.mac MACOSX_DEPLOYMENT_TARGET: "15.0" ASSETCATALOG_COMPILER_APPICON_NAME: AppIcon @@ -462,6 +474,8 @@ targets: APPLICATION_EXTENSION_API_ONLY: NO CODE_SIGN_STYLE: Automatic DEVELOPMENT_TEAM: X329MZU23S + # Required for Developer ID distribution + notarization (outside App Store). + ENABLE_HARDENED_RUNTIME: YES dependencies: - sdk: Speech.framework - sdk: AVFoundation.framework