feat(macos): local ASR model manager, menu-bar polish, and release 0.5.2
Adds a bundled local ASR model catalog for the macOS app with one-click Sherpa Qwen3 / SenseVoice downloads (pause/resume, inline actions) and a shared model storage directory used by MLX Qwen3. Fixes the light-mode sidebar material and makes the menu-bar icon follow the system appearance with a refreshed status mark. Renames the built product to OSGKeyboard.app. Bumps version to 0.5.2 (build 19).
This commit is contained in:
@@ -7,20 +7,29 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [0.5.2] - 2026-07-09
|
||||
|
||||
### Added
|
||||
- **macOS local ASR models**: the desktop app ships a bundled model catalog with one-click download of Sherpa Qwen3 (hotwords) and SenseVoice models, plus a shared model storage directory; downloads show a circular progress ring with pause / resume, and each row has inline Download / Delete actions. / **macOS 本地 ASR 模型**:桌面 App 内置模型目录,可一键下载 Sherpa Qwen3(热词)与 SenseVoice 模型,并共用同一模型存储目录;下载显示带暂停 / 继续的环形进度,每行提供内联的下载 / 删除操作。
|
||||
- **Shared model directory for MLX**: Qwen3-ASR MLX now uses a fixed subfolder inside the shared model storage — drop converted weights into the folder opened by "Open folder"; no per-model directory picker. / **MLX 共用模型目录**:Qwen3-ASR MLX 改用共享模型存储中的固定子目录——把转换好的权重放入「打开目录」指向的文件夹即可,不再逐模型选目录。
|
||||
- **iCloud sync hardening**: per-field settings merge (`appSettings.v2`), per-device usage statistics (G-Counter), tombstoned dictionary/history merge, and a low-risk **Sync Now** action in Settings. / **iCloud 同步加固**:设置按字段合并(`appSettings.v2`)、统计按设备 G-Counter 累计、词库/历史带墓碑合并,并在设置中新增低风险的**立即同步**操作。
|
||||
|
||||
### Changed
|
||||
- **API key sync**: cloud provider API keys now replicate through **iCloud Keychain** when settings sync is on — never through iCloud KVS JSON. / **API 密钥同步**:开启设置同步后,云端服务商 API 密钥改由 **iCloud 钥匙串**复制,不再写入 iCloud KVS JSON。
|
||||
- **Speech history cap**: synced history limit is **300** entries (aligned with the sync payload). / **语音历史上限**:可同步历史上限为 **300** 条(与同步载荷一致)。
|
||||
- **macOS app name**: the built product is now `OSGKeyboard.app` (was `OSGKeyboardMac.app`); Dock, About, and Finder all read **OSGKeyboard**. / **macOS 应用名称**:编译产物改为 `OSGKeyboard.app`(原 `OSGKeyboardMac.app`);Dock、关于窗口与 Finder 均显示 **OSGKeyboard**。
|
||||
- **macOS local recognition label**: the Settings entry is now simply "Local Recognition" and no longer names a specific model. / **macOS 本地识别标签**:设置项改为「本地识别」,不再绑定具体模型名称。
|
||||
|
||||
### Fixed
|
||||
- **macOS menu-bar icon in light mode**: the status-bar icon now follows the *system* menu-bar appearance, so forcing the app into Light while the system is Dark no longer renders an unreadable dark icon; a refreshed status mark is used. / **macOS 菜单栏图标(浅色模式)**:状态栏图标改为跟随*系统*菜单栏外观,App 强制浅色而系统为深色时不再出现看不清的深色图标;并更新了状态栏图标。
|
||||
- **macOS light-mode sidebar**: restored the native translucent sidebar material so the light appearance matches system apps (e.g. System Settings, Notes) instead of a flat grey fill. / **macOS 浅色侧边栏**:恢复原生半透明侧栏材质,浅色外观与系统应用(如系统设置、备忘录)一致,不再是扁平灰底。
|
||||
- **Settings sync wiping API keys**: pulling a legacy settings blob without API key fields no longer deletes local Keychain entries. / **设置同步清空 API 密钥**:拉取不含 API 密钥字段的旧版设置包时,不再删除本地 Keychain 项。
|
||||
- **Cross-device settings conflicts**: changing different settings on two devices no longer lets one device's full blob overwrite the other's unrelated fields. / **跨设备设置冲突**:两台设备分别修改不同设置项时,不再因整包覆盖而冲掉对方未改动的字段。
|
||||
- **Usage statistics under-counting**: offline usage on multiple devices now sums correctly instead of taking per-field `max()`. / **使用统计少计**:多设备离线各自累计后合并为求和,不再对总量取 `max()`。
|
||||
- **Dictionary/history resurrection**: deletes and "clear all" on one device propagate via tombstones so older remote entries cannot come back. / **词库/历史复活**:单设备删除或清空会通过墓碑传播,远端旧条目无法复活。
|
||||
- **Flow false-ready mic state**: the keyboard mic now stays orange until the host app publishes a real ready contract (capture engine live + polling idle), not merely a fresh heartbeat; green tap-to-talk and jump-to-host behavior share the same `MicVoiceAvailability` gate, and orphaned `stopped` signals self-heal instead of hanging until timeout. / **Flow 伪就绪麦克风状态**:键盘麦克风在主 App 发布真实就绪合约(音频引擎在跑且轮询空闲)之前保持橙色,不再仅凭心跳误判;绿色「点按说话」与跳转主 App 共用同一 `MicVoiceAvailability` 闸门,孤立的 `stopped` 信号会自愈而不再长时间卡住。
|
||||
- **Flow mic stuck orange after ready**: a single stale cross-process heartbeat read no longer flips a healthy session into a sticky "session ended" error that forced the mic orange. The "session ended" hint now fires only when the (heartbeat-independent) session contract truly drops; a brief read jitter is smoothed by a ready grace window, and a lingering expired hint auto-recovers to green once the host is ready again. / **就绪后麦克风卡橙色**:单次跨进程心跳读数抖动不再把健康会话打成粘滞的「会话已结束」错误、强制麦克风变橙。「会话已结束」提示现仅在(不依赖心跳的)会话合约真正失效时触发;短暂读数抖动由就绪宽限期平滑,遗留的过期提示会在宿主重新就绪后自动恢复为绿色。
|
||||
- **Orphaned Live Activity after force-quit**: force-quitting the app no longer leaves a stale OSGKeyboard status stuck on the Lock Screen / Dynamic Island. The `staleDate` is now ~45s (refreshed by the heartbeat while the session is alive) so the system reclaims a dead session's island on its own, and every app foreground now sweeps leftover Live Activities *before* trying to (re)start a session — so even a start that later fails (e.g. mic proof timeout) still clears the zombie island. / **强杀后遗留 Live Activity**:强制退出 App 不再在锁屏 / 灵动岛留下无法消失的 OSGKeyboard 状态。`staleDate` 缩短为约 45 秒(会话存活期间由心跳持续刷新),系统会自动回收已死会话的灵动岛;且每次 App 回到前台都会**先**清扫遗留的 Live Activity 再尝试(重新)启动会话——即便本次启动随后失败(如麦克风就绪超时),也不会留下僵尸灵动岛。
|
||||
|
||||
## [0.5.0] - 2026-07-07
|
||||
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
{
|
||||
"images" : [
|
||||
{
|
||||
"filename" : "OSGStatusMark.png",
|
||||
"idiom" : "universal",
|
||||
"scale" : "1x"
|
||||
},
|
||||
{
|
||||
"idiom" : "universal",
|
||||
"scale" : "2x"
|
||||
},
|
||||
{
|
||||
"idiom" : "universal",
|
||||
"scale" : "3x"
|
||||
}
|
||||
],
|
||||
"info" : {
|
||||
"author" : "xcode",
|
||||
"version" : 1
|
||||
},
|
||||
"properties" : {
|
||||
"preserves-vector-representation" : true,
|
||||
"template-rendering-intent" : "template"
|
||||
}
|
||||
}
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 3.9 KiB |
@@ -10,12 +10,16 @@ import OSGKeyboardShared
|
||||
|
||||
enum FlowLiveActivityController {
|
||||
nonisolated(unsafe) private static var currentActivity: Activity<FlowActivityAttributes>?
|
||||
/// Last phase pushed to the Live Activity so `keepAlive()` can refresh the
|
||||
/// `staleDate` without changing what the user sees.
|
||||
nonisolated(unsafe) private static var currentPhase: FlowActivityAttributes.ContentState.Phase = .idle
|
||||
|
||||
/// If the host app is force-quit its `endSession()` never runs, orphaning
|
||||
/// the Live Activity. A `staleDate` lets the system grey it out and become
|
||||
/// willing to reclaim it without our process — refreshed on every update
|
||||
/// so a genuinely active, in-use session never looks stale.
|
||||
private static let staleWindow: TimeInterval = 60 * 60
|
||||
/// the Live Activity. A short `staleDate` lets the system grey it out and
|
||||
/// reclaim it on its own within ~45s of the process dying. While the host is
|
||||
/// alive the heartbeat calls `keepAlive()` well inside this window, so a
|
||||
/// genuinely active session never looks stale.
|
||||
private static let staleWindow: TimeInterval = 45
|
||||
|
||||
private static func freshContent(
|
||||
phase: FlowActivityAttributes.ContentState.Phase
|
||||
@@ -41,6 +45,7 @@ enum FlowLiveActivityController {
|
||||
}
|
||||
|
||||
do {
|
||||
currentPhase = .idle
|
||||
currentActivity = try Activity.request(
|
||||
attributes: FlowActivityAttributes(),
|
||||
content: freshContent(phase: .idle),
|
||||
@@ -54,14 +59,28 @@ enum FlowLiveActivityController {
|
||||
|
||||
static func update(phase: FlowActivityAttributes.ContentState.Phase) {
|
||||
guard let activity = currentActivity else { return }
|
||||
currentPhase = phase
|
||||
let content = freshContent(phase: phase)
|
||||
Task {
|
||||
await activity.update(content)
|
||||
}
|
||||
}
|
||||
|
||||
/// Push a fresh `staleDate` without changing the visible phase. The host
|
||||
/// heartbeat calls this well inside `staleWindow` so an in-use session
|
||||
/// never looks stale; once the process dies the refreshes stop and the
|
||||
/// system reclaims the orphaned Live Activity on its own.
|
||||
static func keepAlive() {
|
||||
guard let activity = currentActivity else { return }
|
||||
let content = freshContent(phase: currentPhase)
|
||||
Task {
|
||||
await activity.update(content)
|
||||
}
|
||||
}
|
||||
|
||||
/// Dismiss the island presentation when the Flow session ends.
|
||||
static func endSession() {
|
||||
currentPhase = .idle
|
||||
guard let activity = currentActivity else {
|
||||
endStaleActivities()
|
||||
return
|
||||
@@ -74,6 +93,18 @@ enum FlowLiveActivityController {
|
||||
}
|
||||
}
|
||||
|
||||
/// Clear Live Activities orphaned by a previous (force-quit) host process.
|
||||
///
|
||||
/// Safe to call on every app foreground: when this process already owns a
|
||||
/// Live Activity (`currentActivity != nil`) we leave it alone so a healthy
|
||||
/// running session is never torn down; we only sweep leftovers that belong
|
||||
/// to a dead process. Call this *before* attempting to (re)start a session
|
||||
/// so a failed start (e.g. mic timeout) still clears the stale island.
|
||||
static func clearOrphanedActivities() {
|
||||
guard currentActivity == nil else { return }
|
||||
endStaleActivities()
|
||||
}
|
||||
|
||||
/// Host relaunch can leave orphan activities; clear them before starting anew.
|
||||
private static func endStaleActivities() {
|
||||
let staleActivities = Activity<FlowActivityAttributes>.activities
|
||||
|
||||
@@ -25,9 +25,7 @@ final class FlowSessionManager: ObservableObject {
|
||||
private let capture = FlowContinuousCapture()
|
||||
private let store = AppGroupStore()
|
||||
/// Cloud-engine polish; local engine runs through built-in DeepSeek polish.
|
||||
private var polisher: PolishingService {
|
||||
PolishingService()
|
||||
}
|
||||
private let polisher = PolishingService()
|
||||
/// Cached ASR instance. v0.2.0: the only on-device backend is iOS
|
||||
/// `SpeechAnalyzer`, which has no warm-up step — we can hand the
|
||||
/// factory-built service straight back without going through the
|
||||
@@ -49,17 +47,26 @@ final class FlowSessionManager: ObservableObject {
|
||||
private var expiryTask: Task<Void, Never>?
|
||||
private var levelTask: Task<Void, Never>?
|
||||
private var startTask: Task<Void, Never>?
|
||||
private var commandObserver: FlowSessionDarwinObserver?
|
||||
/// Last recording state the poll loop observed — logs only on transition.
|
||||
private var lastObservedRecordingState: FlowSessionKeys.RecordingState = .idle
|
||||
private var activeSessionId: UUID?
|
||||
private var currentUtteranceId: UUID?
|
||||
private var currentCommandSeq: Int64 = 0
|
||||
private var lastHandledCommandSeq: Int64 = 0
|
||||
private var isUtteranceRecording = false
|
||||
/// True from `stopped` until the result/error is written back to App Group.
|
||||
private var isUtteranceProcessing = false
|
||||
private var finalizeTask: Task<Void, Never>?
|
||||
private var asrTask: Task<Void, Never>?
|
||||
private var utteranceSafetyTask: Task<Void, Never>?
|
||||
private var chunkedPipeline: ChunkedUtterancePipeline?
|
||||
private var currentPartial = ""
|
||||
private var lastFinal = ""
|
||||
private var chunkWarnings: [String] = []
|
||||
private var lastReadyTraceSignature = ""
|
||||
private var lastCommandFingerprint = ""
|
||||
private var lastIgnoredCommandSignature = ""
|
||||
/// Wall-clock span of the current mic-open utterance (excludes LLM polish).
|
||||
private var utteranceRecordingStartedAt: Date?
|
||||
/// True while the host app scene is `.active` — drives foreground renewal.
|
||||
@@ -67,7 +74,11 @@ final class FlowSessionManager: ObservableObject {
|
||||
private var backgroundTaskID: UIBackgroundTaskIdentifier = .invalid
|
||||
/// True while handling a keyboard-initiated `startflow` cold start.
|
||||
private var isColdStartHandoff = false
|
||||
private static let coldStartAudioProofTimeout: TimeInterval = 2.5
|
||||
private var coldStartRecoveryTask: Task<Void, Never>?
|
||||
/// Initial proof window — cold mic sessions often need >2.5s after app switch.
|
||||
private static let coldStartAudioProofTimeout: TimeInterval = 6
|
||||
/// Extra window after the first timeout while the overlay shows a failure hint.
|
||||
private static let coldStartRecoveryProofTimeout: TimeInterval = 12
|
||||
|
||||
init() {
|
||||
// Sessions are (re)started explicitly on app foreground via
|
||||
@@ -83,6 +94,10 @@ final class FlowSessionManager: ObservableObject {
|
||||
|
||||
/// Starts a Flow session: permissions → continuous capture → App Group active.
|
||||
func startSession(duration: TimeInterval? = nil, coldStart: Bool = false) {
|
||||
traceState(
|
||||
"startSession.request",
|
||||
extra: "coldStart=\(coldStart) duration=\(Int(duration ?? FlowSessionPolicy.sessionDuration()))"
|
||||
)
|
||||
guard AppGroup.isAvailable else {
|
||||
debug("cannot start flow session: App Group unavailable")
|
||||
return
|
||||
@@ -106,7 +121,10 @@ final class FlowSessionManager: ObservableObject {
|
||||
return
|
||||
}
|
||||
|
||||
guard !isStarting else { return }
|
||||
guard !isStarting else {
|
||||
traceState("startSession.ignored", extra: "reason=alreadyStarting")
|
||||
return
|
||||
}
|
||||
|
||||
startTask?.cancel()
|
||||
startTask = Task { @MainActor [weak self] in
|
||||
@@ -148,6 +166,14 @@ final class FlowSessionManager: ObservableObject {
|
||||
/// behind (its `endSession()` could not run at kill time).
|
||||
func activateOnForeground() {
|
||||
guard AppGroup.isAvailable else { return }
|
||||
// Sweep any Live Activity a previous (force-quit) process left behind
|
||||
// *before* we try to (re)start a session. Doing it here — rather than
|
||||
// only inside `startSession()`'s success path — means a start that
|
||||
// later fails (e.g. mic proof timeout) still clears the stale island
|
||||
// instead of leaving a zombie on the lock screen / Dynamic Island.
|
||||
// No-op when this process already owns a healthy Live Activity.
|
||||
FlowLiveActivityController.clearOrphanedActivities()
|
||||
|
||||
guard AppPermissions.flowRequirementsMet else {
|
||||
sessionWarning = permissionWarningMessage()
|
||||
FlowSessionBridge.setHostReady(false)
|
||||
@@ -161,7 +187,8 @@ final class FlowSessionManager: ObservableObject {
|
||||
}
|
||||
|
||||
func dismissColdStartOverlay() {
|
||||
guard coldStartContext?.state != .preparing else { return }
|
||||
coldStartRecoveryTask?.cancel()
|
||||
coldStartRecoveryTask = nil
|
||||
if isActive {
|
||||
refreshHostReady()
|
||||
}
|
||||
@@ -189,8 +216,11 @@ final class FlowSessionManager: ObservableObject {
|
||||
|
||||
coldStartContext = nil
|
||||
isColdStartHandoff = false
|
||||
coldStartRecoveryTask?.cancel()
|
||||
coldStartRecoveryTask = nil
|
||||
startTask?.cancel()
|
||||
startTask = nil
|
||||
commandObserver = nil
|
||||
pollingTask?.cancel()
|
||||
pollingTask = nil
|
||||
heartbeatTask?.cancel()
|
||||
@@ -201,6 +231,8 @@ final class FlowSessionManager: ObservableObject {
|
||||
levelTask = nil
|
||||
finalizeTask?.cancel()
|
||||
finalizeTask = nil
|
||||
utteranceSafetyTask?.cancel()
|
||||
utteranceSafetyTask = nil
|
||||
|
||||
if isUtteranceRecording || isUtteranceProcessing {
|
||||
capture.cancelUtterance()
|
||||
@@ -210,6 +242,10 @@ final class FlowSessionManager: ObservableObject {
|
||||
}
|
||||
asrTask = nil
|
||||
chunkedPipeline = nil
|
||||
activeSessionId = nil
|
||||
currentUtteranceId = nil
|
||||
currentCommandSeq = 0
|
||||
lastHandledCommandSeq = 0
|
||||
isUtteranceRecording = false
|
||||
isUtteranceProcessing = false
|
||||
|
||||
@@ -305,9 +341,11 @@ final class FlowSessionManager: ObservableObject {
|
||||
guard isActive else { return }
|
||||
|
||||
if capture.running {
|
||||
capture.reassertIfRunning()
|
||||
if capture.engineHasRecentAudio() {
|
||||
let reasserted = capture.reassertIfRunning()
|
||||
if reasserted, capture.engineHasRecentAudio() {
|
||||
sessionWarning = nil
|
||||
} else if !reasserted {
|
||||
sessionWarning = AppL10n.string("flow.error.audioUnavailable")
|
||||
}
|
||||
refreshHostReady()
|
||||
return
|
||||
@@ -329,22 +367,94 @@ final class FlowSessionManager: ObservableObject {
|
||||
/// Publish whether the keyboard can start a new utterance without jumping to the host app.
|
||||
private func refreshHostReady() {
|
||||
guard isActive else {
|
||||
FlowSessionBridge.setHostReady(false)
|
||||
FlowSessionBridge.writeReadySnapshot(
|
||||
FlowReadySnapshot(
|
||||
sessionId: activeSessionId,
|
||||
ready: false,
|
||||
reason: .noSession,
|
||||
engineMode: store.engineMode,
|
||||
localeId: store.localeId,
|
||||
sessionExpiresAt: FlowSessionBridge.sessionExpiresAt()
|
||||
)
|
||||
)
|
||||
return
|
||||
}
|
||||
|
||||
let pollingAlive = pollingTask != nil && pollingTask?.isCancelled != true
|
||||
// Steady-state ready uses structural engine liveness. The stricter
|
||||
// "recent audio frame" proof is reserved for cold-start handoff only
|
||||
// (`waitForAudioProof`) so brief UI-driven session hiccups do not
|
||||
// drop the keyboard back to orange while the host app is foreground.
|
||||
let hasRecentAudio = capture.engineHasRecentAudio(maxAge: 2)
|
||||
let canAcceptUtterance = capture.engineIsLive
|
||||
&& pollingAlive
|
||||
&& hasRecentAudio
|
||||
&& !isUtteranceRecording
|
||||
&& !isUtteranceProcessing
|
||||
&& sessionWarning == nil
|
||||
|
||||
FlowSessionBridge.setHostReady(canAcceptUtterance)
|
||||
let reason: FlowReadySnapshot.Reason
|
||||
if canAcceptUtterance {
|
||||
reason = .ready
|
||||
} else if sessionWarning != nil {
|
||||
reason = .error
|
||||
} else if isUtteranceRecording {
|
||||
reason = .recording
|
||||
} else if isUtteranceProcessing {
|
||||
reason = .processing
|
||||
} else if !capture.engineIsLive {
|
||||
reason = .audioEngineNotLive
|
||||
} else if !hasRecentAudio {
|
||||
reason = .waitingForAudioProof
|
||||
} else {
|
||||
reason = .starting
|
||||
}
|
||||
|
||||
let now = Date().timeIntervalSince1970
|
||||
FlowSessionBridge.writeReadySnapshot(
|
||||
FlowReadySnapshot(
|
||||
sessionId: activeSessionId,
|
||||
ready: canAcceptUtterance,
|
||||
reason: reason,
|
||||
heartbeatAt: now,
|
||||
readyAt: canAcceptUtterance ? now : nil,
|
||||
audioProofAt: hasRecentAudio ? now : nil,
|
||||
engineMode: store.engineMode,
|
||||
localeId: store.localeId,
|
||||
busyUtteranceId: isUtteranceRecording || isUtteranceProcessing ? currentUtteranceId : nil,
|
||||
sessionExpiresAt: FlowSessionBridge.sessionExpiresAt()
|
||||
)
|
||||
)
|
||||
let signature = [
|
||||
canAcceptUtterance ? "ready=1" : "ready=0",
|
||||
"reason=\(reason.rawValue)",
|
||||
capture.engineIsLive ? "engine=live" : "engine=dead",
|
||||
hasRecentAudio ? "audio=fresh" : "audio=stale",
|
||||
isUtteranceRecording ? "recording=1" : "recording=0",
|
||||
isUtteranceProcessing ? "processing=1" : "processing=0",
|
||||
sessionWarning == nil ? "warning=0" : "warning=1"
|
||||
].joined(separator: "|")
|
||||
if signature != lastReadyTraceSignature {
|
||||
lastReadyTraceSignature = signature
|
||||
traceState("hostReady.update", extra: signature)
|
||||
}
|
||||
reconcileColdStartOverlayIfRecovered()
|
||||
}
|
||||
|
||||
/// When the host contract turns green while the cold-start overlay still
|
||||
/// shows a stale preparing/failed snapshot, heal automatically.
|
||||
private func reconcileColdStartOverlayIfRecovered() {
|
||||
guard isColdStartHandoff, isActive else { return }
|
||||
guard FlowSessionBridge.isHostReady() else { return }
|
||||
guard let context = coldStartContext else { return }
|
||||
|
||||
switch context.state {
|
||||
case .preparing:
|
||||
presentColdStartReadyOverlay()
|
||||
case .failed:
|
||||
sessionWarning = nil
|
||||
coldStartRecoveryTask?.cancel()
|
||||
coldStartRecoveryTask = nil
|
||||
dismissColdStartOverlay()
|
||||
case .ready:
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
/// Home preview field gained focus while this app is the Flow host.
|
||||
@@ -378,12 +488,14 @@ final class FlowSessionManager: ObservableObject {
|
||||
// MARK: - Session start
|
||||
|
||||
private func startSessionAsync(duration: TimeInterval?) async {
|
||||
traceState("startSessionAsync.begin")
|
||||
isStarting = true
|
||||
sessionWarning = nil
|
||||
defer { isStarting = false }
|
||||
|
||||
guard AppPermissions.flowRequirementsMet else {
|
||||
sessionWarning = permissionWarningMessage()
|
||||
traceState("startSessionAsync.blocked", extra: "reason=permissions")
|
||||
FlowSessionBridge.setHostReady(false)
|
||||
if isColdStartHandoff {
|
||||
showColdStartPermissionFailure()
|
||||
@@ -397,6 +509,7 @@ final class FlowSessionManager: ObservableObject {
|
||||
} catch {
|
||||
let message = (error as? LocalizedError)?.errorDescription ?? error.localizedDescription
|
||||
sessionWarning = message
|
||||
traceState("startSessionAsync.failed", extra: "reason=captureStart error=\(message)")
|
||||
FlowSessionBridge.setHostReady(false)
|
||||
if isColdStartHandoff {
|
||||
showColdStartAudioFailure(message: message)
|
||||
@@ -408,23 +521,36 @@ final class FlowSessionManager: ObservableObject {
|
||||
guard await waitForAudioProof() else {
|
||||
let message = AppL10n.string("flow.coldStart.error.audioTimeout")
|
||||
sessionWarning = message
|
||||
capture.stop()
|
||||
traceState("startSessionAsync.failed", extra: "reason=audioProofTimeout")
|
||||
FlowSessionBridge.setHostReady(false)
|
||||
if isColdStartHandoff {
|
||||
showColdStartAudioFailure(message: message)
|
||||
scheduleColdStartRecovery(duration: duration)
|
||||
} else {
|
||||
capture.stop()
|
||||
}
|
||||
debug("continuous capture did not produce audio frames before timeout")
|
||||
return
|
||||
}
|
||||
|
||||
activateFlowSessionAfterAudioProof(duration: duration)
|
||||
traceState("startSessionAsync.ready")
|
||||
debug("Flow session started (\(Int(duration ?? FlowSessionPolicy.sessionDuration()))s inactivity window), continuous capture running")
|
||||
}
|
||||
|
||||
private func activateFlowSessionAfterAudioProof(duration: TimeInterval?) {
|
||||
let resolvedDuration = duration ?? FlowSessionPolicy.sessionDuration()
|
||||
FlowSessionBridge.markSessionActive(duration: resolvedDuration)
|
||||
let sessionId = activeSessionId ?? UUID()
|
||||
activeSessionId = sessionId
|
||||
lastHandledCommandSeq = 0
|
||||
FlowSessionBridge.markSessionActive(duration: resolvedDuration, sessionId: sessionId)
|
||||
FlowSessionDarwin.postSessionChanged()
|
||||
isActive = true
|
||||
ScreenWakeLock.acquire()
|
||||
sessionExpiresAt = Date().addingTimeInterval(resolvedDuration)
|
||||
|
||||
startHeartbeat()
|
||||
startCommandObserver()
|
||||
startPolling()
|
||||
startLevelPublishing()
|
||||
scheduleExpiry(after: resolvedDuration)
|
||||
@@ -434,7 +560,7 @@ final class FlowSessionManager: ObservableObject {
|
||||
FlowLiveActivityController.startSession()
|
||||
|
||||
refreshHostReady()
|
||||
debug("Flow session started (\(Int(resolvedDuration))s inactivity window), continuous capture running")
|
||||
traceState("activateFlowSessionAfterAudioProof.done")
|
||||
}
|
||||
|
||||
private func prepareExistingSessionForColdStartReturn() async {
|
||||
@@ -445,6 +571,7 @@ final class FlowSessionManager: ObservableObject {
|
||||
sessionWarning = message
|
||||
FlowSessionBridge.setHostReady(false)
|
||||
showColdStartAudioFailure(message: message)
|
||||
scheduleColdStartRecovery(duration: nil)
|
||||
debug("existing session failed cold-start audio proof")
|
||||
return
|
||||
}
|
||||
@@ -466,25 +593,54 @@ final class FlowSessionManager: ObservableObject {
|
||||
let message = AppL10n.string("flow.coldStart.error.audioTimeout")
|
||||
sessionWarning = message
|
||||
showColdStartAudioFailure(message: message)
|
||||
scheduleColdStartRecovery(duration: nil)
|
||||
debug("cold-start blocked: host ready contract not published")
|
||||
return
|
||||
}
|
||||
|
||||
let hostEntry = HostReturnService.pendingHostEntry()
|
||||
let skipSwitch = FlowSessionPolicy.skipAppSwitch()
|
||||
coldStartContext = FlowColdStartContext(hostEntry: hostEntry, state: .ready)
|
||||
presentColdStartReadyOverlay()
|
||||
}
|
||||
|
||||
if skipSwitch, hostEntry != nil {
|
||||
Task { @MainActor [weak self] in
|
||||
try? await Task.sleep(nanoseconds: 450_000_000)
|
||||
guard let self, self.coldStartContext?.state == .ready else { return }
|
||||
if HostReturnService.openPendingHostIfPossible() {
|
||||
self.dismissColdStartOverlay()
|
||||
}
|
||||
private func presentColdStartReadyOverlay() {
|
||||
let hostEntry = HostReturnService.pendingHostEntry()
|
||||
coldStartContext = FlowColdStartContext(hostEntry: hostEntry, state: .ready)
|
||||
scheduleAutoReturnToHostIfNeeded(hostEntry: hostEntry)
|
||||
}
|
||||
|
||||
private func scheduleAutoReturnToHostIfNeeded(hostEntry: HostAppEntry?) {
|
||||
let skipSwitch = FlowSessionPolicy.skipAppSwitch()
|
||||
guard skipSwitch, hostEntry != nil else { return }
|
||||
Task { @MainActor [weak self] in
|
||||
try? await Task.sleep(nanoseconds: 450_000_000)
|
||||
guard let self, self.coldStartContext?.state == .ready else { return }
|
||||
if HostReturnService.openPendingHostIfPossible() {
|
||||
self.dismissColdStartOverlay()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Keeps proving mic readiness after the first timeout instead of tearing
|
||||
/// capture down — many handoffs become ready a few seconds later.
|
||||
private func scheduleColdStartRecovery(duration: TimeInterval?) {
|
||||
coldStartRecoveryTask?.cancel()
|
||||
coldStartRecoveryTask = Task { @MainActor [weak self] in
|
||||
guard let self else { return }
|
||||
let recovered = await self.capture.awaitAudioFlowing(
|
||||
timeout: Self.coldStartRecoveryProofTimeout
|
||||
)
|
||||
guard !Task.isCancelled else { return }
|
||||
guard self.isColdStartHandoff else { return }
|
||||
guard recovered else { return }
|
||||
|
||||
self.sessionWarning = nil
|
||||
self.traceState("coldStartRecovery.recovered")
|
||||
if !self.isActive {
|
||||
self.activateFlowSessionAfterAudioProof(duration: duration)
|
||||
}
|
||||
self.refreshHostReady()
|
||||
}
|
||||
}
|
||||
|
||||
private func showColdStartPreparing() {
|
||||
coldStartContext = FlowColdStartContext(
|
||||
hostEntry: HostReturnService.pendingHostEntry(),
|
||||
@@ -546,6 +702,14 @@ final class FlowSessionManager: ObservableObject {
|
||||
|
||||
// MARK: - Polling
|
||||
|
||||
private func startCommandObserver() {
|
||||
commandObserver = FlowSessionDarwinObserver(
|
||||
notificationName: FlowSessionDarwin.commandNotificationName
|
||||
) { [weak self] in
|
||||
self?.handleKeyboardSignal()
|
||||
}
|
||||
}
|
||||
|
||||
private func startPolling() {
|
||||
pollingTask?.cancel()
|
||||
lastObservedRecordingState = FlowSessionBridge.recordingState()
|
||||
@@ -556,46 +720,122 @@ final class FlowSessionManager: ObservableObject {
|
||||
pollingTask = Task { @MainActor [weak self] in
|
||||
while !Task.isCancelled {
|
||||
self?.handleKeyboardSignal()
|
||||
try? await Task.sleep(nanoseconds: 50_000_000)
|
||||
try? await Task.sleep(nanoseconds: 500_000_000)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private func handleKeyboardSignal() {
|
||||
let signal = FlowSessionBridge.recordingState()
|
||||
if signal != lastObservedRecordingState {
|
||||
// The single most important cross-process signal: proves whether the
|
||||
// host actually SEES the keyboard's recording state writes.
|
||||
FlowDiagnostics.log(
|
||||
"poll observed recordingState \(lastObservedRecordingState.rawValue) → \(signal.rawValue) " +
|
||||
"[rec=\(isUtteranceRecording) proc=\(isUtteranceProcessing) fg=\(isAppForeground)]"
|
||||
)
|
||||
lastObservedRecordingState = signal
|
||||
guard let command = FlowSessionBridge.latestCommand() else {
|
||||
lastCommandFingerprint = ""
|
||||
return
|
||||
}
|
||||
switch signal {
|
||||
case .recording:
|
||||
let fingerprint = "\(command.sessionId.uuidString)|\(command.utteranceId.uuidString)|\(command.action.rawValue)|\(command.commandSeq)"
|
||||
guard fingerprint != lastCommandFingerprint else { return }
|
||||
lastCommandFingerprint = fingerprint
|
||||
handleFlowCommand(command)
|
||||
}
|
||||
|
||||
private func handleFlowCommand(_ command: FlowCommand) {
|
||||
guard let activeSessionId, command.sessionId == activeSessionId else {
|
||||
traceIgnoredCommand(
|
||||
reason: "staleSession",
|
||||
command: command,
|
||||
detail: "commandSession=\(command.sessionId)"
|
||||
)
|
||||
return
|
||||
}
|
||||
guard command.commandSeq > lastHandledCommandSeq else {
|
||||
traceIgnoredCommand(
|
||||
reason: "seqNotIncreasing",
|
||||
command: command,
|
||||
detail: "last=\(lastHandledCommandSeq)"
|
||||
)
|
||||
return
|
||||
}
|
||||
lastHandledCommandSeq = command.commandSeq
|
||||
lastIgnoredCommandSignature = ""
|
||||
|
||||
FlowDiagnostics.log(
|
||||
"command \(command.action.rawValue) seq=\(command.commandSeq) utterance=\(command.utteranceId)"
|
||||
)
|
||||
|
||||
switch command.action {
|
||||
case .startRecording:
|
||||
guard !isUtteranceRecording, !isUtteranceProcessing else { return }
|
||||
beginUtterance()
|
||||
case .stopped:
|
||||
beginUtterance(utteranceId: command.utteranceId, commandSeq: command.commandSeq)
|
||||
case .stopRecording:
|
||||
guard currentUtteranceId == command.utteranceId else { return }
|
||||
if isUtteranceRecording {
|
||||
endUtterance()
|
||||
} else if !isUtteranceProcessing {
|
||||
FlowSessionBridge.setRecordingState(.idle)
|
||||
FlowSessionBridge.storeTranscriptionError(
|
||||
storeCurrentError(
|
||||
AppL10n.string("flow.error.recognitionInterrupted"),
|
||||
kind: .recognitionInterrupted
|
||||
)
|
||||
debug("stopped without active utterance — notified keyboard")
|
||||
debug("stop command without active utterance — notified keyboard")
|
||||
}
|
||||
case .aborted:
|
||||
case .abort:
|
||||
guard currentUtteranceId == command.utteranceId else { return }
|
||||
abortUtterance()
|
||||
case .idle, .processing:
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
private func beginUtterance() {
|
||||
guard capture.engineIsLive else {
|
||||
private func storeCurrentPartial(_ text: String) {
|
||||
guard let activeSessionId, let currentUtteranceId else { return }
|
||||
let trimmed = text.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||
guard !trimmed.isEmpty else { return }
|
||||
FlowSessionBridge.writeResult(
|
||||
FlowResult(
|
||||
sessionId: activeSessionId,
|
||||
utteranceId: currentUtteranceId,
|
||||
commandSeq: currentCommandSeq,
|
||||
status: .partial,
|
||||
text: trimmed
|
||||
)
|
||||
)
|
||||
}
|
||||
|
||||
private func storeCurrentFinal(_ text: String, warning: String? = nil) {
|
||||
let trimmed = text.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||
guard !trimmed.isEmpty else {
|
||||
storeCurrentError(AppL10n.string("flow.error.noSpeech"), kind: .noSpeech)
|
||||
return
|
||||
}
|
||||
guard let activeSessionId, let currentUtteranceId else { return }
|
||||
FlowSessionBridge.writeResult(
|
||||
FlowResult(
|
||||
sessionId: activeSessionId,
|
||||
utteranceId: currentUtteranceId,
|
||||
commandSeq: currentCommandSeq,
|
||||
status: .final,
|
||||
text: trimmed,
|
||||
warning: warning
|
||||
)
|
||||
)
|
||||
}
|
||||
|
||||
private func storeCurrentError(
|
||||
_ message: String,
|
||||
kind: FlowSessionKeys.TranscriptionErrorKind = .generic,
|
||||
status: FlowResult.Status = .error
|
||||
) {
|
||||
guard let activeSessionId, let currentUtteranceId else { return }
|
||||
FlowSessionBridge.writeResult(
|
||||
FlowResult(
|
||||
sessionId: activeSessionId,
|
||||
utteranceId: currentUtteranceId,
|
||||
commandSeq: currentCommandSeq,
|
||||
status: status,
|
||||
text: message,
|
||||
errorKind: kind
|
||||
)
|
||||
)
|
||||
}
|
||||
|
||||
private func beginUtterance(utteranceId: UUID? = nil, commandSeq: Int64 = 0) {
|
||||
guard capture.engineHasRecentAudio(maxAge: 2) else {
|
||||
traceState("beginUtterance.blocked", extra: "reason=audioNotRecent")
|
||||
failUtterance(
|
||||
message: AppL10n.string("flow.error.audioUnavailable"),
|
||||
kind: .audioUnavailable
|
||||
@@ -603,13 +843,25 @@ final class FlowSessionManager: ObservableObject {
|
||||
return
|
||||
}
|
||||
guard !isUtteranceProcessing else {
|
||||
traceState("beginUtterance.ignored", extra: "reason=processing")
|
||||
debug("beginUtterance ignored — previous utterance still processing")
|
||||
return
|
||||
}
|
||||
bindSessionASRIfNeeded()
|
||||
let expectedEngine = store.engineMode
|
||||
if sessionASREngineMode != expectedEngine {
|
||||
traceState(
|
||||
"beginUtterance.rebindMismatch",
|
||||
extra: "expectedEngine=\(expectedEngine) boundEngine=\(sessionASREngineMode ?? "nil")"
|
||||
)
|
||||
bindSessionASRIfNeeded(force: true)
|
||||
}
|
||||
|
||||
// Usually already warm from session start; refresh without blocking the mic gate.
|
||||
scheduleASRWarmup()
|
||||
|
||||
currentUtteranceId = utteranceId ?? UUID()
|
||||
currentCommandSeq = commandSeq
|
||||
currentPartial = ""
|
||||
lastFinal = ""
|
||||
chunkWarnings = []
|
||||
@@ -617,6 +869,7 @@ final class FlowSessionManager: ObservableObject {
|
||||
let localeId = store.localeId
|
||||
FlowSessionBridge.setTranscriptionLanguage(localeId)
|
||||
FlowSessionBridge.clearPendingTranscription()
|
||||
FlowSessionBridge.clearResult()
|
||||
|
||||
let locale = SpeechLocaleResolver.resolve(localeId)
|
||||
let stream = capture.beginUtterance()
|
||||
@@ -625,6 +878,7 @@ final class FlowSessionManager: ObservableObject {
|
||||
|
||||
isUtteranceRecording = true
|
||||
utteranceRecordingStartedAt = Date()
|
||||
startUtteranceSafetyTimer()
|
||||
refreshHostReady()
|
||||
FlowLiveActivityController.update(phase: .recording)
|
||||
FlowDiagnostics.log(
|
||||
@@ -637,8 +891,9 @@ final class FlowSessionManager: ObservableObject {
|
||||
asrTask = Task.detached(priority: .userInitiated) { [weak manager = self] in
|
||||
let outcome = await pipeline.transcribe(stream: stream) { partial in
|
||||
Task { @MainActor in
|
||||
manager?.currentPartial = partial
|
||||
FlowSessionBridge.storeTranscriptionPartial(partial)
|
||||
guard let manager else { return }
|
||||
manager.currentPartial = partial
|
||||
manager.storeCurrentPartial(partial)
|
||||
}
|
||||
}
|
||||
// Re-bind `manager` inside the `@MainActor` block so the
|
||||
@@ -672,14 +927,33 @@ final class FlowSessionManager: ObservableObject {
|
||||
debug("utterance recording started")
|
||||
}
|
||||
|
||||
private func startUtteranceSafetyTimer() {
|
||||
utteranceSafetyTask?.cancel()
|
||||
let utteranceId = currentUtteranceId
|
||||
utteranceSafetyTask = Task { @MainActor [weak self] in
|
||||
let timeout = FlowSessionKeys.maxUtteranceDuration + 10
|
||||
try? await Task.sleep(nanoseconds: UInt64(timeout * 1_000_000_000))
|
||||
guard let self, !Task.isCancelled else { return }
|
||||
guard self.isUtteranceRecording, self.currentUtteranceId == utteranceId else { return }
|
||||
self.storeCurrentError(
|
||||
AppL10n.string("flow.error.recognitionInterrupted"),
|
||||
kind: .recognitionInterrupted,
|
||||
status: .timeout
|
||||
)
|
||||
self.abortUtterance()
|
||||
self.debug("utterance safety timer aborted stale recording")
|
||||
}
|
||||
}
|
||||
|
||||
private func endUtterance() {
|
||||
guard isUtteranceRecording else { return }
|
||||
|
||||
// Close the mic gate first, then mark processing before dropping the
|
||||
// recording flag so the poll loop cannot start a second utterance.
|
||||
FlowSessionBridge.setRecordingState(.processing)
|
||||
isUtteranceRecording = false
|
||||
isUtteranceProcessing = true
|
||||
utteranceSafetyTask?.cancel()
|
||||
utteranceSafetyTask = nil
|
||||
refreshHostReady()
|
||||
FlowLiveActivityController.update(phase: .processing)
|
||||
|
||||
@@ -699,6 +973,8 @@ final class FlowSessionManager: ObservableObject {
|
||||
isUtteranceRecording = false
|
||||
isUtteranceProcessing = false
|
||||
utteranceRecordingStartedAt = nil
|
||||
utteranceSafetyTask?.cancel()
|
||||
utteranceSafetyTask = nil
|
||||
finalizeTask?.cancel()
|
||||
finalizeTask = nil
|
||||
asrTask?.cancel()
|
||||
@@ -709,8 +985,8 @@ final class FlowSessionManager: ObservableObject {
|
||||
currentPartial = ""
|
||||
lastFinal = ""
|
||||
chunkWarnings = []
|
||||
FlowSessionBridge.storeTranscriptionPartial("")
|
||||
FlowSessionBridge.setRecordingState(.idle)
|
||||
currentUtteranceId = nil
|
||||
currentCommandSeq = 0
|
||||
FlowLiveActivityController.update(phase: .idle)
|
||||
refreshHostReady()
|
||||
debug("utterance aborted")
|
||||
@@ -723,6 +999,8 @@ final class FlowSessionManager: ObservableObject {
|
||||
isUtteranceRecording = false
|
||||
isUtteranceProcessing = false
|
||||
utteranceRecordingStartedAt = nil
|
||||
utteranceSafetyTask?.cancel()
|
||||
utteranceSafetyTask = nil
|
||||
finalizeTask?.cancel()
|
||||
finalizeTask = nil
|
||||
asrTask?.cancel()
|
||||
@@ -733,9 +1011,9 @@ final class FlowSessionManager: ObservableObject {
|
||||
currentPartial = ""
|
||||
lastFinal = ""
|
||||
chunkWarnings = []
|
||||
FlowSessionBridge.storeTranscriptionPartial("")
|
||||
FlowSessionBridge.storeTranscriptionError(message, kind: kind)
|
||||
FlowSessionBridge.setRecordingState(.idle)
|
||||
storeCurrentError(message, kind: kind)
|
||||
currentUtteranceId = nil
|
||||
currentCommandSeq = 0
|
||||
FlowLiveActivityController.update(phase: .idle)
|
||||
refreshHostReady()
|
||||
debug("utterance failed: \(message)")
|
||||
@@ -747,28 +1025,36 @@ final class FlowSessionManager: ObservableObject {
|
||||
) {
|
||||
isUtteranceProcessing = false
|
||||
utteranceRecordingStartedAt = nil
|
||||
utteranceSafetyTask?.cancel()
|
||||
utteranceSafetyTask = nil
|
||||
finalizeTask?.cancel()
|
||||
finalizeTask = nil
|
||||
chunkedPipeline = nil
|
||||
currentPartial = ""
|
||||
lastFinal = ""
|
||||
chunkWarnings = []
|
||||
FlowSessionBridge.storeTranscriptionPartial("")
|
||||
FlowSessionBridge.storeTranscriptionError(message, kind: kind)
|
||||
FlowSessionBridge.setRecordingState(.idle)
|
||||
storeCurrentError(message, kind: kind)
|
||||
currentUtteranceId = nil
|
||||
currentCommandSeq = 0
|
||||
FlowLiveActivityController.update(phase: .idle)
|
||||
refreshHostReady()
|
||||
debug("utterance processing failed: \(message)")
|
||||
}
|
||||
|
||||
private func finalizeUtterance() async {
|
||||
let finalizeSessionId = activeSessionId
|
||||
let finalizeUtteranceId = currentUtteranceId
|
||||
let pipelineStarted = Date()
|
||||
defer {
|
||||
isUtteranceProcessing = false
|
||||
FlowSessionBridge.setRecordingState(.idle)
|
||||
FlowLiveActivityController.update(phase: .idle)
|
||||
touchSessionActivity()
|
||||
refreshHostReady()
|
||||
if activeSessionId == finalizeSessionId,
|
||||
currentUtteranceId == finalizeUtteranceId {
|
||||
isUtteranceProcessing = false
|
||||
FlowLiveActivityController.update(phase: .idle)
|
||||
touchSessionActivity()
|
||||
currentUtteranceId = nil
|
||||
currentCommandSeq = 0
|
||||
refreshHostReady()
|
||||
}
|
||||
}
|
||||
|
||||
let asrWait = asrWaitTimeout()
|
||||
@@ -783,9 +1069,10 @@ final class FlowSessionManager: ObservableObject {
|
||||
try? await Task.sleep(nanoseconds: 100_000_000)
|
||||
}
|
||||
|
||||
if lastFinal.isEmpty, let asrTask {
|
||||
FlowDiagnostics.log("ASR wait elapsed — awaiting asrTask completion")
|
||||
_ = await asrTask.value
|
||||
if lastFinal.isEmpty {
|
||||
FlowDiagnostics.log("ASR wait elapsed — cancelling ASR task and using best available transcript")
|
||||
asrTask?.cancel()
|
||||
Task { await chunkedPipeline?.cancel() }
|
||||
}
|
||||
|
||||
let asrElapsed = Date().timeIntervalSince(pipelineStarted)
|
||||
@@ -803,10 +1090,7 @@ final class FlowSessionManager: ObservableObject {
|
||||
(asrTask?.isCancelled == true) ? .recognitionInterrupted : .noSpeech
|
||||
FlowDiagnostics.log("finalize failed: empty transcript after \(String(format: "%.1f", asrElapsed))s")
|
||||
utteranceRecordingStartedAt = nil
|
||||
FlowSessionBridge.storeTranscriptionError(
|
||||
AppL10n.string(key),
|
||||
kind: kind
|
||||
)
|
||||
storeCurrentError(AppL10n.string(key), kind: kind)
|
||||
return
|
||||
}
|
||||
|
||||
@@ -832,7 +1116,7 @@ final class FlowSessionManager: ObservableObject {
|
||||
providerIdOverride: pipelineStore.polishProviderIdOverride
|
||||
)
|
||||
delivered = polished
|
||||
FlowSessionBridge.storeTranscriptionResult(polished, polishWarning: chunkNote)
|
||||
storeCurrentFinal(polished, warning: chunkNote)
|
||||
FlowDiagnostics.log(
|
||||
"polish done in \(String(format: "%.1f", Date().timeIntervalSince(polishStarted)))s " +
|
||||
"total=\(String(format: "%.1f", Date().timeIntervalSince(pipelineStarted)))s"
|
||||
@@ -854,7 +1138,7 @@ final class FlowSessionManager: ObservableObject {
|
||||
"\(error.localizedDescription)"
|
||||
)
|
||||
delivered = fallback.text
|
||||
FlowSessionBridge.storeTranscriptionResult(fallback.text, polishWarning: fallback.polishWarning)
|
||||
storeCurrentFinal(fallback.text, warning: fallback.polishWarning)
|
||||
}
|
||||
|
||||
SpeechHistoryStore.shared.recordUtterance(
|
||||
@@ -867,7 +1151,6 @@ final class FlowSessionManager: ObservableObject {
|
||||
currentPartial = ""
|
||||
lastFinal = ""
|
||||
chunkWarnings = []
|
||||
FlowSessionBridge.storeTranscriptionPartial("")
|
||||
chunkedPipeline = nil
|
||||
debug("utterance finalized length=\(text.count)")
|
||||
}
|
||||
@@ -964,6 +1247,12 @@ final class FlowSessionManager: ObservableObject {
|
||||
heartbeatTask?.cancel()
|
||||
FlowSessionBridge.writeHeartbeat()
|
||||
heartbeatTask = Task { @MainActor [weak self] in
|
||||
// Refresh the Live Activity `staleDate` every N heartbeat ticks
|
||||
// (1 Hz) — well inside `FlowLiveActivityController.staleWindow` so a
|
||||
// live session never looks stale, while a force-quit stops these
|
||||
// refreshes and lets the system reclaim the orphaned island.
|
||||
let liveActivityKeepAliveEveryTicks = 15
|
||||
var tick = 0
|
||||
while !Task.isCancelled {
|
||||
guard let self else { break }
|
||||
if self.isActive, !self.capture.engineIsLive {
|
||||
@@ -971,6 +1260,10 @@ final class FlowSessionManager: ObservableObject {
|
||||
}
|
||||
FlowSessionBridge.writeHeartbeat()
|
||||
self.refreshHostReady()
|
||||
tick += 1
|
||||
if tick % liveActivityKeepAliveEveryTicks == 0 {
|
||||
FlowLiveActivityController.keepAlive()
|
||||
}
|
||||
try? await Task.sleep(nanoseconds: 1_000_000_000)
|
||||
guard self.isActive else { break }
|
||||
}
|
||||
@@ -989,4 +1282,43 @@ final class FlowSessionManager: ObservableObject {
|
||||
private func debug(_ message: String) {
|
||||
FlowDiagnostics.log(message)
|
||||
}
|
||||
|
||||
private func traceIgnoredCommand(reason: String, command: FlowCommand, detail: String) {
|
||||
let signature = "\(reason)|\(command.action.rawValue)|\(command.commandSeq)|\(command.sessionId.uuidString)|\(command.utteranceId.uuidString)|\(detail)"
|
||||
guard signature != lastIgnoredCommandSignature else { return }
|
||||
lastIgnoredCommandSignature = signature
|
||||
traceState(
|
||||
"command.ignored",
|
||||
extra: "reason=\(reason) action=\(command.action.rawValue) seq=\(command.commandSeq) \(detail)"
|
||||
)
|
||||
}
|
||||
|
||||
private func traceState(_ event: String, extra: String? = nil) {
|
||||
let staleness = FlowSessionBridge.heartbeatStaleness().map { String(format: "%.1f", $0) } ?? "nil"
|
||||
let sessionId = activeSessionId?.uuidString ?? "nil"
|
||||
let utteranceId = currentUtteranceId?.uuidString ?? "nil"
|
||||
let summary = [
|
||||
"event=\(event)",
|
||||
"active=\(isActive)",
|
||||
"starting=\(isStarting)",
|
||||
"coldStart=\(isColdStartHandoff)",
|
||||
"sessionId=\(sessionId)",
|
||||
"utteranceId=\(utteranceId)",
|
||||
"cmdSeq=\(currentCommandSeq)",
|
||||
"lastCmd=\(lastHandledCommandSeq)",
|
||||
"recording=\(isUtteranceRecording)",
|
||||
"processing=\(isUtteranceProcessing)",
|
||||
"storeEngine=\(store.engineMode)",
|
||||
"boundEngine=\(sessionASREngineMode ?? "nil")",
|
||||
"engineLive=\(capture.engineIsLive)",
|
||||
"hostReady=\(FlowSessionBridge.isHostReady())",
|
||||
"sessionActive=\(FlowSessionBridge.isSessionActive())",
|
||||
"heartbeatStaleness=\(staleness)"
|
||||
].joined(separator: " ")
|
||||
if let extra, !extra.isEmpty {
|
||||
debug("[trace] \(summary) \(extra)")
|
||||
} else {
|
||||
debug("[trace] \(summary)")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -38,6 +38,17 @@ struct FlowColdStartOverlay: View {
|
||||
/// Fraction of the screen height the bottom gradient occupies.
|
||||
private let gradientHeightFraction: CGFloat = 0.50
|
||||
|
||||
/// Ready and failure states dismiss on blank tap; preparing stays
|
||||
/// informational only (no accidental dismiss while proving audio).
|
||||
private var allowsBlankTapDismiss: Bool {
|
||||
switch context.state {
|
||||
case .ready, .failed:
|
||||
return true
|
||||
case .preparing:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
var body: some View {
|
||||
GeometryReader { geo in
|
||||
ZStack(alignment: .bottom) {
|
||||
@@ -56,6 +67,16 @@ struct FlowColdStartOverlay: View {
|
||||
.frame(maxWidth: .infinity, alignment: .bottom)
|
||||
.allowsHitTesting(false)
|
||||
|
||||
if allowsBlankTapDismiss {
|
||||
// Captures taps on empty overlay space (dismiss) and blocks
|
||||
// pass-through to the host shell underneath. Action buttons in
|
||||
// `content` sit above this layer and remain tappable.
|
||||
Color.clear
|
||||
.contentShape(Rectangle())
|
||||
.onTapGesture(perform: onDismiss)
|
||||
.ignoresSafeArea()
|
||||
}
|
||||
|
||||
VStack(spacing: Spacing.lg) {
|
||||
content
|
||||
.padding(.horizontal, Spacing.xl)
|
||||
@@ -63,14 +84,6 @@ struct FlowColdStartOverlay: View {
|
||||
homeIndicator
|
||||
.padding(.bottom, max(geo.safeAreaInsets.bottom, Spacing.sm))
|
||||
}
|
||||
.allowsHitTesting(false)
|
||||
|
||||
if context.state == .ready {
|
||||
Color.clear
|
||||
.contentShape(Rectangle())
|
||||
.onTapGesture(perform: onDismiss)
|
||||
.ignoresSafeArea()
|
||||
}
|
||||
}
|
||||
.frame(maxWidth: .infinity, maxHeight: .infinity, alignment: .bottom)
|
||||
.ignoresSafeArea()
|
||||
|
||||
@@ -42,6 +42,10 @@ final class KeyboardFlowCoordinator {
|
||||
private static let hostReadyGrace: TimeInterval = 4
|
||||
private var flowSessionMonitorTask: Task<Void, Never>?
|
||||
private var isAwaitingFlowResult = false
|
||||
private var activeSessionId: UUID?
|
||||
private var currentUtteranceId: UUID?
|
||||
private var currentCommandSeq: Int64 = 0
|
||||
private var lastAvailabilityTraceSignature = ""
|
||||
|
||||
init(
|
||||
state: KeyboardState,
|
||||
@@ -133,7 +137,9 @@ final class KeyboardFlowCoordinator {
|
||||
|
||||
private func recomputeMicVoiceAvailability() {
|
||||
FlowSessionBridge.reloadFromDisk()
|
||||
let hostReady = FlowSessionBridge.isHostReady()
|
||||
let readySnapshot = FlowSessionBridge.readySnapshot()
|
||||
activeSessionId = readySnapshot?.sessionId ?? activeSessionId
|
||||
let hostReady = readySnapshot?.ready == true && FlowSessionBridge.isHostReady()
|
||||
let now = Date().timeIntervalSince1970
|
||||
if hostReady { lastHostReadyAt = now }
|
||||
// Grace window: the host was ready very recently, so treat a momentary
|
||||
@@ -145,7 +151,7 @@ final class KeyboardFlowCoordinator {
|
||||
let hostWarming = !hostReady
|
||||
&& FlowSessionBridge.isSessionActive()
|
||||
&& (FlowSessionBridge.isHostReachable() || isPendingFlowStart || withinReadyGrace)
|
||||
state.flowSessionActive = hostReady
|
||||
state.flowSessionActive = FlowSessionBridge.isSessionActive()
|
||||
state.micVoiceAvailability = MicVoiceAvailabilityResolver.resolve(
|
||||
phase: state.phase,
|
||||
micDisabled: state.micDisabled,
|
||||
@@ -154,6 +160,20 @@ final class KeyboardFlowCoordinator {
|
||||
hostReady: hostReady,
|
||||
isPreparingSession: isPendingFlowStart || hostWarming
|
||||
)
|
||||
let signature = [
|
||||
"phase=\(String(describing: state.phase))",
|
||||
"availability=\(String(describing: state.micVoiceAvailability))",
|
||||
hostReady ? "hostReady=1" : "hostReady=0",
|
||||
state.flowSessionActive ? "sessionActive=1" : "sessionActive=0",
|
||||
isPendingFlowStart ? "pending=1" : "pending=0",
|
||||
isFlowRecording ? "recording=1" : "recording=0",
|
||||
isAwaitingFlowResult ? "awaiting=1" : "awaiting=0",
|
||||
readySnapshot?.reason.rawValue ?? "snapshot=nil"
|
||||
].joined(separator: "|")
|
||||
if signature != lastAvailabilityTraceSignature {
|
||||
lastAvailabilityTraceSignature = signature
|
||||
traceState("availability.update", extra: signature)
|
||||
}
|
||||
}
|
||||
|
||||
/// Session is live but the ready contract has not landed yet — poll
|
||||
@@ -228,7 +248,8 @@ final class KeyboardFlowCoordinator {
|
||||
scheduleAutoClearError()
|
||||
recomputeMicVoiceAvailability()
|
||||
case .unavailable(.preparingSession):
|
||||
return
|
||||
detectAndStoreAppContext()
|
||||
beginFlowStart()
|
||||
case .unavailable(.hostNotReady):
|
||||
detectAndStoreAppContext()
|
||||
beginFlowStart()
|
||||
@@ -247,8 +268,8 @@ final class KeyboardFlowCoordinator {
|
||||
isFlowRecording = false
|
||||
stopUtteranceCountdown()
|
||||
ExtensionScreenWakeLock.release()
|
||||
FlowSessionBridge.setRecordingState(.stopped)
|
||||
debug("pressEnded wrote .stopped (readback=\(FlowSessionBridge.recordingState().rawValue))")
|
||||
writeCommand(.stopRecording)
|
||||
debug("pressEnded wrote stop command")
|
||||
state.phase = .processing
|
||||
state.lastTranscript = ExtL10n.string("keyboard.flow.transcribing")
|
||||
recomputeMicVoiceAvailability()
|
||||
@@ -256,7 +277,10 @@ final class KeyboardFlowCoordinator {
|
||||
}
|
||||
|
||||
func beginFlowStart() {
|
||||
guard !isPendingFlowStart else { return }
|
||||
guard !isPendingFlowStart else {
|
||||
traceState("beginFlowStart.ignored", extra: "reason=pendingAlreadyTrue")
|
||||
return
|
||||
}
|
||||
isPendingFlowStart = true
|
||||
isFlowRecording = false
|
||||
flowStartDeadline = Date().timeIntervalSince1970 + FlowWatchdog.startTimeout
|
||||
@@ -264,11 +288,11 @@ final class KeyboardFlowCoordinator {
|
||||
recomputeMicVoiceAvailability()
|
||||
openHostApp("startflow")
|
||||
startFlowStartWatchdog()
|
||||
debug("beginFlowStart")
|
||||
traceState("beginFlowStart.started")
|
||||
}
|
||||
|
||||
func handleHostAppOpenResult(path: String, success: Bool) {
|
||||
debug("openHostApp path=\(path) success=\(success)")
|
||||
traceState("openHostApp.result", extra: "path=\(path) success=\(success)")
|
||||
guard !success else { return }
|
||||
|
||||
// The open genuinely failed (iOS blocked it / no Full Access). Don't
|
||||
@@ -278,6 +302,7 @@ final class KeyboardFlowCoordinator {
|
||||
isPendingFlowStart = false
|
||||
flowStartDeadline = 0
|
||||
stopFlowWatchdog()
|
||||
traceState("openHostApp.failed", extra: "path=startflow cancelPending=1")
|
||||
showManualOpenHint(path: "startflow")
|
||||
recomputeMicVoiceAvailability()
|
||||
return
|
||||
@@ -290,9 +315,10 @@ final class KeyboardFlowCoordinator {
|
||||
guard !isAwaitingFlowResult else { return }
|
||||
if isFlowRecording || isPendingFlowStart {
|
||||
if isFlowRecording {
|
||||
FlowSessionBridge.setRecordingState(.aborted)
|
||||
writeCommand(.abort)
|
||||
ExtensionScreenWakeLock.release()
|
||||
}
|
||||
currentUtteranceId = nil
|
||||
isFlowRecording = false
|
||||
isPendingFlowStart = false
|
||||
stopUtteranceCountdown()
|
||||
@@ -304,17 +330,56 @@ final class KeyboardFlowCoordinator {
|
||||
|
||||
// MARK: - Private
|
||||
|
||||
private func nextCommandSeq() -> Int64 {
|
||||
let millis = Int64(Date().timeIntervalSince1970 * 1_000)
|
||||
currentCommandSeq = max(currentCommandSeq + 1, millis)
|
||||
return currentCommandSeq
|
||||
}
|
||||
|
||||
private func writeCommand(_ action: FlowCommand.Action) {
|
||||
guard let activeSessionId, let currentUtteranceId else { return }
|
||||
let command = FlowCommand(
|
||||
sessionId: activeSessionId,
|
||||
utteranceId: currentUtteranceId,
|
||||
commandSeq: nextCommandSeq(),
|
||||
action: action,
|
||||
localeId: state.localeId
|
||||
)
|
||||
FlowSessionBridge.writeCommand(command)
|
||||
debug(
|
||||
"command \(action.rawValue) seq=\(command.commandSeq) " +
|
||||
"utterance=\(currentUtteranceId.uuidString)"
|
||||
)
|
||||
}
|
||||
|
||||
private func consumePendingFlowDeliveryIfNeeded() {
|
||||
if isAwaitingFlowResult {
|
||||
if let delivery = FlowSessionBridge.consumeTranscriptionDelivery() {
|
||||
if let result = matchingResult(), result.status == .final, let text = result.text, !text.isEmpty {
|
||||
isAwaitingFlowResult = false
|
||||
stopFlowWatchdog()
|
||||
textInserter.handleFlowTranscript(delivery)
|
||||
FlowSessionBridge.writeAck(
|
||||
FlowAck(
|
||||
sessionId: result.sessionId,
|
||||
utteranceId: result.utteranceId,
|
||||
commandSeq: result.commandSeq
|
||||
)
|
||||
)
|
||||
FlowSessionBridge.clearResult()
|
||||
currentUtteranceId = nil
|
||||
textInserter.handleFlowTranscript(
|
||||
TranscriptionDelivery(text: text, polishWarning: result.warning)
|
||||
)
|
||||
return
|
||||
}
|
||||
if let error = FlowSessionBridge.consumeTranscriptionError() {
|
||||
if let result = matchingResult(), isTerminalFailure(result) {
|
||||
isAwaitingFlowResult = false
|
||||
stopFlowWatchdog()
|
||||
FlowSessionBridge.clearResult()
|
||||
currentUtteranceId = nil
|
||||
let error = FlowTranscriptionError(
|
||||
message: result.text ?? ExtL10n.string("keyboard.flow.resultTimeout"),
|
||||
kind: result.errorKind ?? .generic
|
||||
)
|
||||
state.phase = .error(
|
||||
.fromFlowTranscription(error),
|
||||
message: error.message
|
||||
@@ -330,6 +395,20 @@ final class KeyboardFlowCoordinator {
|
||||
}
|
||||
}
|
||||
|
||||
private func matchingResult() -> FlowResult? {
|
||||
guard let result = FlowSessionBridge.latestResult() else { return nil }
|
||||
guard let activeSessionId, let currentUtteranceId else { return nil }
|
||||
guard result.sessionId == activeSessionId,
|
||||
result.utteranceId == currentUtteranceId else {
|
||||
return nil
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
private func isTerminalFailure(_ result: FlowResult) -> Bool {
|
||||
result.status == .error || result.status == .timeout || result.status == .aborted
|
||||
}
|
||||
|
||||
/// When the host process died mid-utterance, abort local recording / waiting
|
||||
/// so the user is not stuck until the long result watchdog fires.
|
||||
private func recoverFromDeadHostIfNeeded() {
|
||||
@@ -339,7 +418,8 @@ final class KeyboardFlowCoordinator {
|
||||
isFlowRecording = false
|
||||
stopUtteranceCountdown()
|
||||
ExtensionScreenWakeLock.release()
|
||||
FlowSessionBridge.setRecordingState(.aborted)
|
||||
writeCommand(.abort)
|
||||
currentUtteranceId = nil
|
||||
stopFlowWatchdog()
|
||||
state.level = 0
|
||||
state.phase = .idle
|
||||
@@ -355,12 +435,14 @@ final class KeyboardFlowCoordinator {
|
||||
}
|
||||
|
||||
private func failHostDisconnected() {
|
||||
traceState("hostDisconnected.fail")
|
||||
isAwaitingFlowResult = false
|
||||
isFlowRecording = false
|
||||
isPendingFlowStart = false
|
||||
stopUtteranceCountdown()
|
||||
ExtensionScreenWakeLock.release()
|
||||
FlowSessionBridge.setRecordingState(.aborted)
|
||||
writeCommand(.abort)
|
||||
currentUtteranceId = nil
|
||||
stopFlowWatchdog()
|
||||
state.level = 0
|
||||
let message = ExtL10n.string("keyboard.flow.hostDisconnected")
|
||||
@@ -396,6 +478,7 @@ final class KeyboardFlowCoordinator {
|
||||
private func startFlowRecording() {
|
||||
recomputeMicVoiceAvailability()
|
||||
guard state.micVoiceAvailability.isReady else {
|
||||
traceState("startFlowRecording.blocked", extra: "availability=\(String(describing: state.micVoiceAvailability))")
|
||||
beginFlowStart()
|
||||
return
|
||||
}
|
||||
@@ -403,8 +486,14 @@ final class KeyboardFlowCoordinator {
|
||||
flowStartDeadline = 0
|
||||
stopFlowWatchdog()
|
||||
|
||||
FlowSessionBridge.setTranscriptionLanguage(state.localeId)
|
||||
FlowSessionBridge.setRecordingState(.recording)
|
||||
guard let sessionId = FlowSessionBridge.readySnapshot()?.sessionId else {
|
||||
traceState("startFlowRecording.blocked", extra: "reason=missingSessionIdInReadySnapshot")
|
||||
beginFlowStart()
|
||||
return
|
||||
}
|
||||
activeSessionId = sessionId
|
||||
currentUtteranceId = UUID()
|
||||
writeCommand(.startRecording)
|
||||
isFlowRecording = true
|
||||
state.lastTranscript = ""
|
||||
state.phase = .recording
|
||||
@@ -414,9 +503,7 @@ final class KeyboardFlowCoordinator {
|
||||
}
|
||||
startUtteranceCountdown()
|
||||
startFlowLevelWatchdog()
|
||||
// Read back in-process to confirm the write landed before we rely on
|
||||
// the host polling it out cross-process.
|
||||
debug("startFlowRecording wrote .recording (readback=\(FlowSessionBridge.recordingState().rawValue))")
|
||||
traceState("startFlowRecording.started")
|
||||
}
|
||||
|
||||
private func startUtteranceCountdown() {
|
||||
@@ -450,6 +537,7 @@ final class KeyboardFlowCoordinator {
|
||||
state.phase = .idle
|
||||
state.lastTranscript = ""
|
||||
recomputeMicVoiceAvailability()
|
||||
traceState("pendingStart.cancelledByUser")
|
||||
}
|
||||
|
||||
private func startFlowStartWatchdog() {
|
||||
@@ -465,6 +553,7 @@ final class KeyboardFlowCoordinator {
|
||||
if self.flowStartDeadline > 0, now > self.flowStartDeadline {
|
||||
self.isPendingFlowStart = false
|
||||
self.flowStartDeadline = 0
|
||||
self.traceState("startWatchdog.timeout")
|
||||
self.showManualOpenHint(path: "startflow")
|
||||
return
|
||||
}
|
||||
@@ -480,7 +569,7 @@ final class KeyboardFlowCoordinator {
|
||||
state.lastTranscript = ""
|
||||
refreshSessionState()
|
||||
startFlowRecording()
|
||||
debug("completeFlowStartHandoff → auto startFlowRecording")
|
||||
traceState("completeFlowStartHandoff.done")
|
||||
}
|
||||
|
||||
private func startFlowLevelWatchdog() {
|
||||
@@ -492,6 +581,12 @@ final class KeyboardFlowCoordinator {
|
||||
self.state.level = Double(peak)
|
||||
}
|
||||
self.refreshFlowPartialIfNeeded()
|
||||
let staleness = FlowSessionBridge.heartbeatStaleness() ?? .infinity
|
||||
if staleness > 5 {
|
||||
self.debug("levelWatchdog: host heartbeat stale while recording")
|
||||
self.failHostDisconnected()
|
||||
return
|
||||
}
|
||||
try? await Task.sleep(nanoseconds: FlowWatchdog.pollIntervalNs)
|
||||
}
|
||||
}
|
||||
@@ -501,7 +596,10 @@ final class KeyboardFlowCoordinator {
|
||||
guard isFlowRecording || isAwaitingFlowResult else { return }
|
||||
switch state.phase {
|
||||
case .recording, .processing:
|
||||
if let partial = FlowSessionBridge.transcriptionPartial() {
|
||||
if let result = matchingResult(),
|
||||
result.status == .partial,
|
||||
let partial = result.text,
|
||||
!partial.isEmpty {
|
||||
state.lastTranscript = partial
|
||||
}
|
||||
default:
|
||||
@@ -517,16 +615,33 @@ final class KeyboardFlowCoordinator {
|
||||
debug("resultWatchdog started timeout=\(Int(resultTimeout))s engine=\(state.engineMode)")
|
||||
flowWatchdogTask = Task { @MainActor [weak self] in
|
||||
while let self, !Task.isCancelled {
|
||||
if let delivery = FlowSessionBridge.consumeTranscriptionDelivery() {
|
||||
if let result = self.matchingResult(), result.status == .final, let text = result.text, !text.isEmpty {
|
||||
self.isAwaitingFlowResult = false
|
||||
self.stopFlowWatchdog()
|
||||
self.debug("resultWatchdog consumed delivery len=\(delivery.text.count)")
|
||||
self.textInserter.handleFlowTranscript(delivery)
|
||||
FlowSessionBridge.writeAck(
|
||||
FlowAck(
|
||||
sessionId: result.sessionId,
|
||||
utteranceId: result.utteranceId,
|
||||
commandSeq: result.commandSeq
|
||||
)
|
||||
)
|
||||
FlowSessionBridge.clearResult()
|
||||
self.currentUtteranceId = nil
|
||||
self.debug("resultWatchdog consumed delivery len=\(text.count)")
|
||||
self.textInserter.handleFlowTranscript(
|
||||
TranscriptionDelivery(text: text, polishWarning: result.warning)
|
||||
)
|
||||
return
|
||||
}
|
||||
if let error = FlowSessionBridge.consumeTranscriptionError() {
|
||||
if let result = self.matchingResult(), self.isTerminalFailure(result) {
|
||||
self.isAwaitingFlowResult = false
|
||||
self.stopFlowWatchdog()
|
||||
FlowSessionBridge.clearResult()
|
||||
self.currentUtteranceId = nil
|
||||
let error = FlowTranscriptionError(
|
||||
message: result.text ?? ExtL10n.string("keyboard.flow.resultTimeout"),
|
||||
kind: result.errorKind ?? .generic
|
||||
)
|
||||
self.debug("resultWatchdog consumed error kind=\(error.kind.rawValue)")
|
||||
self.state.phase = .error(
|
||||
.fromFlowTranscription(error),
|
||||
@@ -539,6 +654,11 @@ final class KeyboardFlowCoordinator {
|
||||
self.refreshFlowPartialIfNeeded()
|
||||
let now = Date().timeIntervalSince1970
|
||||
let staleness = FlowSessionBridge.heartbeatStaleness() ?? .infinity
|
||||
if self.isFlowRecording, staleness > 5 {
|
||||
self.debug("level/result watchdog: host heartbeat stale while recording")
|
||||
self.failHostDisconnected()
|
||||
return
|
||||
}
|
||||
if staleness > FlowSessionKeys.heartbeatZombieInterval {
|
||||
self.debug("resultWatchdog: host heartbeat zombie (staleness=\(String(format: "%.1f", staleness))s)")
|
||||
self.failHostDisconnected()
|
||||
@@ -553,6 +673,7 @@ final class KeyboardFlowCoordinator {
|
||||
if now - startedAt > resultTimeout {
|
||||
self.isAwaitingFlowResult = false
|
||||
self.stopFlowWatchdog()
|
||||
self.currentUtteranceId = nil
|
||||
self.debug("resultWatchdog TIMEOUT after \(Int(resultTimeout))s — no result from host")
|
||||
let msg = ExtL10n.string("keyboard.flow.resultTimeout")
|
||||
self.state.phase = .error(.flowResultTimeout, message: msg)
|
||||
@@ -573,4 +694,29 @@ final class KeyboardFlowCoordinator {
|
||||
private func debug(_ message: String) {
|
||||
OSGLog.keyboardExt.info("\(message, privacy: .public)")
|
||||
}
|
||||
|
||||
private func traceState(_ event: String, extra: String? = nil) {
|
||||
let staleness = FlowSessionBridge.heartbeatStaleness().map { String(format: "%.1f", $0) } ?? "nil"
|
||||
let sessionId = activeSessionId?.uuidString ?? "nil"
|
||||
let utteranceId = currentUtteranceId?.uuidString ?? "nil"
|
||||
let summary = [
|
||||
"event=\(event)",
|
||||
"phase=\(String(describing: state.phase))",
|
||||
"availability=\(String(describing: state.micVoiceAvailability))",
|
||||
"pending=\(isPendingFlowStart)",
|
||||
"recording=\(isFlowRecording)",
|
||||
"awaiting=\(isAwaitingFlowResult)",
|
||||
"sessionId=\(sessionId)",
|
||||
"utteranceId=\(utteranceId)",
|
||||
"cmdSeq=\(currentCommandSeq)",
|
||||
"sessionActive=\(FlowSessionBridge.isSessionActive())",
|
||||
"hostReady=\(FlowSessionBridge.isHostReady())",
|
||||
"heartbeatStaleness=\(staleness)"
|
||||
].joined(separator: " ")
|
||||
if let extra, !extra.isEmpty {
|
||||
debug("[trace] \(summary) \(extra)")
|
||||
} else {
|
||||
debug("[trace] \(summary)")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -52,7 +52,7 @@ private struct FlowLiveActivityLockScreenView: View {
|
||||
|
||||
var body: some View {
|
||||
HStack(spacing: 12) {
|
||||
FlowLiveActivityBrandMark(height: 18)
|
||||
FlowLiveActivityBrandMark(height: 13)
|
||||
VStack(alignment: .leading, spacing: 4) {
|
||||
Text("OSGKeyboard")
|
||||
.font(.headline)
|
||||
@@ -99,9 +99,10 @@ private struct FlowLiveActivityTrailingGlyph: View {
|
||||
.foregroundStyle(.red)
|
||||
.symbolEffect(.variableColor.iterative, options: .repeating)
|
||||
case .processing:
|
||||
ProgressView()
|
||||
.progressViewStyle(.circular)
|
||||
.tint(.white)
|
||||
Image(systemName: "ellipsis")
|
||||
.font(.title3.weight(.semibold))
|
||||
.foregroundStyle(.white)
|
||||
.symbolEffect(.variableColor.iterative, options: .repeating)
|
||||
case .idle:
|
||||
// Session ready but NOT listening — avoid a mic glyph so users
|
||||
// don't think the keyboard is recording in the background.
|
||||
|
||||
@@ -183,37 +183,48 @@ struct BottomDictationBar: View {
|
||||
.fixedSize()
|
||||
}
|
||||
|
||||
// 麦克风按钮始终居中固定:录音时的波形放进按钮内部,
|
||||
// “按停止”提示作为浮层显示在按钮上方,二者均不参与布局,
|
||||
// 因此按下 Option 触发录音时按钮位置不会发生偏移。
|
||||
private var recordControl: some View {
|
||||
HStack(spacing: Spacing.sm) {
|
||||
if viewModel.isRecording {
|
||||
MiniWaveform(level: viewModel.audioLevel)
|
||||
recordButton
|
||||
.overlay(alignment: .top) {
|
||||
if viewModel.isRecording {
|
||||
Text(MacL10n.string("mac.record.pressStop", language: lang))
|
||||
.font(TypeStyle.caption)
|
||||
.foregroundStyle(palette.textTertiary)
|
||||
.fixedSize()
|
||||
.offset(y: -22)
|
||||
}
|
||||
}
|
||||
Button(action: viewModel.toggleRecording) {
|
||||
ZStack {
|
||||
Circle()
|
||||
.fill(viewModel.isRecording ? palette.recordRed : palette.accent)
|
||||
.frame(width: 52, height: 52)
|
||||
.macGlassSurface(in: Circle(), fillOpacity: 0.2)
|
||||
.shadow(
|
||||
color: (viewModel.isRecording ? palette.recordRed : palette.accent).opacity(0.5),
|
||||
radius: pulse ? 14 : 6
|
||||
)
|
||||
Image(systemName: viewModel.isRecording ? "stop.fill" : "mic.fill")
|
||||
}
|
||||
|
||||
private var recordButton: some View {
|
||||
Button(action: viewModel.toggleRecording) {
|
||||
ZStack {
|
||||
Circle()
|
||||
.fill(viewModel.isRecording ? palette.recordRed : palette.accent)
|
||||
.frame(width: 52, height: 52)
|
||||
.macGlassSurface(in: Circle(), fillOpacity: 0.2)
|
||||
.shadow(
|
||||
color: (viewModel.isRecording ? palette.recordRed : palette.accent).opacity(0.5),
|
||||
radius: pulse ? 14 : 6
|
||||
)
|
||||
if viewModel.isRecording {
|
||||
// 与 iOS 一致:录音时在红色按钮内部显示实时波形
|
||||
MiniWaveform(level: viewModel.audioLevel, barCount: 4, tint: palette.textOnAccent)
|
||||
} else {
|
||||
Image(systemName: "mic.fill")
|
||||
.font(.system(size: 20, weight: .bold))
|
||||
.foregroundStyle(palette.textOnAccent)
|
||||
}
|
||||
}
|
||||
.buttonStyle(.plain)
|
||||
.disabled(viewModel.isProcessing)
|
||||
.onAppear {
|
||||
withAnimation(.easeInOut(duration: 1.1).repeatForever(autoreverses: true)) {
|
||||
pulse = true
|
||||
}
|
||||
}
|
||||
if viewModel.isRecording {
|
||||
Text(MacL10n.string("mac.record.pressStop", language: lang))
|
||||
.font(TypeStyle.caption)
|
||||
.foregroundStyle(palette.textTertiary)
|
||||
}
|
||||
.buttonStyle(.plain)
|
||||
.disabled(viewModel.isProcessing)
|
||||
.onAppear {
|
||||
withAnimation(.easeInOut(duration: 1.1).repeatForever(autoreverses: true)) {
|
||||
pulse = true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -31,9 +31,30 @@ enum MacDictationPipeline {
|
||||
|
||||
let locale = Locale(identifier: store.localeId.isEmpty ? "zh-CN" : store.localeId)
|
||||
let raw: String
|
||||
var localBias: LocalASRBiasPayload?
|
||||
|
||||
if store.engineMode == "local" {
|
||||
raw = try await MacLocalASRService.transcribe(samples: samples, locale: locale)
|
||||
MacAppContextService.captureAndPersist(to: store)
|
||||
let capabilities = MacLocalASRService.currentCapabilities()
|
||||
let bias = LocalASRBiasAdapter.adapt(
|
||||
LocalASRBiasRequest(
|
||||
dictionary: store.personalDictionary,
|
||||
locale: locale,
|
||||
frontAppBundleId: MacAppContextService.frontmostBundleIdentifier(),
|
||||
capabilities: capabilities
|
||||
)
|
||||
)
|
||||
localBias = bias
|
||||
LocalASRBiasDiagnosticsStore.save(
|
||||
payload: bias,
|
||||
modelId: MacLocalASRService.selectedModelDefinition()?.id,
|
||||
backendLabel: MacLocalASRService.currentBackendLabel()
|
||||
)
|
||||
raw = try await MacLocalASRService.transcribe(
|
||||
samples: samples,
|
||||
locale: locale,
|
||||
bias: bias
|
||||
)
|
||||
} else {
|
||||
let strategy = CloudASRModelCatalog.strategy(for: store.providerId)
|
||||
guard strategy != .localFallback else { throw MacDictationError.providerHasNoCloudASR }
|
||||
@@ -51,13 +72,33 @@ enum MacDictationPipeline {
|
||||
let trimmed = raw.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||
guard !trimmed.isEmpty else { throw MacDictationError.emptyTranscript }
|
||||
|
||||
let postASR: String
|
||||
if let localBias, !localBias.correctionPairs.isEmpty {
|
||||
postASR = LocalASRTranscriptCorrector.apply(trimmed, pairs: localBias.correctionPairs)
|
||||
} else {
|
||||
postASR = trimmed
|
||||
}
|
||||
|
||||
let polishContext: PolishContext?
|
||||
if let supplement = localBias?.polishFragment.trimmingCharacters(in: .whitespacesAndNewlines),
|
||||
!supplement.isEmpty {
|
||||
polishContext = PolishContext(
|
||||
appContext: store.detectedAppContext?.context ?? .unknown,
|
||||
intensity: store.polishIntensity,
|
||||
dictionarySupplement: supplement
|
||||
)
|
||||
} else {
|
||||
polishContext = nil
|
||||
}
|
||||
|
||||
if let polished = try? await PolishingService(store: store).polish(
|
||||
trimmed,
|
||||
mode: store.polishModeForPipeline
|
||||
postASR,
|
||||
mode: store.polishModeForPipeline,
|
||||
context: polishContext
|
||||
),
|
||||
!polished.isEmpty {
|
||||
return polished
|
||||
}
|
||||
return trimmed
|
||||
return postASR
|
||||
}
|
||||
}
|
||||
|
||||
@@ -107,8 +107,9 @@ final class MacDictationViewModel: ObservableObject {
|
||||
/// Pre-load MLX weights + Metal shaders so the first dictation is fast.
|
||||
func warmUpQwen3IfNeeded() {
|
||||
guard config.engineMode == "local",
|
||||
MacLocalASRPreferences.backend == .qwen3MLX,
|
||||
MacLocalASRPreferences.qwen3ModelIsInstalled() else { return }
|
||||
let model = MacLocalASRService.selectedModelDefinition(),
|
||||
model.backend == .mlx,
|
||||
MacLocalASRService.isModelInstalled(model) else { return }
|
||||
let path = MacLocalASRPreferences.qwen3ModelPath
|
||||
Task.detached(priority: .utility) {
|
||||
_ = try? await MacQwen3ASREngine.shared.prepareIfNeeded(modelPath: path)
|
||||
@@ -156,6 +157,39 @@ final class MacDictationViewModel: ObservableObject {
|
||||
return "\(seconds)s"
|
||||
}
|
||||
|
||||
var localModelReady: Bool {
|
||||
_ = localModelRevision
|
||||
if let model = MacLocalASRService.selectedModelDefinition() {
|
||||
return MacLocalASRService.isModelInstalled(model)
|
||||
}
|
||||
return MacLocalASRPreferences.qwen3ModelIsInstalled()
|
||||
}
|
||||
|
||||
/// Context-aware warning when local engine is selected but the active model is not ready.
|
||||
var localModelWarningMessage: String? {
|
||||
_ = localModelRevision
|
||||
guard config.engineMode == "local" else { return nil }
|
||||
if localModelReady { return nil }
|
||||
guard let model = MacLocalASRService.selectedModelDefinition() else {
|
||||
return MacL10n.string("mac.settings.localModelFallbackApple", language: config.uiLanguage)
|
||||
}
|
||||
if model.installKind == .manual {
|
||||
return MacL10n.string("mac.settings.mlxModelMissing", language: config.uiLanguage)
|
||||
}
|
||||
return MacL10n.format(
|
||||
"mac.settings.selectedModelMissing",
|
||||
language: config.uiLanguage,
|
||||
model.displayName
|
||||
)
|
||||
}
|
||||
|
||||
@Published private(set) var localModelRevision = 0
|
||||
|
||||
func bumpLocalModelRevision() {
|
||||
localModelRevision += 1
|
||||
objectWillChange.send()
|
||||
}
|
||||
|
||||
var qwen3ModelInstalled: Bool {
|
||||
MacLocalASRPreferences.qwen3ModelIsInstalled()
|
||||
}
|
||||
|
||||
@@ -0,0 +1,455 @@
|
||||
// MacLocalASRModelSettingsView.swift
|
||||
// OSGKeyboard · Mac
|
||||
//
|
||||
// Local ASR model catalog, download progress, MLX path, and bias diagnostics.
|
||||
|
||||
import AppKit
|
||||
import SwiftUI
|
||||
|
||||
@MainActor
|
||||
final class MacLocalASRModelSettingsViewModel: ObservableObject {
|
||||
@Published var catalog: LocalASRCatalogDocument?
|
||||
@Published var selectedModelId: String = MacLocalASRPreferences.selectedModelId
|
||||
@Published var installProgress = LocalASRModelInstallProgress.idle
|
||||
@Published var diagnosticsSnapshot = LocalASRBiasDiagnosticsStore.load()
|
||||
@Published var statusMessage = ""
|
||||
@Published var isInstalling = false
|
||||
@Published var isDownloadPaused = false
|
||||
|
||||
var onLocalModelStateChanged: (() -> Void)?
|
||||
|
||||
private let manager = LocalASRModelManager.shared
|
||||
private var progressPollTask: Task<Void, Never>?
|
||||
|
||||
deinit {
|
||||
progressPollTask?.cancel()
|
||||
}
|
||||
|
||||
func reload() {
|
||||
catalog = try? LocalASRModelCatalog.loadBundled()
|
||||
if let catalog {
|
||||
let manifest = LocalASRInstalledManifestIO.load(defaultModelId: catalog.defaultModelId)
|
||||
selectedModelId = manifest.selectedModelId.isEmpty
|
||||
? MacLocalASRPreferences.selectedModelId
|
||||
: manifest.selectedModelId
|
||||
}
|
||||
diagnosticsSnapshot = LocalASRBiasDiagnosticsStore.load()
|
||||
onLocalModelStateChanged?()
|
||||
}
|
||||
|
||||
func isInstalled(_ model: LocalASRModelDefinition) -> Bool {
|
||||
MacLocalASRService.isModelInstalled(model)
|
||||
}
|
||||
|
||||
func isInstallingModel(_ model: LocalASRModelDefinition) -> Bool {
|
||||
isInstalling && installProgress.activeItemId == model.id
|
||||
}
|
||||
|
||||
func installedDiskUsage(_ model: LocalASRModelDefinition) -> String? {
|
||||
guard model.installKind == .archive,
|
||||
let relative = model.installRelativePath,
|
||||
isInstalled(model) else { return nil }
|
||||
let dir = LocalASRModelInstallState.installDirectory(for: relative)
|
||||
let bytes = LocalASRModelInstallState.directoryByteCount(at: dir)
|
||||
guard bytes > 0 else { return nil }
|
||||
return ByteCountFormatter.string(fromByteCount: bytes, countStyle: .file)
|
||||
}
|
||||
|
||||
func currentRuntime(in catalog: LocalASRCatalogDocument) -> LocalASRRuntimeDefinition? {
|
||||
LocalASRModelCatalog.runtime(for: LocalASRModelCatalog.currentRuntimePlatform(), in: catalog)
|
||||
}
|
||||
|
||||
func isRuntimeInstalled(_ runtime: LocalASRRuntimeDefinition) -> Bool {
|
||||
LocalASRModelInstallState.isRuntimeInstalled(runtime)
|
||||
}
|
||||
|
||||
func selectModel(_ modelId: String) {
|
||||
guard let catalog, !isInstalling else { return }
|
||||
selectedModelId = modelId
|
||||
MacLocalASRPreferences.selectedModelId = modelId
|
||||
var manifest = LocalASRInstalledManifestIO.load(defaultModelId: catalog.defaultModelId)
|
||||
manifest.selectedModelId = modelId
|
||||
manifest.updatedAt = Date()
|
||||
try? LocalASRInstalledManifestIO.save(manifest)
|
||||
onLocalModelStateChanged?()
|
||||
}
|
||||
|
||||
func installModel(_ model: LocalASRModelDefinition) {
|
||||
guard let catalog, !isInstalling else { return }
|
||||
statusMessage = ""
|
||||
isInstalling = true
|
||||
isDownloadPaused = false
|
||||
startProgressPolling()
|
||||
Task {
|
||||
do {
|
||||
try await manager.installModel(model, catalog: catalog)
|
||||
installProgress = await manager.currentProgress()
|
||||
selectModel(model.id)
|
||||
statusMessage = MacL10n.string("mac.localASR.installDone")
|
||||
} catch {
|
||||
installProgress = await manager.currentProgress()
|
||||
statusMessage = error.localizedDescription
|
||||
}
|
||||
isInstalling = false
|
||||
isDownloadPaused = false
|
||||
stopProgressPolling()
|
||||
reload()
|
||||
}
|
||||
}
|
||||
|
||||
func pauseDownload() {
|
||||
Task {
|
||||
do {
|
||||
try await manager.pauseDownload()
|
||||
isDownloadPaused = true
|
||||
installProgress = await manager.currentProgress()
|
||||
} catch {
|
||||
statusMessage = error.localizedDescription
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func resumeDownload() {
|
||||
Task {
|
||||
do {
|
||||
try await manager.resumeDownload()
|
||||
isDownloadPaused = false
|
||||
installProgress = await manager.currentProgress()
|
||||
} catch {
|
||||
statusMessage = error.localizedDescription
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func deleteModel(_ model: LocalASRModelDefinition) {
|
||||
guard let catalog, !isInstalling else { return }
|
||||
Task {
|
||||
do {
|
||||
try await manager.deleteModel(model, catalog: catalog)
|
||||
statusMessage = MacL10n.string("mac.localASR.deleteDone")
|
||||
reload()
|
||||
} catch {
|
||||
statusMessage = error.localizedDescription
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func revealModelInFinder(_ model: LocalASRModelDefinition) {
|
||||
guard let relative = model.installRelativePath else { return }
|
||||
let url = LocalASRModelInstallState.installDirectory(for: relative)
|
||||
NSWorkspace.shared.activateFileViewerSelecting([url])
|
||||
}
|
||||
|
||||
/// Opens (creating if needed) the model's shared subfolder so the user can
|
||||
/// drop in manually-converted weights (used by the MLX model).
|
||||
func revealModelFolder(_ model: LocalASRModelDefinition) {
|
||||
guard let relative = model.installRelativePath else { return }
|
||||
let url = LocalASRModelInstallState.installDirectory(for: relative)
|
||||
try? FileManager.default.createDirectory(at: url, withIntermediateDirectories: true)
|
||||
NSWorkspace.shared.open(url)
|
||||
}
|
||||
|
||||
func revealStorageRoot() {
|
||||
let url = LocalASRModelInstallState.rootDirectory()
|
||||
try? FileManager.default.createDirectory(at: url, withIntermediateDirectories: true)
|
||||
NSWorkspace.shared.open(url)
|
||||
}
|
||||
|
||||
func progressLabel(for progress: LocalASRModelInstallProgress, language: AppUILanguage) -> String {
|
||||
let phaseKey: String
|
||||
switch progress.phase {
|
||||
case .downloading: phaseKey = "mac.localASR.phase.downloading"
|
||||
case .paused: phaseKey = "mac.localASR.phase.paused"
|
||||
case .extracting: phaseKey = "mac.localASR.phase.extracting"
|
||||
case .validating: phaseKey = "mac.localASR.phase.validating"
|
||||
case .finalizing: phaseKey = "mac.localASR.phase.finalizing"
|
||||
case .failed: phaseKey = "mac.localASR.phase.failed"
|
||||
case .completed: phaseKey = "mac.localASR.phase.completed"
|
||||
case .idle: return progress.message
|
||||
}
|
||||
let phase = MacL10n.string(phaseKey, language: language)
|
||||
if let received = progress.bytesReceived, let total = progress.bytesTotal, total > 0 {
|
||||
let recv = ByteCountFormatter.string(fromByteCount: received, countStyle: .file)
|
||||
let tot = ByteCountFormatter.string(fromByteCount: total, countStyle: .file)
|
||||
return "\(phase) · \(progress.message) (\(recv) / \(tot))"
|
||||
}
|
||||
return "\(phase) · \(progress.message)"
|
||||
}
|
||||
|
||||
private func startProgressPolling() {
|
||||
progressPollTask?.cancel()
|
||||
progressPollTask = Task { [weak self] in
|
||||
while !Task.isCancelled {
|
||||
guard let self else { return }
|
||||
let current = await manager.currentProgress()
|
||||
await MainActor.run {
|
||||
self.installProgress = current
|
||||
self.isDownloadPaused = current.phase == .paused
|
||||
}
|
||||
try? await Task.sleep(for: .milliseconds(120))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private func stopProgressPolling() {
|
||||
progressPollTask?.cancel()
|
||||
progressPollTask = nil
|
||||
}
|
||||
|
||||
func formattedSize(_ bytes: Int) -> String {
|
||||
ByteCountFormatter.string(fromByteCount: Int64(bytes), countStyle: .file)
|
||||
}
|
||||
}
|
||||
|
||||
struct MacLocalASRModelSettingsView: View {
|
||||
@ObservedObject var viewModel: MacDictationViewModel
|
||||
@StateObject private var modelVM = MacLocalASRModelSettingsViewModel()
|
||||
@Environment(\.themePalette) private var palette
|
||||
|
||||
private var lang: AppUILanguage { viewModel.config.uiLanguage }
|
||||
|
||||
var body: some View {
|
||||
Group {
|
||||
if let catalog = modelVM.catalog {
|
||||
modelPickerSection(catalog: catalog)
|
||||
runtimeSection(catalog: catalog)
|
||||
} else {
|
||||
Text(MacL10n.string("mac.localASR.catalogMissing", language: lang))
|
||||
.foregroundStyle(palette.textSecondary)
|
||||
}
|
||||
}
|
||||
.onAppear {
|
||||
modelVM.onLocalModelStateChanged = { viewModel.bumpLocalModelRevision() }
|
||||
modelVM.reload()
|
||||
}
|
||||
}
|
||||
|
||||
private func modelPickerSection(catalog: LocalASRCatalogDocument) -> some View {
|
||||
Section {
|
||||
ForEach(catalog.models) { model in
|
||||
modelRow(model)
|
||||
}
|
||||
|
||||
if modelVM.isInstalling,
|
||||
modelVM.installProgress.phase == .extracting
|
||||
|| modelVM.installProgress.phase == .validating
|
||||
|| modelVM.installProgress.phase == .finalizing {
|
||||
ProgressView(value: modelVM.installProgress.fraction) {
|
||||
Text(modelVM.progressLabel(for: modelVM.installProgress, language: lang))
|
||||
.font(TypeStyle.caption)
|
||||
}
|
||||
}
|
||||
|
||||
if !modelVM.statusMessage.isEmpty {
|
||||
Text(modelVM.statusMessage)
|
||||
.font(TypeStyle.caption)
|
||||
.foregroundStyle(palette.textSecondary)
|
||||
}
|
||||
|
||||
Button(MacL10n.string("mac.localASR.openStorage", language: lang)) {
|
||||
modelVM.revealStorageRoot()
|
||||
}
|
||||
} header: {
|
||||
Text(MacL10n.string("mac.localASR.models", language: lang))
|
||||
} footer: {
|
||||
Text(MacL10n.string("mac.localASR.modelsDesc", language: lang))
|
||||
.font(TypeStyle.caption)
|
||||
.foregroundStyle(palette.textSecondary)
|
||||
}
|
||||
}
|
||||
|
||||
private func runtimeSection(catalog: LocalASRCatalogDocument) -> some View {
|
||||
Group {
|
||||
if let runtime = modelVM.currentRuntime(in: catalog) {
|
||||
Section {
|
||||
LabeledContent(runtime.displayName) {
|
||||
Text(
|
||||
modelVM.isRuntimeInstalled(runtime)
|
||||
? MacL10n.string("mac.localASR.installed", language: lang)
|
||||
: MacL10n.string("mac.localASR.notInstalled", language: lang)
|
||||
)
|
||||
}
|
||||
Text(MacL10n.string("mac.localASR.runtimeDesc", language: lang))
|
||||
.font(TypeStyle.caption)
|
||||
.foregroundStyle(palette.textSecondary)
|
||||
} header: {
|
||||
Text(MacL10n.string("mac.localASR.runtime", language: lang))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@ViewBuilder
|
||||
private func modelRow(_ model: LocalASRModelDefinition) -> some View {
|
||||
let installed = modelVM.isInstalled(model)
|
||||
let selected = modelVM.selectedModelId == model.id
|
||||
let installing = modelVM.isInstallingModel(model)
|
||||
|
||||
VStack(alignment: .leading, spacing: Spacing.xs) {
|
||||
HStack(alignment: .top) {
|
||||
Button {
|
||||
modelVM.selectModel(model.id)
|
||||
} label: {
|
||||
HStack(spacing: Spacing.sm) {
|
||||
Image(systemName: selected ? "largecircle.fill.circle" : "circle")
|
||||
.foregroundStyle(selected ? palette.accent : palette.textTertiary)
|
||||
VStack(alignment: .leading, spacing: 2) {
|
||||
Text(model.displayName)
|
||||
.foregroundStyle(palette.textPrimary)
|
||||
Text(modelSubtitle(model, installed: installed))
|
||||
.font(TypeStyle.caption)
|
||||
.foregroundStyle(palette.textSecondary)
|
||||
}
|
||||
}
|
||||
}
|
||||
.buttonStyle(.plain)
|
||||
.disabled(modelVM.isInstalling)
|
||||
|
||||
Spacer()
|
||||
|
||||
modelRowActions(model: model, installed: installed, installing: installing)
|
||||
}
|
||||
}
|
||||
.padding(.vertical, 2)
|
||||
}
|
||||
|
||||
@ViewBuilder
|
||||
private func modelRowActions(
|
||||
model: LocalASRModelDefinition,
|
||||
installed: Bool,
|
||||
installing: Bool
|
||||
) -> some View {
|
||||
if installing {
|
||||
HStack(spacing: Spacing.sm) {
|
||||
circularInstallProgress(for: model)
|
||||
if modelVM.installProgress.phase == .downloading
|
||||
|| modelVM.installProgress.phase == .paused {
|
||||
Button {
|
||||
if modelVM.isDownloadPaused {
|
||||
modelVM.resumeDownload()
|
||||
} else {
|
||||
modelVM.pauseDownload()
|
||||
}
|
||||
} label: {
|
||||
Image(systemName: modelVM.isDownloadPaused ? "play.fill" : "pause.fill")
|
||||
.font(.system(size: 12, weight: .semibold))
|
||||
.frame(width: 28, height: 28)
|
||||
}
|
||||
.buttonStyle(.bordered)
|
||||
.controlSize(.small)
|
||||
.help(
|
||||
modelVM.isDownloadPaused
|
||||
? MacL10n.string("mac.localASR.resume", language: lang)
|
||||
: MacL10n.string("mac.localASR.pause", language: lang)
|
||||
)
|
||||
}
|
||||
}
|
||||
} else if model.installKind == .manual {
|
||||
Button(MacL10n.string("mac.localASR.openFolder", language: lang)) {
|
||||
modelVM.revealModelFolder(model)
|
||||
}
|
||||
.buttonStyle(.bordered)
|
||||
.controlSize(.small)
|
||||
} else if installed {
|
||||
Button(MacL10n.string("mac.localASR.delete", language: lang), role: .destructive) {
|
||||
modelVM.deleteModel(model)
|
||||
}
|
||||
.buttonStyle(.bordered)
|
||||
.controlSize(.small)
|
||||
} else {
|
||||
Button(MacL10n.string("mac.localASR.download", language: lang)) {
|
||||
modelVM.installModel(model)
|
||||
}
|
||||
.buttonStyle(.borderedProminent)
|
||||
.controlSize(.small)
|
||||
}
|
||||
}
|
||||
|
||||
private func circularInstallProgress(for model: LocalASRModelDefinition) -> some View {
|
||||
let fraction: Double = {
|
||||
if modelVM.installProgress.phase == .downloading || modelVM.installProgress.phase == .paused,
|
||||
let received = modelVM.installProgress.bytesReceived,
|
||||
let total = modelVM.installProgress.bytesTotal,
|
||||
total > 0 {
|
||||
return min(1, max(0, Double(received) / Double(total)))
|
||||
}
|
||||
return modelVM.installProgress.fraction
|
||||
}()
|
||||
return ZStack {
|
||||
Circle()
|
||||
.stroke(palette.textTertiary.opacity(0.25), lineWidth: 3)
|
||||
Circle()
|
||||
.trim(from: 0, to: fraction)
|
||||
.stroke(palette.accent, style: StrokeStyle(lineWidth: 3, lineCap: .round))
|
||||
.rotationEffect(.degrees(-90))
|
||||
.animation(.linear(duration: 0.15), value: fraction)
|
||||
if modelVM.installProgress.phase == .paused {
|
||||
Image(systemName: "pause.fill")
|
||||
.font(.system(size: 10, weight: .bold))
|
||||
.foregroundStyle(palette.textSecondary)
|
||||
} else {
|
||||
Text("\(Int(fraction * 100))%")
|
||||
.font(.system(size: 9, weight: .medium, design: .rounded))
|
||||
.foregroundStyle(palette.textSecondary)
|
||||
}
|
||||
}
|
||||
.frame(width: 36, height: 36)
|
||||
.accessibilityLabel(modelVM.progressLabel(for: modelVM.installProgress, language: lang))
|
||||
}
|
||||
|
||||
private func modelSubtitle(_ model: LocalASRModelDefinition, installed: Bool) -> String {
|
||||
let size = modelVM.formattedSize(model.sizeBytes)
|
||||
let hotword = model.supportsHotwords
|
||||
? MacL10n.string("mac.localASR.hotwordsYes", language: lang)
|
||||
: MacL10n.string("mac.localASR.hotwordsNo", language: lang)
|
||||
let state = installed
|
||||
? MacL10n.string("mac.localASR.installed", language: lang)
|
||||
: MacL10n.string("mac.localASR.notInstalled", language: lang)
|
||||
if let usage = modelVM.installedDiskUsage(model) {
|
||||
return "\(size) · \(hotword) · \(state) · \(usage)"
|
||||
}
|
||||
return "\(size) · \(hotword) · \(state)"
|
||||
}
|
||||
|
||||
private var diagnosticsSection: some View {
|
||||
Section {
|
||||
if let snapshot = modelVM.diagnosticsSnapshot {
|
||||
LabeledContent(MacL10n.string("mac.localASR.diagBackend", language: lang)) {
|
||||
Text(snapshot.backendLabel ?? "—")
|
||||
}
|
||||
LabeledContent(MacL10n.string("mac.localASR.diagUserTerms", language: lang)) {
|
||||
Text("\(snapshot.diagnostics.userTermCount)")
|
||||
}
|
||||
LabeledContent(MacL10n.string("mac.localASR.diagBuiltinTerms", language: lang)) {
|
||||
Text("\(snapshot.diagnostics.builtinTermCount)")
|
||||
}
|
||||
LabeledContent(MacL10n.string("mac.localASR.diagHotwords", language: lang)) {
|
||||
Text("\(snapshot.hotwordCount)")
|
||||
}
|
||||
LabeledContent(MacL10n.string("mac.localASR.diagPrompt", language: lang)) {
|
||||
Text("\(snapshot.promptBiasLength)")
|
||||
}
|
||||
if snapshot.diagnostics.truncated {
|
||||
Label(
|
||||
snapshot.diagnostics.truncationReason ?? MacL10n.string("mac.localASR.diagTruncated", language: lang),
|
||||
systemImage: "exclamationmark.triangle"
|
||||
)
|
||||
.font(TypeStyle.caption)
|
||||
.foregroundStyle(palette.warning)
|
||||
}
|
||||
Text(snapshot.diagnostics.selectedSources.joined(separator: ", "))
|
||||
.font(TypeStyle.caption)
|
||||
.foregroundStyle(palette.textSecondary)
|
||||
} else {
|
||||
Text(MacL10n.string("mac.localASR.diagEmpty", language: lang))
|
||||
.foregroundStyle(palette.textSecondary)
|
||||
}
|
||||
} header: {
|
||||
Text(MacL10n.string("mac.localASR.diagnostics", language: lang))
|
||||
} footer: {
|
||||
Text(MacL10n.string("mac.localASR.diagnosticsDesc", language: lang))
|
||||
.font(TypeStyle.caption)
|
||||
.foregroundStyle(palette.textSecondary)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,13 +1,15 @@
|
||||
// MacLocalASRService.swift
|
||||
// OSGKeyboard · Mac
|
||||
//
|
||||
// On-device ASR for macOS. Primary: Qwen3-ASR-1.7B (MLX via mlx-swift-asr).
|
||||
// Falls back to Apple Speech when Qwen3 weights are absent or backend is Apple Speech.
|
||||
// On-device ASR for macOS. Routes through the bundled local ASR catalog:
|
||||
// Qwen3 MLX (default), Sherpa Qwen3 hotwords POC, SenseVoice, Apple Speech fallback.
|
||||
|
||||
import Foundation
|
||||
|
||||
enum MacLocalASRBackend: String, Sendable, CaseIterable {
|
||||
case qwen3MLX
|
||||
case sherpaQwen3
|
||||
case sherpaSenseVoice
|
||||
case appleSpeech
|
||||
}
|
||||
|
||||
@@ -39,28 +41,33 @@ enum MacLocalASRError: Error, LocalizedError {
|
||||
|
||||
enum MacLocalASRPreferences {
|
||||
static let backendKey = "mac.localASR.backend"
|
||||
static let qwen3ModelPathKey = "mac.localASR.qwen3ModelPath"
|
||||
static let selectedModelIdKey = LocalASRPreferenceKeys.selectedModelId
|
||||
/// Shared managed subfolder for the manually-provided MLX weights.
|
||||
static let qwen3ModelRelativePath = "models/qwen3-asr-1.7b-mlx"
|
||||
|
||||
static var backend: MacLocalASRBackend {
|
||||
static var selectedModelId: String {
|
||||
get {
|
||||
guard let raw = UserDefaults.standard.string(forKey: backendKey),
|
||||
let value = MacLocalASRBackend(rawValue: raw) else {
|
||||
return .qwen3MLX
|
||||
if let raw = UserDefaults.standard.string(forKey: selectedModelIdKey), !raw.isEmpty {
|
||||
return raw
|
||||
}
|
||||
return value
|
||||
return legacyBackend == .appleSpeech ? "apple-speech-fallback" : "qwen3-mlx-1.7b"
|
||||
}
|
||||
set { UserDefaults.standard.set(newValue.rawValue, forKey: backendKey) }
|
||||
set { UserDefaults.standard.set(newValue, forKey: selectedModelIdKey) }
|
||||
}
|
||||
|
||||
static var legacyBackend: MacLocalASRBackend {
|
||||
guard let raw = UserDefaults.standard.string(forKey: backendKey),
|
||||
let value = MacLocalASRBackend(rawValue: raw) else {
|
||||
return .qwen3MLX
|
||||
}
|
||||
return value
|
||||
}
|
||||
|
||||
/// Fixed location inside the shared managed model storage root. All three
|
||||
/// catalog models live under the same directory, so MLX no longer needs a
|
||||
/// per-model folder picker — the user drops converted weights here.
|
||||
static var qwen3ModelPath: String {
|
||||
get { UserDefaults.standard.string(forKey: qwen3ModelPathKey) ?? defaultQwen3ModelPath }
|
||||
set { UserDefaults.standard.set(newValue, forKey: qwen3ModelPathKey) }
|
||||
}
|
||||
|
||||
/// Default install location for MLX-converted Qwen3-ASR weights.
|
||||
static var defaultQwen3ModelPath: String {
|
||||
let appSupport = FileManager.default.urls(for: .applicationSupportDirectory, in: .userDomainMask).first!
|
||||
return appSupport.appendingPathComponent("OSGKeyboard/models/qwen3-asr-1.7b-mlx", isDirectory: true).path
|
||||
LocalASRModelInstallState.installDirectory(for: qwen3ModelRelativePath).path
|
||||
}
|
||||
|
||||
static func qwen3ModelIsInstalled(at path: String = qwen3ModelPath) -> Bool {
|
||||
@@ -80,24 +87,93 @@ enum MacLocalASRPreferences {
|
||||
}
|
||||
|
||||
enum MacLocalASRService {
|
||||
/// Transcribe using the user's preferred local backend with automatic
|
||||
/// fallback to Apple Speech when Qwen3 weights are not present.
|
||||
static func transcribe(samples: [Float], locale: Locale) async throws -> String {
|
||||
let preferQwen3 = MacLocalASRPreferences.backend == .qwen3MLX
|
||||
if preferQwen3, MacLocalASRPreferences.qwen3ModelIsInstalled() {
|
||||
|
||||
static func loadCatalog() -> LocalASRCatalogDocument? {
|
||||
try? LocalASRModelCatalog.loadBundled()
|
||||
}
|
||||
|
||||
static func selectedModelDefinition() -> LocalASRModelDefinition? {
|
||||
guard let catalog = loadCatalog() else { return nil }
|
||||
let manifest = LocalASRInstalledManifestIO.load(defaultModelId: catalog.defaultModelId)
|
||||
let selectedId = manifest.selectedModelId.isEmpty
|
||||
? MacLocalASRPreferences.selectedModelId
|
||||
: manifest.selectedModelId
|
||||
if selectedId == "apple-speech-fallback" { return nil }
|
||||
return LocalASRModelCatalog.model(selectedId, in: catalog)
|
||||
?? LocalASRModelCatalog.model(catalog.defaultModelId, in: catalog)
|
||||
}
|
||||
|
||||
static func currentCapabilities() -> LocalASRCapabilities {
|
||||
guard let model = selectedModelDefinition() else { return .appleSpeech }
|
||||
return LocalASRModelCatalog.capabilities(for: model)
|
||||
}
|
||||
|
||||
static func currentBackendLabel() -> String {
|
||||
guard let model = selectedModelDefinition() else { return "Apple Speech" }
|
||||
return model.displayName
|
||||
}
|
||||
|
||||
static func isModelInstalled(_ model: LocalASRModelDefinition) -> Bool {
|
||||
LocalASRModelInstallState.isInstalled(
|
||||
model,
|
||||
manualMLXPath: MacLocalASRPreferences.qwen3ModelPath
|
||||
)
|
||||
}
|
||||
|
||||
/// Transcribe using the selected catalog model, with MLX → Apple Speech fallback.
|
||||
static func transcribe(
|
||||
samples: [Float],
|
||||
locale: Locale,
|
||||
bias: LocalASRBiasPayload? = nil
|
||||
) async throws -> String {
|
||||
if let model = selectedModelDefinition(), isModelInstalled(model) {
|
||||
do {
|
||||
return try await MacQwen3LocalASR.transcribe(
|
||||
samples: samples,
|
||||
sampleRate: 16_000,
|
||||
locale: locale,
|
||||
modelPath: MacLocalASRPreferences.qwen3ModelPath
|
||||
)
|
||||
} catch MacLocalASRError.qwen3ModelMissing {
|
||||
// Fall through to Apple Speech when weights are absent.
|
||||
return try await transcribeWithModel(model, samples: samples, locale: locale, bias: bias)
|
||||
} catch {
|
||||
throw error
|
||||
if model.backend != .mlx {
|
||||
throw error
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if MacLocalASRPreferences.qwen3ModelIsInstalled() {
|
||||
return try await MacQwen3LocalASR.transcribe(
|
||||
samples: samples,
|
||||
sampleRate: 16_000,
|
||||
locale: locale,
|
||||
modelPath: MacLocalASRPreferences.qwen3ModelPath,
|
||||
bias: bias
|
||||
)
|
||||
}
|
||||
|
||||
return try await MacSpeechLocalASR.transcribe(samples: samples, locale: locale)
|
||||
}
|
||||
|
||||
private static func transcribeWithModel(
|
||||
_ model: LocalASRModelDefinition,
|
||||
samples: [Float],
|
||||
locale: Locale,
|
||||
bias: LocalASRBiasPayload?
|
||||
) async throws -> String {
|
||||
switch model.backend {
|
||||
case .mlx:
|
||||
return try await MacQwen3LocalASR.transcribe(
|
||||
samples: samples,
|
||||
sampleRate: 16_000,
|
||||
locale: locale,
|
||||
modelPath: MacLocalASRPreferences.qwen3ModelPath,
|
||||
bias: bias
|
||||
)
|
||||
case .sherpaQwen3, .sherpaSenseVoice:
|
||||
return try await MacSherpaLocalASR.transcribe(
|
||||
samples: samples,
|
||||
sampleRate: 16_000,
|
||||
locale: locale,
|
||||
model: model,
|
||||
bias: bias
|
||||
)
|
||||
case .appleSpeech:
|
||||
return try await MacSpeechLocalASR.transcribe(samples: samples, locale: locale)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -50,14 +50,15 @@ actor MacQwen3ASREngine {
|
||||
func transcribe(
|
||||
samples: [Float],
|
||||
language: String?,
|
||||
modelPath: String
|
||||
modelPath: String,
|
||||
context: String? = nil
|
||||
) async throws -> String {
|
||||
try await prepareIfNeeded(modelPath: modelPath)
|
||||
guard let stt else {
|
||||
throw MacLocalASRError.qwen3LoadFailed("Engine not initialized")
|
||||
}
|
||||
|
||||
let result = try await stt.transcribe(audio: samples, language: language)
|
||||
let result = try await stt.transcribe(audio: samples, language: language, context: context)
|
||||
let text = result.text.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||
guard !text.isEmpty else {
|
||||
throw MacLocalASRError.emptyTranscript
|
||||
|
||||
@@ -12,7 +12,8 @@ enum MacQwen3LocalASR {
|
||||
samples: [Float],
|
||||
sampleRate: Int,
|
||||
locale: Locale,
|
||||
modelPath: String
|
||||
modelPath: String,
|
||||
bias: LocalASRBiasPayload? = nil
|
||||
) async throws -> String {
|
||||
guard MacLocalASRPreferences.qwen3ModelIsInstalled(at: modelPath) else {
|
||||
throw MacLocalASRError.qwen3ModelMissing
|
||||
@@ -24,11 +25,14 @@ enum MacQwen3LocalASR {
|
||||
}
|
||||
|
||||
let language = MacQwen3LanguageHint.from(locale: locale)
|
||||
let context = bias?.promptBias?.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||
let promptContext = (context?.isEmpty == false) ? context : nil
|
||||
do {
|
||||
return try await MacQwen3ASREngine.shared.transcribe(
|
||||
samples: samples,
|
||||
language: language,
|
||||
modelPath: modelPath
|
||||
modelPath: modelPath,
|
||||
context: promptContext
|
||||
)
|
||||
} catch let error as MacLocalASRError {
|
||||
throw error
|
||||
|
||||
@@ -56,7 +56,6 @@ struct MacRootView: View {
|
||||
Spacer()
|
||||
devicesFooter
|
||||
}
|
||||
.background(palette.surfaceMuted)
|
||||
}
|
||||
|
||||
private func sidebarRow(_ section: MacSection) -> some View {
|
||||
|
||||
@@ -37,7 +37,7 @@ struct MacSettingsView: View {
|
||||
providerSection
|
||||
}
|
||||
if viewModel.config.engineMode == "local" {
|
||||
qwen3Section
|
||||
MacLocalASRModelSettingsView(viewModel: viewModel)
|
||||
}
|
||||
inputSection
|
||||
syncSection
|
||||
@@ -147,12 +147,6 @@ struct MacSettingsView: View {
|
||||
systemImage: "cpu",
|
||||
selected: viewModel.config.engineMode == "local"
|
||||
) { viewModel.setEngineMode("local") }
|
||||
|
||||
if viewModel.config.engineMode == "local", !viewModel.qwen3ModelInstalled {
|
||||
Label(MacL10n.string("mac.settings.qwen3Missing", language: lang), systemImage: "exclamationmark.triangle")
|
||||
.font(TypeStyle.caption)
|
||||
.foregroundStyle(palette.warning)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -196,25 +190,7 @@ struct MacSettingsView: View {
|
||||
}
|
||||
}
|
||||
|
||||
// MARK: - Qwen3 model path
|
||||
|
||||
private var qwen3Section: some View {
|
||||
Section {
|
||||
HStack(spacing: Spacing.sm) {
|
||||
TextField("", text: qwen3PathBinding, prompt: Text(verbatim: "~/Models/Qwen3-ASR"))
|
||||
.macFieldStyle()
|
||||
Button(MacL10n.string("mac.settings.qwen3Browse", language: lang)) {
|
||||
pickQwen3Folder()
|
||||
}
|
||||
}
|
||||
} header: {
|
||||
Text(MacL10n.string("mac.settings.qwen3Model", language: lang))
|
||||
} footer: {
|
||||
Text(MacL10n.string("mac.settings.qwen3ModelDesc", language: lang))
|
||||
.font(TypeStyle.caption)
|
||||
.foregroundStyle(palette.textSecondary)
|
||||
}
|
||||
}
|
||||
// MARK: - Qwen3 model path (legacy — see MacLocalASRModelSettingsView)
|
||||
|
||||
// MARK: - Row helpers
|
||||
|
||||
@@ -366,17 +342,6 @@ struct MacSettingsView: View {
|
||||
)
|
||||
}
|
||||
|
||||
private var qwen3PathBinding: Binding<String> {
|
||||
Binding(
|
||||
get: { MacLocalASRPreferences.qwen3ModelPath },
|
||||
set: { newPath in
|
||||
MacLocalASRPreferences.qwen3ModelPath = newPath
|
||||
Task { await MacQwen3ASREngine.shared.unload() }
|
||||
viewModel.warmUpQwen3IfNeeded()
|
||||
}
|
||||
)
|
||||
}
|
||||
|
||||
// MARK: - AppKit actions (macOS only)
|
||||
|
||||
private func openAccessibilitySettings() {
|
||||
@@ -413,19 +378,4 @@ struct MacSettingsView: View {
|
||||
private var accessibilityStatusNeeded: String {
|
||||
lang.resolvedLanguageCode().hasPrefix("zh") ? "未授权" : "Needed"
|
||||
}
|
||||
|
||||
private func pickQwen3Folder() {
|
||||
#if os(macOS)
|
||||
let panel = NSOpenPanel()
|
||||
panel.canChooseDirectories = true
|
||||
panel.canChooseFiles = false
|
||||
panel.allowsMultipleSelection = false
|
||||
panel.begin { response in
|
||||
guard response == .OK, let url = panel.url else { return }
|
||||
MacLocalASRPreferences.qwen3ModelPath = url.path
|
||||
Task { await MacQwen3ASREngine.shared.unload() }
|
||||
viewModel.warmUpQwen3IfNeeded()
|
||||
}
|
||||
#endif
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
// MacSherpaLocalASR.swift
|
||||
// OSGKeyboard · Mac
|
||||
//
|
||||
// Sherpa-onnx backed local ASR (Qwen3 hotwords POC + SenseVoice baseline).
|
||||
|
||||
import Foundation
|
||||
|
||||
enum MacSherpaLocalASR {
|
||||
|
||||
static func transcribe(
|
||||
samples: [Float],
|
||||
sampleRate: Int,
|
||||
locale: Locale,
|
||||
model: LocalASRModelDefinition,
|
||||
bias: LocalASRBiasPayload?
|
||||
) async throws -> String {
|
||||
let catalog = try LocalASRModelCatalog.loadBundled()
|
||||
let manager = LocalASRModelManager.shared
|
||||
guard let layout = model.layout,
|
||||
let modelRoot = LocalASRModelInstallState.modelRootURL(model) else {
|
||||
throw MacLocalASRError.qwen3ModelMissing
|
||||
}
|
||||
|
||||
try await manager.ensureRuntimeInstalled(catalog: catalog)
|
||||
guard let runtime = LocalASRModelCatalog.runtime(
|
||||
for: LocalASRModelCatalog.currentRuntimePlatform(),
|
||||
in: catalog
|
||||
),
|
||||
let binary = LocalASRModelInstallState.resolveRuntimeBinary(runtime: runtime) else {
|
||||
throw MacLocalASRError.qwen3LoadFailed("Sherpa runtime binary missing")
|
||||
}
|
||||
|
||||
switch model.backend {
|
||||
case .sherpaQwen3:
|
||||
return try await MacSherpaONNXRunner.transcribeQwen3(
|
||||
samples: samples,
|
||||
sampleRate: sampleRate,
|
||||
locale: locale,
|
||||
modelRoot: modelRoot,
|
||||
layout: layout,
|
||||
runtimeBinary: binary,
|
||||
bias: bias
|
||||
)
|
||||
case .sherpaSenseVoice:
|
||||
return try await MacSherpaONNXRunner.transcribeSenseVoice(
|
||||
samples: samples,
|
||||
sampleRate: sampleRate,
|
||||
modelRoot: modelRoot,
|
||||
layout: layout,
|
||||
runtimeBinary: binary
|
||||
)
|
||||
default:
|
||||
throw MacLocalASRError.qwen3InferenceFailed("Unsupported Sherpa backend")
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,153 @@
|
||||
// MacSherpaONNXRunner.swift
|
||||
// OSGKeyboard · Mac
|
||||
//
|
||||
// Invokes the downloaded `sherpa-onnx-offline` binary for Sherpa-backed POC models.
|
||||
|
||||
import Foundation
|
||||
|
||||
enum MacSherpaONNXRunner {
|
||||
|
||||
static func transcribeQwen3(
|
||||
samples: [Float],
|
||||
sampleRate: Int,
|
||||
locale: Locale,
|
||||
modelRoot: URL,
|
||||
layout: LocalASRModelLayout,
|
||||
runtimeBinary: URL,
|
||||
bias: LocalASRBiasPayload?
|
||||
) async throws -> String {
|
||||
guard sampleRate == 16_000 else {
|
||||
throw MacLocalASRError.qwen3InferenceFailed("Sherpa expects 16 kHz audio")
|
||||
}
|
||||
guard let conv = layout.convFrontend,
|
||||
let encoder = layout.encoder,
|
||||
let decoder = layout.decoder,
|
||||
let tokenizer = layout.tokenizer else {
|
||||
throw MacLocalASRError.qwen3InferenceFailed("Incomplete Sherpa Qwen3 layout")
|
||||
}
|
||||
|
||||
let wavURL = try writeTemporaryWAV(samples: samples, sampleRate: sampleRate)
|
||||
defer { try? FileManager.default.removeItem(at: wavURL) }
|
||||
|
||||
var arguments = [
|
||||
"--qwen3-asr-conv-frontend=\(modelRoot.appendingPathComponent(conv).path)",
|
||||
"--qwen3-asr-encoder=\(modelRoot.appendingPathComponent(encoder).path)",
|
||||
"--qwen3-asr-decoder=\(modelRoot.appendingPathComponent(decoder).path)",
|
||||
"--qwen3-asr-tokenizer=\(modelRoot.appendingPathComponent(tokenizer).path)",
|
||||
"--qwen3-asr-max-new-tokens=512",
|
||||
"--num-threads=2",
|
||||
]
|
||||
|
||||
if let language = MacQwen3LanguageHint.from(locale: locale) {
|
||||
arguments.append("--qwen3-asr-language=\(language)")
|
||||
}
|
||||
|
||||
if let hotwords = bias?.hardHotwords, !hotwords.isEmpty {
|
||||
arguments.append("--qwen3-asr-hotwords=\(hotwords.joined(separator: ","))")
|
||||
}
|
||||
|
||||
arguments.append(wavURL.path)
|
||||
return try await run(binary: runtimeBinary, arguments: arguments)
|
||||
}
|
||||
|
||||
static func transcribeSenseVoice(
|
||||
samples: [Float],
|
||||
sampleRate: Int,
|
||||
modelRoot: URL,
|
||||
layout: LocalASRModelLayout,
|
||||
runtimeBinary: URL
|
||||
) async throws -> String {
|
||||
guard sampleRate == 16_000 else {
|
||||
throw MacLocalASRError.qwen3InferenceFailed("Sherpa expects 16 kHz audio")
|
||||
}
|
||||
guard let model = layout.senseVoiceModel,
|
||||
let tokens = layout.tokens else {
|
||||
throw MacLocalASRError.qwen3InferenceFailed("Incomplete SenseVoice layout")
|
||||
}
|
||||
|
||||
let wavURL = try writeTemporaryWAV(samples: samples, sampleRate: sampleRate)
|
||||
defer { try? FileManager.default.removeItem(at: wavURL) }
|
||||
|
||||
let arguments = [
|
||||
"--tokens=\(modelRoot.appendingPathComponent(tokens).path)",
|
||||
"--sense-voice-model=\(modelRoot.appendingPathComponent(model).path)",
|
||||
"--num-threads=2",
|
||||
wavURL.path,
|
||||
]
|
||||
return try await run(binary: runtimeBinary, arguments: arguments)
|
||||
}
|
||||
|
||||
// MARK: - Private
|
||||
|
||||
private static func writeTemporaryWAV(samples: [Float], sampleRate: Int) throws -> URL {
|
||||
let data = PCMSampleWavEncoder.encode(samples: samples, sampleRate: sampleRate)
|
||||
let url = FileManager.default.temporaryDirectory
|
||||
.appendingPathComponent("osg-sherpa-\(UUID().uuidString).wav")
|
||||
try data.write(to: url, options: .atomic)
|
||||
return url
|
||||
}
|
||||
|
||||
private static func run(binary: URL, arguments: [String]) async throws -> String {
|
||||
try await withCheckedThrowingContinuation { continuation in
|
||||
let process = Process()
|
||||
process.executableURL = binary
|
||||
process.arguments = arguments
|
||||
process.currentDirectoryURL = binary.deletingLastPathComponent()
|
||||
|
||||
let outputPipe = Pipe()
|
||||
let errorPipe = Pipe()
|
||||
process.standardOutput = outputPipe
|
||||
process.standardError = errorPipe
|
||||
|
||||
process.terminationHandler = { proc in
|
||||
let outputData = outputPipe.fileHandleForReading.readDataToEndOfFile()
|
||||
let errorData = errorPipe.fileHandleForReading.readDataToEndOfFile()
|
||||
let stdout = String(data: outputData, encoding: .utf8) ?? ""
|
||||
let stderr = String(data: errorData, encoding: .utf8) ?? ""
|
||||
|
||||
guard proc.terminationStatus == 0 else {
|
||||
let detail = stderr.isEmpty ? stdout : stderr
|
||||
continuation.resume(
|
||||
throwing: MacLocalASRError.qwen3InferenceFailed(
|
||||
detail.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||
)
|
||||
)
|
||||
return
|
||||
}
|
||||
|
||||
let text = parseTranscript(stdout: stdout)
|
||||
if text.isEmpty {
|
||||
continuation.resume(throwing: MacLocalASRError.emptyTranscript)
|
||||
} else {
|
||||
continuation.resume(returning: text)
|
||||
}
|
||||
}
|
||||
|
||||
do {
|
||||
try process.run()
|
||||
} catch {
|
||||
continuation.resume(throwing: MacLocalASRError.qwen3InferenceFailed(error.localizedDescription))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private static func parseTranscript(stdout: String) -> String {
|
||||
let lines = stdout
|
||||
.split(whereSeparator: \.isNewline)
|
||||
.map { String($0).trimmingCharacters(in: .whitespacesAndNewlines) }
|
||||
.filter { !$0.isEmpty }
|
||||
|
||||
for line in lines.reversed() {
|
||||
if line.hasPrefix("{"), let data = line.data(using: .utf8),
|
||||
let object = try? JSONSerialization.jsonObject(with: data) as? [String: Any],
|
||||
let text = object["text"] as? String {
|
||||
let trimmed = text.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||
if !trimmed.isEmpty { return trimmed }
|
||||
}
|
||||
if !line.hasPrefix("/"), !line.hasPrefix("--"), line.count > 1 {
|
||||
return line
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
}
|
||||
@@ -3,7 +3,7 @@
|
||||
<plist version="1.0">
|
||||
<dict>
|
||||
<key>com.apple.security.app-sandbox</key>
|
||||
<true/>
|
||||
<false/>
|
||||
<key>com.apple.security.device.audio-input</key>
|
||||
<true/>
|
||||
<key>com.apple.security.network.client</key>
|
||||
|
||||
@@ -86,6 +86,20 @@ final class MacAppDelegate: NSObject, NSApplicationDelegate {
|
||||
MacAppearancePreference.applyToApp(.current)
|
||||
configurePopover()
|
||||
configureStatusItem()
|
||||
|
||||
// The menu bar always follows the *system* appearance, so the status
|
||||
// item must ignore the app's forced light/dark override. Re-pin the
|
||||
// button appearance whenever the system theme flips.
|
||||
DistributedNotificationCenter.default.addObserver(
|
||||
self,
|
||||
selector: #selector(systemAppearanceDidChange),
|
||||
name: NSNotification.Name("AppleInterfaceThemeChangedNotification"),
|
||||
object: nil
|
||||
)
|
||||
}
|
||||
|
||||
deinit {
|
||||
DistributedNotificationCenter.default.removeObserver(self)
|
||||
}
|
||||
|
||||
/// Keep the app alive after the last window closes — it lives in the menu bar.
|
||||
@@ -96,18 +110,47 @@ final class MacAppDelegate: NSObject, NSApplicationDelegate {
|
||||
private func configureStatusItem() {
|
||||
let item = NSStatusBar.system.statusItem(withLength: NSStatusItem.variableLength)
|
||||
if let button = item.button {
|
||||
// Prefer the brand mark; fall back to an SF Symbol so the item is
|
||||
// never invisible even if the asset fails to resolve.
|
||||
let image = NSImage(named: "OSGBrandMark")
|
||||
?? NSImage(systemSymbolName: "mic.circle.fill", accessibilityDescription: "OSGKeyboard")
|
||||
image?.isTemplate = true
|
||||
image?.size = NSSize(width: 18, height: 18)
|
||||
button.image = image
|
||||
button.image = Self.makeStatusBarImage()
|
||||
button.image?.accessibilityDescription = "OSGKeyboard"
|
||||
button.action = #selector(togglePopover(_:))
|
||||
button.target = self
|
||||
}
|
||||
statusItem = item
|
||||
applyStatusItemAppearance()
|
||||
}
|
||||
|
||||
/// Builds the menu-bar glyph from the dedicated horizontal status mark.
|
||||
/// Height is pinned to the tallest practical menu-bar slot so the logo reads
|
||||
/// clearly; width follows the asset's aspect ratio.
|
||||
private static func makeStatusBarImage() -> NSImage? {
|
||||
guard let image = NSImage(named: "OSGStatusMark")
|
||||
?? NSImage(named: "OSGBrandMark")
|
||||
?? NSImage(systemSymbolName: "mic.circle.fill", accessibilityDescription: "OSGKeyboard") else {
|
||||
return nil
|
||||
}
|
||||
let height: CGFloat = 9
|
||||
let aspect = max(image.size.width / max(image.size.height, 1), 1)
|
||||
image.size = NSSize(width: height * aspect, height: height)
|
||||
image.isTemplate = true
|
||||
return image
|
||||
}
|
||||
|
||||
/// Pins the status-bar button to the current *system* appearance so its
|
||||
/// template image tint matches the real menu-bar background — regardless of
|
||||
/// the in-app light/dark preference forced on `NSApp.appearance`.
|
||||
private func applyStatusItemAppearance() {
|
||||
guard let button = statusItem?.button else { return }
|
||||
let isDark = UserDefaults.standard.string(forKey: "AppleInterfaceStyle")?
|
||||
.lowercased().contains("dark") ?? false
|
||||
button.appearance = NSAppearance(named: isDark ? .darkAqua : .aqua)
|
||||
}
|
||||
|
||||
@objc private func systemAppearanceDidChange() {
|
||||
// The global-domain default lags the notification by a hair; hop to the
|
||||
// next runloop tick so `AppleInterfaceStyle` reflects the new value.
|
||||
DispatchQueue.main.async { [weak self] in
|
||||
self?.applyStatusItemAppearance()
|
||||
}
|
||||
}
|
||||
|
||||
private func configurePopover() {
|
||||
|
||||
@@ -0,0 +1,99 @@
|
||||
// LocalASRBiasPayload.swift
|
||||
// OSGKeyboard · Shared
|
||||
//
|
||||
// Output of `LocalASRBiasAdapter` — vocabulary signals for each pipeline layer.
|
||||
|
||||
import Foundation
|
||||
|
||||
public struct LocalASRCorrectionPair: Sendable, Equatable {
|
||||
public let alias: String
|
||||
public let term: String
|
||||
|
||||
public init(alias: String, term: String) {
|
||||
self.alias = alias
|
||||
self.term = term
|
||||
}
|
||||
}
|
||||
|
||||
public struct LocalASRBiasDiagnostics: Sendable, Equatable, Codable {
|
||||
public var userTermCount: Int
|
||||
public var builtinTermCount: Int
|
||||
public var truncated: Bool
|
||||
public var truncationReason: String?
|
||||
public var selectedSources: [String]
|
||||
|
||||
public init(
|
||||
userTermCount: Int = 0,
|
||||
builtinTermCount: Int = 0,
|
||||
truncated: Bool = false,
|
||||
truncationReason: String? = nil,
|
||||
selectedSources: [String] = []
|
||||
) {
|
||||
self.userTermCount = userTermCount
|
||||
self.builtinTermCount = builtinTermCount
|
||||
self.truncated = truncated
|
||||
self.truncationReason = truncationReason
|
||||
self.selectedSources = selectedSources
|
||||
}
|
||||
}
|
||||
|
||||
public struct LocalASRBiasPayload: Sendable, Equatable {
|
||||
public var hardHotwords: [String]
|
||||
public var promptBias: String?
|
||||
public var corpusContext: String?
|
||||
public var polishFragment: String
|
||||
public var correctionPairs: [LocalASRCorrectionPair]
|
||||
public var diagnostics: LocalASRBiasDiagnostics
|
||||
|
||||
public static let empty = LocalASRBiasPayload(
|
||||
hardHotwords: [],
|
||||
promptBias: nil,
|
||||
corpusContext: nil,
|
||||
polishFragment: "",
|
||||
correctionPairs: [],
|
||||
diagnostics: LocalASRBiasDiagnostics()
|
||||
)
|
||||
|
||||
public init(
|
||||
hardHotwords: [String],
|
||||
promptBias: String?,
|
||||
corpusContext: String?,
|
||||
polishFragment: String,
|
||||
correctionPairs: [LocalASRCorrectionPair],
|
||||
diagnostics: LocalASRBiasDiagnostics
|
||||
) {
|
||||
self.hardHotwords = hardHotwords
|
||||
self.promptBias = promptBias
|
||||
self.corpusContext = corpusContext
|
||||
self.polishFragment = polishFragment
|
||||
self.correctionPairs = correctionPairs
|
||||
self.diagnostics = diagnostics
|
||||
}
|
||||
}
|
||||
|
||||
public struct LocalASRBiasRequest: Sendable {
|
||||
public var dictionary: PersonalDictionary
|
||||
public var locale: Locale
|
||||
public var frontAppBundleId: String?
|
||||
public var capabilities: LocalASRCapabilities
|
||||
/// Max builtin `phrases.tsv` terms considered for ASR bias (not polish-only).
|
||||
public var builtinASRLimit: Int
|
||||
/// Max builtin terms referenced in the polish supplement block.
|
||||
public var builtinPolishLimit: Int
|
||||
|
||||
public init(
|
||||
dictionary: PersonalDictionary,
|
||||
locale: Locale,
|
||||
frontAppBundleId: String? = nil,
|
||||
capabilities: LocalASRCapabilities,
|
||||
builtinASRLimit: Int = 300,
|
||||
builtinPolishLimit: Int = 40
|
||||
) {
|
||||
self.dictionary = dictionary
|
||||
self.locale = locale
|
||||
self.frontAppBundleId = frontAppBundleId
|
||||
self.capabilities = capabilities
|
||||
self.builtinASRLimit = builtinASRLimit
|
||||
self.builtinPolishLimit = builtinPolishLimit
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,82 @@
|
||||
// LocalASRCapabilities.swift
|
||||
// OSGKeyboard · Shared
|
||||
//
|
||||
// Declares what each on-device ASR backend can accept for vocabulary bias.
|
||||
// Callers must consult capabilities before building a `LocalASRBiasPayload`.
|
||||
|
||||
import Foundation
|
||||
|
||||
/// How a backend accepts vocabulary hints (honest matrix — not every model
|
||||
/// supports hard hotwords).
|
||||
public enum LocalASRHotwordMode: String, Sendable, Codable, Equatable {
|
||||
case none
|
||||
case promptOnly
|
||||
case perRequest
|
||||
case recognizerScoped
|
||||
case cloudVocabulary
|
||||
}
|
||||
|
||||
/// Cost of refreshing hotwords on a backend (e.g. Sherpa Qwen3 reloads recognizer).
|
||||
public enum LocalASRHotwordReloadCost: String, Sendable, Codable, Equatable {
|
||||
case none
|
||||
case recognizerReload
|
||||
case modelReload
|
||||
}
|
||||
|
||||
public struct LocalASRCapabilities: Sendable, Equatable {
|
||||
public let hotwordMode: LocalASRHotwordMode
|
||||
public let maxHotwordCount: Int
|
||||
public let maxPromptCharacters: Int
|
||||
public let supportsStreaming: Bool
|
||||
public let hotwordReloadCost: LocalASRHotwordReloadCost
|
||||
|
||||
public init(
|
||||
hotwordMode: LocalASRHotwordMode,
|
||||
maxHotwordCount: Int,
|
||||
maxPromptCharacters: Int,
|
||||
supportsStreaming: Bool,
|
||||
hotwordReloadCost: LocalASRHotwordReloadCost
|
||||
) {
|
||||
self.hotwordMode = hotwordMode
|
||||
self.maxHotwordCount = maxHotwordCount
|
||||
self.maxPromptCharacters = maxPromptCharacters
|
||||
self.supportsStreaming = supportsStreaming
|
||||
self.hotwordReloadCost = hotwordReloadCost
|
||||
}
|
||||
|
||||
/// Qwen3 MLX via mlx-swift-asr — `context` soft prompt on `transcribe`.
|
||||
public static let qwen3MLX = LocalASRCapabilities(
|
||||
hotwordMode: .promptOnly,
|
||||
maxHotwordCount: 0,
|
||||
maxPromptCharacters: 800,
|
||||
supportsStreaming: false,
|
||||
hotwordReloadCost: .none
|
||||
)
|
||||
|
||||
/// Apple Speech on macOS — no project-controlled hotword API today.
|
||||
public static let appleSpeech = LocalASRCapabilities(
|
||||
hotwordMode: .none,
|
||||
maxHotwordCount: 0,
|
||||
maxPromptCharacters: 0,
|
||||
supportsStreaming: false,
|
||||
hotwordReloadCost: .none
|
||||
)
|
||||
|
||||
/// Sherpa Qwen3 — hard hotwords via `--qwen3-asr-hotwords`.
|
||||
public static let sherpaQwen3 = LocalASRCapabilities(
|
||||
hotwordMode: .recognizerScoped,
|
||||
maxHotwordCount: 100,
|
||||
maxPromptCharacters: 0,
|
||||
supportsStreaming: false,
|
||||
hotwordReloadCost: .recognizerReload
|
||||
)
|
||||
|
||||
/// Sherpa SenseVoice — fast Chinese baseline without hotwords.
|
||||
public static let sherpaSenseVoice = LocalASRCapabilities(
|
||||
hotwordMode: .none,
|
||||
maxHotwordCount: 0,
|
||||
maxPromptCharacters: 0,
|
||||
supportsStreaming: false,
|
||||
hotwordReloadCost: .none
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,134 @@
|
||||
// LocalASRModelCatalog.swift
|
||||
// OSGKeyboard · Shared
|
||||
//
|
||||
// Bundled catalog of downloadable / manual local ASR models and Sherpa runtimes.
|
||||
|
||||
import Foundation
|
||||
|
||||
public enum LocalASRModelBackend: String, Codable, Sendable, Equatable {
|
||||
case mlx
|
||||
case sherpaQwen3
|
||||
case sherpaSenseVoice
|
||||
case appleSpeech
|
||||
}
|
||||
|
||||
public enum LocalASRInstallKind: String, Codable, Sendable, Equatable {
|
||||
case manual
|
||||
case archive
|
||||
case runtime
|
||||
}
|
||||
|
||||
public struct LocalASRDownloadSource: Codable, Sendable, Equatable {
|
||||
public let type: String
|
||||
public let priority: Int
|
||||
public let url: String
|
||||
}
|
||||
|
||||
public struct LocalASRModelLayout: Codable, Sendable, Equatable {
|
||||
public var convFrontend: String?
|
||||
public var encoder: String?
|
||||
public var decoder: String?
|
||||
public var tokenizer: String?
|
||||
public var senseVoiceModel: String?
|
||||
public var tokens: String?
|
||||
}
|
||||
|
||||
public struct LocalASRRuntimeDefinition: Codable, Sendable, Equatable, Identifiable {
|
||||
public let id: String
|
||||
public let displayName: String
|
||||
public let installRelativePath: String
|
||||
public let binaryCandidates: [String]
|
||||
public let archiveFileName: String
|
||||
public let sizeBytes: Int
|
||||
public let platform: String
|
||||
public let sources: [LocalASRDownloadSource]
|
||||
}
|
||||
|
||||
public struct LocalASRModelDefinition: Codable, Sendable, Equatable, Identifiable {
|
||||
public let id: String
|
||||
public let displayName: String
|
||||
public let backend: LocalASRModelBackend
|
||||
public let sizeBytes: Int
|
||||
public let recommendedLocales: [String]
|
||||
public let supportsHotwords: Bool
|
||||
public let hotwordMode: LocalASRHotwordMode
|
||||
public let installKind: LocalASRInstallKind
|
||||
public let installRelativePath: String?
|
||||
public let archiveBaseName: String?
|
||||
public let layout: LocalASRModelLayout?
|
||||
public let requiredRelativeFiles: [String]?
|
||||
public let runtimePlatform: String?
|
||||
public let sources: [LocalASRDownloadSource]?
|
||||
}
|
||||
|
||||
public struct LocalASRCatalogDocument: Codable, Sendable, Equatable {
|
||||
public let schemaVersion: Int
|
||||
public let defaultModelId: String
|
||||
public let runtimes: [LocalASRRuntimeDefinition]
|
||||
public let models: [LocalASRModelDefinition]
|
||||
}
|
||||
|
||||
public enum LocalASRModelCatalog {
|
||||
|
||||
public static func loadBundled() throws -> LocalASRCatalogDocument {
|
||||
let bundle = Bundle(for: LocalASRCatalogBundleToken.self)
|
||||
guard let url = bundle.url(forResource: "local-asr-catalog", withExtension: "json") else {
|
||||
throw LocalASRModelCatalogError.missingBundledCatalog
|
||||
}
|
||||
let data = try Data(contentsOf: url)
|
||||
return try JSONDecoder().decode(LocalASRCatalogDocument.self, from: data)
|
||||
}
|
||||
|
||||
public static func model(_ id: String, in catalog: LocalASRCatalogDocument) -> LocalASRModelDefinition? {
|
||||
catalog.models.first { $0.id == id }
|
||||
}
|
||||
|
||||
public static func capabilities(for model: LocalASRModelDefinition) -> LocalASRCapabilities {
|
||||
switch model.backend {
|
||||
case .mlx:
|
||||
return .qwen3MLX
|
||||
case .sherpaQwen3:
|
||||
return .sherpaQwen3
|
||||
case .sherpaSenseVoice:
|
||||
return .sherpaSenseVoice
|
||||
case .appleSpeech:
|
||||
return .appleSpeech
|
||||
}
|
||||
}
|
||||
|
||||
#if os(macOS)
|
||||
public static func runtime(for platform: String, in catalog: LocalASRCatalogDocument) -> LocalASRRuntimeDefinition? {
|
||||
if platform == "macos-arm64" {
|
||||
return catalog.runtimes.first { $0.platform == "macos-arm64" }
|
||||
}
|
||||
if platform == "macos-x64" {
|
||||
return catalog.runtimes.first { $0.platform == "macos-x64" }
|
||||
}
|
||||
return catalog.runtimes.first
|
||||
}
|
||||
|
||||
public static func currentRuntimePlatform() -> String {
|
||||
#if arch(arm64)
|
||||
return "macos-arm64"
|
||||
#else
|
||||
return "macos-x64"
|
||||
#endif
|
||||
}
|
||||
#endif
|
||||
}
|
||||
|
||||
public enum LocalASRModelCatalogError: Error, LocalizedError {
|
||||
case missingBundledCatalog
|
||||
case modelNotFound(String)
|
||||
|
||||
public var errorDescription: String? {
|
||||
switch self {
|
||||
case .missingBundledCatalog:
|
||||
return "Missing bundled local ASR catalog."
|
||||
case .modelNotFound(let id):
|
||||
return "Local ASR model not found: \(id)"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private final class LocalASRCatalogBundleToken {}
|
||||
@@ -111,4 +111,29 @@ extension PersonalDictionary {
|
||||
if hasNonASCII { return "zh" }
|
||||
return "en"
|
||||
}
|
||||
|
||||
/// Alias → canonical term pairs for deterministic post-ASR correction.
|
||||
/// Sorted longest-alias-first by the caller (`LocalASRTranscriptCorrector`).
|
||||
public func localCorrectionPairs() -> [LocalASRCorrectionPair] {
|
||||
var seen = Set<String>()
|
||||
var pairs: [LocalASRCorrectionPair] = []
|
||||
for entry in effectiveEntries {
|
||||
let term = entry.term.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||
guard !term.isEmpty else { continue }
|
||||
for alias in entry.aliases {
|
||||
let trimmed = alias.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||
guard !trimmed.isEmpty else { continue }
|
||||
guard trimmed.caseInsensitiveCompare(term) != .orderedSame else { continue }
|
||||
let key = "\(trimmed.lowercased())|\(term.lowercased())"
|
||||
guard seen.insert(key).inserted else { continue }
|
||||
pairs.append(LocalASRCorrectionPair(alias: trimmed, term: term))
|
||||
}
|
||||
}
|
||||
return pairs.sorted { lhs, rhs in
|
||||
if lhs.alias.count != rhs.alias.count {
|
||||
return lhs.alias.count > rhs.alias.count
|
||||
}
|
||||
return lhs.alias.localizedCaseInsensitiveCompare(rhs.alias) == .orderedAscending
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -24,6 +24,10 @@ public struct PolishContext: Sendable {
|
||||
/// bias terminology choices.
|
||||
public let precedingText: String?
|
||||
|
||||
/// Extra dictionary block appended after `PersonalDictionary.promptFragment()`
|
||||
/// (e.g. builtin `phrases.tsv` terms on macOS local ASR).
|
||||
public let dictionarySupplement: String?
|
||||
|
||||
/// Cap on how many characters of `precedingText` we actually
|
||||
/// include in the prompt. The full preceding text is often
|
||||
/// hundreds of KB in a long note — we only need the tail.
|
||||
@@ -33,11 +37,13 @@ public struct PolishContext: Sendable {
|
||||
appContext: AppContext = .unknown,
|
||||
intensity: PolishIntensity = .default,
|
||||
precedingText: String? = nil,
|
||||
dictionarySupplement: String? = nil,
|
||||
maxPrecedingChars: Int = 500
|
||||
) {
|
||||
self.appContext = appContext
|
||||
self.intensity = intensity
|
||||
self.precedingText = precedingText
|
||||
self.dictionarySupplement = dictionarySupplement
|
||||
self.maxPrecedingChars = maxPrecedingChars
|
||||
}
|
||||
|
||||
|
||||
@@ -74,7 +74,7 @@ public final class ProviderConfig: ObservableObject, @unchecked Sendable {
|
||||
guard !isApplyingConfiguration, engineMode != configuration.engineMode else { return }
|
||||
configuration.engineMode = engineMode
|
||||
applyEngineModeSideEffects()
|
||||
persistConfiguration()
|
||||
persistConfiguration(postConfigChanged: true)
|
||||
}
|
||||
}
|
||||
@Published public var hasCompletedOnboarding: Bool {
|
||||
|
||||
@@ -0,0 +1,102 @@
|
||||
{
|
||||
"schemaVersion": 1,
|
||||
"defaultModelId": "qwen3-mlx-1.7b",
|
||||
"runtimes": [
|
||||
{
|
||||
"id": "sherpa-onnx-1.13.4-macos-arm64",
|
||||
"displayName": "sherpa-onnx 1.13.4 (Apple Silicon)",
|
||||
"installRelativePath": "runtimes/sherpa-onnx-1.13.4-macos-arm64",
|
||||
"binaryCandidates": ["bin/sherpa-onnx-offline", "sherpa-onnx-offline"],
|
||||
"archiveFileName": "sherpa-onnx-v1.13.4-osx-arm64-static-no-tts.tar.bz2",
|
||||
"sizeBytes": 120000000,
|
||||
"platform": "macos-arm64",
|
||||
"sources": [
|
||||
{
|
||||
"type": "github",
|
||||
"priority": 1,
|
||||
"url": "https://github.com/k2-fsa/sherpa-onnx/releases/download/v1.13.4/sherpa-onnx-v1.13.4-osx-arm64-static-no-tts.tar.bz2"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "sherpa-onnx-1.13.4-macos-x64",
|
||||
"displayName": "sherpa-onnx 1.13.4 (Intel)",
|
||||
"installRelativePath": "runtimes/sherpa-onnx-1.13.4-macos-x64",
|
||||
"binaryCandidates": ["bin/sherpa-onnx-offline", "sherpa-onnx-offline"],
|
||||
"archiveFileName": "sherpa-onnx-v1.13.4-osx-x64-static-no-tts.tar.bz2",
|
||||
"sizeBytes": 130000000,
|
||||
"platform": "macos-x64",
|
||||
"sources": [
|
||||
{
|
||||
"type": "github",
|
||||
"priority": 1,
|
||||
"url": "https://github.com/k2-fsa/sherpa-onnx/releases/download/v1.13.4/sherpa-onnx-v1.13.4-osx-x64-static-no-tts.tar.bz2"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"models": [
|
||||
{
|
||||
"id": "qwen3-mlx-1.7b",
|
||||
"displayName": "Qwen3-ASR 1.7B (MLX)",
|
||||
"backend": "mlx",
|
||||
"sizeBytes": 1400000000,
|
||||
"recommendedLocales": ["zh-CN", "en-US"],
|
||||
"supportsHotwords": true,
|
||||
"hotwordMode": "promptOnly",
|
||||
"installKind": "manual",
|
||||
"installRelativePath": "models/qwen3-asr-1.7b-mlx",
|
||||
"requiredRelativeFiles": ["config.json", "model.safetensors", "vocab.json", "merges.txt"]
|
||||
},
|
||||
{
|
||||
"id": "sherpa-qwen3-0.6b-int8",
|
||||
"displayName": "Qwen3-ASR 0.6B (Sherpa · hotwords)",
|
||||
"backend": "sherpaQwen3",
|
||||
"runtimePlatform": "macos",
|
||||
"sizeBytes": 650000000,
|
||||
"recommendedLocales": ["zh-CN", "en-US"],
|
||||
"supportsHotwords": true,
|
||||
"hotwordMode": "recognizerScoped",
|
||||
"installKind": "archive",
|
||||
"installRelativePath": "models/sherpa-qwen3-0.6b-int8",
|
||||
"archiveBaseName": "sherpa-onnx-qwen3-asr-0.6B-int8-2026-03-25",
|
||||
"layout": {
|
||||
"convFrontend": "conv_frontend.onnx",
|
||||
"encoder": "encoder.int8.onnx",
|
||||
"decoder": "decoder.int8.onnx",
|
||||
"tokenizer": "tokenizer"
|
||||
},
|
||||
"sources": [
|
||||
{
|
||||
"type": "github",
|
||||
"priority": 1,
|
||||
"url": "https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-qwen3-asr-0.6B-int8-2026-03-25.tar.bz2"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "sherpa-sensevoice-small-int8",
|
||||
"displayName": "SenseVoice Small (Sherpa)",
|
||||
"backend": "sherpaSenseVoice",
|
||||
"runtimePlatform": "macos",
|
||||
"sizeBytes": 250000000,
|
||||
"recommendedLocales": ["zh-CN", "en-US", "ja-JP", "ko-KR"],
|
||||
"supportsHotwords": false,
|
||||
"hotwordMode": "none",
|
||||
"installKind": "archive",
|
||||
"installRelativePath": "models/sherpa-sensevoice-small-int8",
|
||||
"archiveBaseName": "sherpa-onnx-sense-voice-zh-en-ja-ko-yue-int8-2024-07-17",
|
||||
"layout": {
|
||||
"senseVoiceModel": "model.int8.onnx",
|
||||
"tokens": "tokens.txt"
|
||||
},
|
||||
"sources": [
|
||||
{
|
||||
"type": "github",
|
||||
"priority": 1,
|
||||
"url": "https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-sense-voice-zh-en-ja-ko-yue-int8-2024-07-17.tar.bz2"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -92,6 +92,7 @@ public struct AppGroupStore: @unchecked Sendable {
|
||||
config.model = openAI.defaultModel
|
||||
}
|
||||
}
|
||||
AppGroupConfigDarwin.postConfigChanged()
|
||||
}
|
||||
|
||||
public func setUILanguage(_ language: AppUILanguage) {
|
||||
|
||||
@@ -0,0 +1,152 @@
|
||||
// BuiltinLexiconIndex.swift
|
||||
// OSGKeyboard · Shared
|
||||
//
|
||||
// In-memory index over bundled `phrases.tsv` (~10k computer terms).
|
||||
// macOS local ASR consumes a Top-N subset; the full index also backs
|
||||
// polish supplements and future retrieval.
|
||||
|
||||
import Foundation
|
||||
|
||||
public final class BuiltinLexiconIndex: @unchecked Sendable {
|
||||
|
||||
public struct Term: Sendable, Equatable {
|
||||
public let word: String
|
||||
public let pinyin: String
|
||||
public let source: String
|
||||
public let weight: Int
|
||||
}
|
||||
|
||||
public static let shared = BuiltinLexiconIndex()
|
||||
|
||||
private let lock = NSLock()
|
||||
private var cachedTerms: [Term]?
|
||||
private let injectedURL: URL?
|
||||
|
||||
/// Production singleton loads from the app bundle.
|
||||
private init() {
|
||||
injectedURL = nil
|
||||
}
|
||||
|
||||
/// Test / preview hook with an explicit TSV file or inline fixture.
|
||||
init(fixtureURL: URL) {
|
||||
injectedURL = fixtureURL
|
||||
}
|
||||
|
||||
/// Parse TSV content without touching the bundle (unit tests).
|
||||
public static func parseTSV(_ content: String) -> [Term] {
|
||||
var terms: [Term] = []
|
||||
terms.reserveCapacity(256)
|
||||
|
||||
for (lineIndex, line) in content.split(whereSeparator: \.isNewline).enumerated() {
|
||||
if lineIndex == 0, line.hasPrefix("word\t") { continue }
|
||||
let columns = line.split(separator: "\t", omittingEmptySubsequences: false)
|
||||
guard columns.count >= 4 else { continue }
|
||||
let word = String(columns[0]).trimmingCharacters(in: .whitespacesAndNewlines)
|
||||
guard !word.isEmpty else { continue }
|
||||
let pinyin = String(columns[1])
|
||||
let source = String(columns[2])
|
||||
let weight = Int(columns[3]) ?? 1
|
||||
terms.append(Term(word: word, pinyin: pinyin, source: source, weight: weight))
|
||||
}
|
||||
return terms
|
||||
}
|
||||
|
||||
public func termCount() -> Int {
|
||||
lock.lock()
|
||||
defer { lock.unlock() }
|
||||
return loadTermsLocked().count
|
||||
}
|
||||
|
||||
/// Returns canonical words ranked for ASR bias injection.
|
||||
public func topTerms(
|
||||
limit: Int,
|
||||
minimumWeight: Int = 4,
|
||||
preferredSources: Set<String>? = nil
|
||||
) -> [String] {
|
||||
guard limit > 0 else { return [] }
|
||||
|
||||
lock.lock()
|
||||
let all = loadTermsLocked()
|
||||
lock.unlock()
|
||||
|
||||
let filtered = all.filter { term in
|
||||
guard term.weight >= minimumWeight else { return false }
|
||||
if let preferredSources, !preferredSources.isEmpty {
|
||||
return preferredSources.contains(term.source)
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
let ranked = filtered.sorted { lhs, rhs in
|
||||
let leftScore = Self.rankingScore(lhs)
|
||||
let rightScore = Self.rankingScore(rhs)
|
||||
if leftScore != rightScore { return leftScore > rightScore }
|
||||
return lhs.word.localizedCaseInsensitiveCompare(rhs.word) == .orderedAscending
|
||||
}
|
||||
|
||||
var seen = Set<String>()
|
||||
var words: [String] = []
|
||||
words.reserveCapacity(min(limit, ranked.count))
|
||||
for term in ranked {
|
||||
let key = term.word.lowercased()
|
||||
guard seen.insert(key).inserted else { continue }
|
||||
words.append(term.word)
|
||||
if words.count >= limit { break }
|
||||
}
|
||||
return words
|
||||
}
|
||||
|
||||
// MARK: - Private
|
||||
|
||||
private func loadTermsLocked() -> [Term] {
|
||||
if let cachedTerms { return cachedTerms }
|
||||
let loaded: [Term]
|
||||
if let injectedURL {
|
||||
loaded = Self.load(from: injectedURL)
|
||||
} else if let url = Self.locateBundledPhrasesURL() {
|
||||
loaded = Self.load(from: url)
|
||||
} else {
|
||||
loaded = []
|
||||
}
|
||||
cachedTerms = loaded
|
||||
return loaded
|
||||
}
|
||||
|
||||
private static func load(from url: URL) -> [Term] {
|
||||
guard let data = try? Data(contentsOf: url),
|
||||
let content = String(data: data, encoding: .utf8) else {
|
||||
return []
|
||||
}
|
||||
return parseTSV(content)
|
||||
}
|
||||
|
||||
private static func locateBundledPhrasesURL() -> URL? {
|
||||
let candidates: [Bundle] = [Bundle.main, Bundle(for: BuiltinLexiconIndex.self)]
|
||||
for bundle in candidates {
|
||||
if let url = bundle.url(
|
||||
forResource: "phrases",
|
||||
withExtension: "tsv",
|
||||
subdirectory: "CustomLanguageModel/v1"
|
||||
) {
|
||||
return url
|
||||
}
|
||||
if let url = bundle.url(forResource: "phrases", withExtension: "tsv") {
|
||||
return url
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
private static func rankingScore(_ term: Term) -> Int {
|
||||
var score = term.weight * 100
|
||||
if containsLatinLetters(term.word) { score += 50 }
|
||||
if term.word.count <= 8 { score += 10 }
|
||||
return score
|
||||
}
|
||||
|
||||
private static func containsLatinLetters(_ text: String) -> Bool {
|
||||
text.unicodeScalars.contains { scalar in
|
||||
scalar.isASCII && CharacterSet.letters.contains(scalar)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -384,19 +384,26 @@ public final class FlowContinuousCapture {
|
||||
|
||||
/// Re-activate capture after returning from background without
|
||||
/// reinstalling the tap (iOS may deactivate the audio session).
|
||||
public func reassertIfRunning() {
|
||||
guard isRunning else { return }
|
||||
@discardableResult
|
||||
public func reassertIfRunning() -> Bool {
|
||||
guard isRunning else { return false }
|
||||
let session = AVAudioSession.sharedInstance()
|
||||
try? session.setCategory(
|
||||
.playAndRecord,
|
||||
mode: .measurement,
|
||||
options: [.defaultToSpeaker, .allowBluetoothHFP, .mixWithOthers]
|
||||
)
|
||||
try? session.setActive(true, options: .notifyOthersOnDeactivation)
|
||||
if !audioEngine.isRunning {
|
||||
try? audioEngine.start()
|
||||
do {
|
||||
try session.setCategory(
|
||||
.playAndRecord,
|
||||
mode: .measurement,
|
||||
options: [.defaultToSpeaker, .allowBluetoothHFP, .mixWithOthers]
|
||||
)
|
||||
try session.setActive(true, options: .notifyOthersOnDeactivation)
|
||||
if !audioEngine.isRunning {
|
||||
try audioEngine.start()
|
||||
}
|
||||
notifyEngineLiveChanged()
|
||||
return engineIsLive
|
||||
} catch {
|
||||
notifyEngineLiveChanged()
|
||||
return false
|
||||
}
|
||||
notifyEngineLiveChanged()
|
||||
}
|
||||
|
||||
public func awaitAudioFlowing(
|
||||
|
||||
@@ -6,6 +6,158 @@
|
||||
|
||||
import Foundation
|
||||
|
||||
public struct FlowCommand: Codable, Equatable, Sendable {
|
||||
public enum Action: String, Codable, Sendable {
|
||||
case startRecording
|
||||
case stopRecording
|
||||
case abort
|
||||
}
|
||||
|
||||
public let protocolVersion: Int
|
||||
public let sessionId: UUID
|
||||
public let utteranceId: UUID
|
||||
public let commandSeq: Int64
|
||||
public let action: Action
|
||||
public let localeId: String
|
||||
public let createdAt: TimeInterval
|
||||
|
||||
public init(
|
||||
protocolVersion: Int = 1,
|
||||
sessionId: UUID,
|
||||
utteranceId: UUID,
|
||||
commandSeq: Int64,
|
||||
action: Action,
|
||||
localeId: String,
|
||||
createdAt: TimeInterval = Date().timeIntervalSince1970
|
||||
) {
|
||||
self.protocolVersion = protocolVersion
|
||||
self.sessionId = sessionId
|
||||
self.utteranceId = utteranceId
|
||||
self.commandSeq = commandSeq
|
||||
self.action = action
|
||||
self.localeId = localeId
|
||||
self.createdAt = createdAt
|
||||
}
|
||||
}
|
||||
|
||||
public struct FlowResult: Codable, Equatable, Sendable {
|
||||
public enum Status: String, Codable, Sendable {
|
||||
case partial
|
||||
case final
|
||||
case error
|
||||
case aborted
|
||||
case timeout
|
||||
}
|
||||
|
||||
public let protocolVersion: Int
|
||||
public let sessionId: UUID
|
||||
public let utteranceId: UUID
|
||||
public let commandSeq: Int64
|
||||
public let status: Status
|
||||
public let text: String?
|
||||
public let warning: String?
|
||||
public let errorKind: FlowSessionKeys.TranscriptionErrorKind?
|
||||
public let createdAt: TimeInterval
|
||||
|
||||
public init(
|
||||
protocolVersion: Int = 1,
|
||||
sessionId: UUID,
|
||||
utteranceId: UUID,
|
||||
commandSeq: Int64,
|
||||
status: Status,
|
||||
text: String? = nil,
|
||||
warning: String? = nil,
|
||||
errorKind: FlowSessionKeys.TranscriptionErrorKind? = nil,
|
||||
createdAt: TimeInterval = Date().timeIntervalSince1970
|
||||
) {
|
||||
self.protocolVersion = protocolVersion
|
||||
self.sessionId = sessionId
|
||||
self.utteranceId = utteranceId
|
||||
self.commandSeq = commandSeq
|
||||
self.status = status
|
||||
self.text = text
|
||||
self.warning = warning
|
||||
self.errorKind = errorKind
|
||||
self.createdAt = createdAt
|
||||
}
|
||||
}
|
||||
|
||||
public struct FlowAck: Codable, Equatable, Sendable {
|
||||
public let protocolVersion: Int
|
||||
public let sessionId: UUID
|
||||
public let utteranceId: UUID
|
||||
public let commandSeq: Int64
|
||||
public let consumedAt: TimeInterval
|
||||
|
||||
public init(
|
||||
protocolVersion: Int = 1,
|
||||
sessionId: UUID,
|
||||
utteranceId: UUID,
|
||||
commandSeq: Int64,
|
||||
consumedAt: TimeInterval = Date().timeIntervalSince1970
|
||||
) {
|
||||
self.protocolVersion = protocolVersion
|
||||
self.sessionId = sessionId
|
||||
self.utteranceId = utteranceId
|
||||
self.commandSeq = commandSeq
|
||||
self.consumedAt = consumedAt
|
||||
}
|
||||
}
|
||||
|
||||
public struct FlowReadySnapshot: Codable, Equatable, Sendable {
|
||||
public enum Reason: String, Codable, Sendable {
|
||||
case ready
|
||||
case noSession
|
||||
case starting
|
||||
case audioEngineNotLive
|
||||
case waitingForAudioProof
|
||||
case recording
|
||||
case processing
|
||||
case permissionMissing
|
||||
case appGroupUnavailable
|
||||
case hostLost
|
||||
case error
|
||||
}
|
||||
|
||||
public let protocolVersion: Int
|
||||
public let sessionId: UUID?
|
||||
public let ready: Bool
|
||||
public let reason: Reason
|
||||
public let heartbeatAt: TimeInterval
|
||||
public let readyAt: TimeInterval?
|
||||
public let audioProofAt: TimeInterval?
|
||||
public let engineMode: String
|
||||
public let localeId: String
|
||||
public let busyUtteranceId: UUID?
|
||||
public let sessionExpiresAt: TimeInterval?
|
||||
|
||||
public init(
|
||||
protocolVersion: Int = 1,
|
||||
sessionId: UUID?,
|
||||
ready: Bool,
|
||||
reason: Reason,
|
||||
heartbeatAt: TimeInterval = Date().timeIntervalSince1970,
|
||||
readyAt: TimeInterval? = nil,
|
||||
audioProofAt: TimeInterval? = nil,
|
||||
engineMode: String,
|
||||
localeId: String,
|
||||
busyUtteranceId: UUID? = nil,
|
||||
sessionExpiresAt: TimeInterval? = nil
|
||||
) {
|
||||
self.protocolVersion = protocolVersion
|
||||
self.sessionId = sessionId
|
||||
self.ready = ready
|
||||
self.reason = reason
|
||||
self.heartbeatAt = heartbeatAt
|
||||
self.readyAt = readyAt
|
||||
self.audioProofAt = audioProofAt
|
||||
self.engineMode = engineMode
|
||||
self.localeId = localeId
|
||||
self.busyUtteranceId = busyUtteranceId
|
||||
self.sessionExpiresAt = sessionExpiresAt
|
||||
}
|
||||
}
|
||||
|
||||
public struct FlowTranscriptionError: Equatable, Sendable {
|
||||
public let message: String
|
||||
public let kind: FlowSessionKeys.TranscriptionErrorKind
|
||||
@@ -36,6 +188,15 @@ public enum FlowSessionBridge {
|
||||
}
|
||||
}
|
||||
|
||||
private static func encode<T: Encodable>(_ value: T) -> Data? {
|
||||
try? JSONEncoder().encode(value)
|
||||
}
|
||||
|
||||
private static func decode<T: Decodable>(_ type: T.Type, from data: Data?) -> T? {
|
||||
guard let data else { return nil }
|
||||
return try? JSONDecoder().decode(type, from: data)
|
||||
}
|
||||
|
||||
/// Keyboard/read side: refresh App Group defaults after the extension was
|
||||
/// suspended so decisions are not based on stale in-process caches.
|
||||
public static func reloadFromDisk(defaults: UserDefaults? = nil) {
|
||||
@@ -45,10 +206,87 @@ public enum FlowSessionBridge {
|
||||
}
|
||||
}
|
||||
|
||||
// MARK: - Typed Flow protocol
|
||||
|
||||
public static func writeCommand(_ command: FlowCommand, defaults: UserDefaults? = nil) {
|
||||
let store = resolvedDefaults(defaults)
|
||||
if let data = encode(command) {
|
||||
store.set(data, forKey: FlowSessionKeys.flowCommandPayload)
|
||||
}
|
||||
flush(store)
|
||||
FlowSessionDarwin.postCommandChanged()
|
||||
}
|
||||
|
||||
public static func latestCommand(defaults: UserDefaults? = nil) -> FlowCommand? {
|
||||
let store = resolvedDefaults(defaults)
|
||||
return decode(FlowCommand.self, from: store.data(forKey: FlowSessionKeys.flowCommandPayload))
|
||||
}
|
||||
|
||||
public static func writeResult(_ result: FlowResult, defaults: UserDefaults? = nil) {
|
||||
let store = resolvedDefaults(defaults)
|
||||
if let data = encode(result) {
|
||||
store.set(data, forKey: FlowSessionKeys.flowResultPayload)
|
||||
}
|
||||
flush(store)
|
||||
FlowSessionDarwin.postTranscriptionChanged()
|
||||
}
|
||||
|
||||
public static func latestResult(defaults: UserDefaults? = nil) -> FlowResult? {
|
||||
let store = resolvedDefaults(defaults)
|
||||
return decode(FlowResult.self, from: store.data(forKey: FlowSessionKeys.flowResultPayload))
|
||||
}
|
||||
|
||||
public static func clearResult(defaults: UserDefaults? = nil) {
|
||||
let store = resolvedDefaults(defaults)
|
||||
store.removeObject(forKey: FlowSessionKeys.flowResultPayload)
|
||||
flush(store)
|
||||
}
|
||||
|
||||
public static func writeAck(_ ack: FlowAck, defaults: UserDefaults? = nil) {
|
||||
let store = resolvedDefaults(defaults)
|
||||
if let data = encode(ack) {
|
||||
store.set(data, forKey: FlowSessionKeys.flowAckPayload)
|
||||
}
|
||||
flush(store)
|
||||
}
|
||||
|
||||
public static func latestAck(defaults: UserDefaults? = nil) -> FlowAck? {
|
||||
let store = resolvedDefaults(defaults)
|
||||
return decode(FlowAck.self, from: store.data(forKey: FlowSessionKeys.flowAckPayload))
|
||||
}
|
||||
|
||||
public static func writeReadySnapshot(_ snapshot: FlowReadySnapshot, defaults: UserDefaults? = nil) {
|
||||
let store = resolvedDefaults(defaults)
|
||||
if let data = encode(snapshot) {
|
||||
store.set(data, forKey: FlowSessionKeys.flowReadyPayload)
|
||||
}
|
||||
if snapshot.ready {
|
||||
store.set(true, forKey: FlowSessionKeys.flowHostReady)
|
||||
if let readyAt = snapshot.readyAt {
|
||||
store.set(readyAt, forKey: FlowSessionKeys.flowHostReadyAt)
|
||||
}
|
||||
} else {
|
||||
clearHostReady(defaults: store, notify: false)
|
||||
store.removeObject(forKey: FlowSessionKeys.flowReadyPayload)
|
||||
}
|
||||
if let expires = snapshot.sessionExpiresAt {
|
||||
store.set(expires, forKey: FlowSessionKeys.flowSessionExpires)
|
||||
}
|
||||
store.set(snapshot.heartbeatAt, forKey: FlowSessionKeys.flowHeartbeat)
|
||||
flush(store)
|
||||
FlowSessionDarwin.postHostReadyChanged()
|
||||
}
|
||||
|
||||
public static func readySnapshot(defaults: UserDefaults? = nil) -> FlowReadySnapshot? {
|
||||
let store = resolvedDefaults(defaults)
|
||||
return decode(FlowReadySnapshot.self, from: store.data(forKey: FlowSessionKeys.flowReadyPayload))
|
||||
}
|
||||
|
||||
// MARK: - Session lifecycle (host app)
|
||||
|
||||
public static func markSessionActive(
|
||||
duration: TimeInterval? = nil,
|
||||
sessionId: UUID? = nil,
|
||||
defaults: UserDefaults? = nil
|
||||
) {
|
||||
let store = resolvedDefaults(defaults)
|
||||
@@ -59,8 +297,26 @@ public enum FlowSessionBridge {
|
||||
store.set(expires, forKey: FlowSessionKeys.flowSessionExpires)
|
||||
store.set(now, forKey: FlowSessionKeys.lastActivityAt)
|
||||
writeHeartbeat(defaults: store)
|
||||
setRecordingState(.idle, defaults: store)
|
||||
clearTranscription(defaults: store)
|
||||
store.removeObject(forKey: FlowSessionKeys.flowCommandPayload)
|
||||
store.removeObject(forKey: FlowSessionKeys.flowResultPayload)
|
||||
store.removeObject(forKey: FlowSessionKeys.flowAckPayload)
|
||||
if let sessionId {
|
||||
let snapshot = FlowReadySnapshot(
|
||||
sessionId: sessionId,
|
||||
ready: false,
|
||||
reason: .starting,
|
||||
heartbeatAt: now,
|
||||
engineMode: AppGroupConfiguration.load(fromAvailable: store).engineMode,
|
||||
localeId: AppGroupConfiguration.load(fromAvailable: store).localeId,
|
||||
sessionExpiresAt: expires
|
||||
)
|
||||
if let data = encode(snapshot) {
|
||||
store.set(data, forKey: FlowSessionKeys.flowReadyPayload)
|
||||
}
|
||||
} else {
|
||||
store.removeObject(forKey: FlowSessionKeys.flowReadyPayload)
|
||||
}
|
||||
flush(store)
|
||||
}
|
||||
|
||||
@@ -69,8 +325,11 @@ public enum FlowSessionBridge {
|
||||
store.set(false, forKey: FlowSessionKeys.flowSessionActive)
|
||||
store.removeObject(forKey: FlowSessionKeys.flowSessionExpires)
|
||||
store.removeObject(forKey: FlowSessionKeys.flowHeartbeat)
|
||||
setRecordingState(.idle, defaults: store)
|
||||
clearTranscription(defaults: store)
|
||||
store.removeObject(forKey: FlowSessionKeys.flowCommandPayload)
|
||||
store.removeObject(forKey: FlowSessionKeys.flowResultPayload)
|
||||
store.removeObject(forKey: FlowSessionKeys.flowAckPayload)
|
||||
store.removeObject(forKey: FlowSessionKeys.flowReadyPayload)
|
||||
clearHostReady(defaults: store, notify: false)
|
||||
flush(store)
|
||||
}
|
||||
@@ -185,6 +444,15 @@ public enum FlowSessionBridge {
|
||||
/// True when the host has published a fresh ready contract (stricter than heartbeat alone).
|
||||
public static func isHostReady(defaults: UserDefaults? = nil) -> Bool {
|
||||
let store = resolvedDefaults(defaults)
|
||||
if let snapshot = readySnapshot(defaults: store) {
|
||||
guard snapshot.ready else { return false }
|
||||
guard isHostReachable(defaults: store) else { return false }
|
||||
if let readyAt = snapshot.readyAt {
|
||||
let skew = abs(snapshot.heartbeatAt - readyAt)
|
||||
guard skew <= FlowSessionKeys.hostReadyMaxHeartbeatSkew else { return false }
|
||||
}
|
||||
return true
|
||||
}
|
||||
guard isHostReachable(defaults: store) else { return false }
|
||||
return store.bool(forKey: FlowSessionKeys.flowHostReady)
|
||||
}
|
||||
@@ -393,6 +661,10 @@ public enum FlowSessionBridge {
|
||||
store.removeObject(forKey: FlowSessionKeys.flowHeartbeat)
|
||||
store.removeObject(forKey: FlowSessionKeys.keyboardRecordingState)
|
||||
store.removeObject(forKey: FlowSessionKeys.transcriptionLanguage)
|
||||
store.removeObject(forKey: FlowSessionKeys.flowCommandPayload)
|
||||
store.removeObject(forKey: FlowSessionKeys.flowResultPayload)
|
||||
store.removeObject(forKey: FlowSessionKeys.flowAckPayload)
|
||||
store.removeObject(forKey: FlowSessionKeys.flowReadyPayload)
|
||||
clearTranscription(defaults: store)
|
||||
store.removeObject(forKey: FlowSessionKeys.audioLevels)
|
||||
store.removeObject(forKey: FlowSessionKeys.pendingHostBundleId)
|
||||
|
||||
@@ -8,6 +8,8 @@ import Foundation
|
||||
|
||||
public enum FlowSessionDarwin {
|
||||
public static let notificationName = "com.osgkeyboard.flow.session.changed"
|
||||
/// Posted when the keyboard writes a command for the host app.
|
||||
public static let commandNotificationName = "com.osgkeyboard.flow.command.changed"
|
||||
/// Posted when the host app writes a transcription result or error.
|
||||
public static let transcriptionNotificationName = "com.osgkeyboard.flow.transcription.changed"
|
||||
/// Posted when the host app publishes or clears the ready contract.
|
||||
@@ -23,6 +25,16 @@ public enum FlowSessionDarwin {
|
||||
)
|
||||
}
|
||||
|
||||
public static func postCommandChanged() {
|
||||
CFNotificationCenterPostNotification(
|
||||
CFNotificationCenterGetDarwinNotifyCenter(),
|
||||
CFNotificationName(commandNotificationName as CFString),
|
||||
nil,
|
||||
nil,
|
||||
true
|
||||
)
|
||||
}
|
||||
|
||||
public static func postTranscriptionChanged() {
|
||||
CFNotificationCenterPostNotification(
|
||||
CFNotificationCenterGetDarwinNotifyCenter(),
|
||||
|
||||
@@ -7,6 +7,10 @@
|
||||
import Foundation
|
||||
|
||||
public enum FlowSessionKeys {
|
||||
public static let flowCommandPayload = "flow.commandPayload.v1"
|
||||
public static let flowResultPayload = "flow.resultPayload.v1"
|
||||
public static let flowAckPayload = "flow.ackPayload.v1"
|
||||
public static let flowReadyPayload = "flow.readyPayload.v1"
|
||||
public static let flowSessionActive = "flow.flowSessionActive"
|
||||
public static let flowSessionExpires = "flow.flowSessionExpires"
|
||||
public static let flowHeartbeat = "flow.flowHeartbeat"
|
||||
@@ -74,7 +78,7 @@ public enum FlowSessionKeys {
|
||||
}
|
||||
|
||||
/// Structured host → keyboard transcription failure kind.
|
||||
public enum TranscriptionErrorKind: String, Sendable, Equatable {
|
||||
public enum TranscriptionErrorKind: String, Sendable, Equatable, Codable {
|
||||
case noSpeech
|
||||
case recognitionInterrupted
|
||||
case audioUnavailable
|
||||
|
||||
@@ -0,0 +1,159 @@
|
||||
// LocalASRBiasAdapter.swift
|
||||
// OSGKeyboard · Shared
|
||||
//
|
||||
// Maps `PersonalDictionary` + builtin lexicon + runtime context into the
|
||||
// layered bias outputs consumed by local ASR, correction, and polish.
|
||||
|
||||
import Foundation
|
||||
|
||||
public enum LocalASRBiasAdapter {
|
||||
|
||||
/// Bundle IDs where computer-science vocabulary is especially likely.
|
||||
private static let codeEditorBundleIDs: Set<String> = [
|
||||
"com.apple.dt.Xcode",
|
||||
"com.microsoft.VSCode",
|
||||
"com.google.android.studio",
|
||||
"com.jetbrains.intellij",
|
||||
"com.jetbrains.AppCode",
|
||||
"com.sublimetext.4",
|
||||
"com.apple.Terminal",
|
||||
"com.googlecode.iterm2",
|
||||
"dev.warp.Warp-Stable",
|
||||
]
|
||||
|
||||
public static func adapt(
|
||||
_ request: LocalASRBiasRequest,
|
||||
lexicon: BuiltinLexiconIndex = .shared
|
||||
) -> LocalASRBiasPayload {
|
||||
let capabilities = request.capabilities
|
||||
let dictionary = request.dictionary
|
||||
|
||||
var selectedSources = ["user"]
|
||||
let preferredSources = Self.preferredLexiconSources(for: request.frontAppBundleId)
|
||||
if preferredSources != nil {
|
||||
selectedSources.append("builtin-computer")
|
||||
} else {
|
||||
selectedSources.append("builtin-top")
|
||||
}
|
||||
|
||||
let userSorted = dictionary.effectiveEntries.sorted { $0.usageCount > $1.usageCount }
|
||||
var mergedTerms: [String] = []
|
||||
var seen = Set<String>()
|
||||
func appendTerm(_ term: String) {
|
||||
let trimmed = term.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||
guard !trimmed.isEmpty else { return }
|
||||
let key = trimmed.lowercased()
|
||||
guard seen.insert(key).inserted else { return }
|
||||
mergedTerms.append(trimmed)
|
||||
}
|
||||
|
||||
for entry in userSorted {
|
||||
appendTerm(entry.term)
|
||||
}
|
||||
let userTermCount = mergedTerms.count
|
||||
|
||||
let builtinWords = lexicon.topTerms(
|
||||
limit: request.builtinASRLimit,
|
||||
minimumWeight: 4,
|
||||
preferredSources: preferredSources
|
||||
)
|
||||
let beforeBuiltin = mergedTerms.count
|
||||
for word in builtinWords {
|
||||
appendTerm(word)
|
||||
}
|
||||
let builtinTermCount = mergedTerms.count - beforeBuiltin
|
||||
|
||||
var hardHotwords: [String] = []
|
||||
switch capabilities.hotwordMode {
|
||||
case .perRequest, .recognizerScoped:
|
||||
let cap = max(capabilities.maxHotwordCount, 1)
|
||||
hardHotwords = Self.hardHotwordList(from: mergedTerms, maxCount: cap)
|
||||
case .cloudVocabulary:
|
||||
hardHotwords = dictionary.asrHotwords(maxCount: max(capabilities.maxHotwordCount, 1))
|
||||
case .none, .promptOnly:
|
||||
break
|
||||
}
|
||||
|
||||
var promptBias: String?
|
||||
var truncated = false
|
||||
var truncationReason: String?
|
||||
|
||||
if capabilities.hotwordMode == .promptOnly, capabilities.maxPromptCharacters > 0 {
|
||||
let built = Self.buildPromptBias(
|
||||
dictionary: dictionary,
|
||||
builtinTerms: builtinWords,
|
||||
maxCharacters: capabilities.maxPromptCharacters
|
||||
)
|
||||
if built.count > capabilities.maxPromptCharacters {
|
||||
truncated = true
|
||||
truncationReason = "promptBias exceeded \(capabilities.maxPromptCharacters) characters"
|
||||
}
|
||||
promptBias = built.isEmpty ? nil : built
|
||||
}
|
||||
|
||||
let polishFragment = Self.buildPolishFragment(
|
||||
dictionary: dictionary,
|
||||
builtinTerms: builtinWords,
|
||||
maxTerms: request.builtinPolishLimit
|
||||
)
|
||||
|
||||
let correctionPairs = dictionary.localCorrectionPairs()
|
||||
|
||||
return LocalASRBiasPayload(
|
||||
hardHotwords: hardHotwords,
|
||||
promptBias: promptBias,
|
||||
corpusContext: promptBias,
|
||||
polishFragment: polishFragment,
|
||||
correctionPairs: correctionPairs,
|
||||
diagnostics: LocalASRBiasDiagnostics(
|
||||
userTermCount: userTermCount,
|
||||
builtinTermCount: builtinTermCount,
|
||||
truncated: truncated,
|
||||
truncationReason: truncationReason,
|
||||
selectedSources: selectedSources
|
||||
)
|
||||
)
|
||||
}
|
||||
|
||||
// MARK: - Private
|
||||
|
||||
private static func preferredLexiconSources(for bundleId: String?) -> Set<String>? {
|
||||
guard let bundleId, codeEditorBundleIDs.contains(bundleId) else { return nil }
|
||||
return ["computer_terms"]
|
||||
}
|
||||
|
||||
private static func hardHotwordList(from terms: [String], maxCount: Int) -> [String] {
|
||||
Array(terms.prefix(maxCount))
|
||||
}
|
||||
|
||||
private static func buildPromptBias(
|
||||
dictionary: PersonalDictionary,
|
||||
builtinTerms: [String],
|
||||
maxCharacters: Int
|
||||
) -> String {
|
||||
let userBias = dictionary.asrPromptBias(maxCharacters: maxCharacters)
|
||||
let userTermsLower = Set(dictionary.effectiveEntries.map { $0.term.lowercased() })
|
||||
let extras = builtinTerms.filter { !userTermsLower.contains($0.lowercased()) }
|
||||
guard !extras.isEmpty else { return userBias }
|
||||
|
||||
let extraBlock = "常见技术词汇:\(extras.prefix(80).joined(separator: "、"))"
|
||||
if userBias.isEmpty {
|
||||
return String(extraBlock.prefix(maxCharacters))
|
||||
}
|
||||
let combined = userBias + ";" + extraBlock
|
||||
return String(combined.prefix(maxCharacters))
|
||||
}
|
||||
|
||||
private static func buildPolishFragment(
|
||||
dictionary: PersonalDictionary,
|
||||
builtinTerms: [String],
|
||||
maxTerms: Int
|
||||
) -> String {
|
||||
let userTermsLower = Set(dictionary.effectiveEntries.map { $0.term.lowercased() })
|
||||
let extras = builtinTerms
|
||||
.filter { !userTermsLower.contains($0.lowercased()) }
|
||||
.prefix(maxTerms)
|
||||
guard !extras.isEmpty else { return "" }
|
||||
return "内置技术词汇参考(需原样保留):\(extras.joined(separator: "、"))"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
// LocalASRBiasDiagnosticsStore.swift
|
||||
// OSGKeyboard · Shared
|
||||
//
|
||||
// Persists the most recent local ASR bias diagnostics for settings / debug UI.
|
||||
|
||||
import Foundation
|
||||
|
||||
public struct LocalASRBiasDiagnosticsSnapshot: Codable, Sendable, Equatable {
|
||||
public var capturedAt: Date
|
||||
public var modelId: String?
|
||||
public var backendLabel: String?
|
||||
public var diagnostics: LocalASRBiasDiagnostics
|
||||
public var hotwordCount: Int
|
||||
public var promptBiasLength: Int
|
||||
|
||||
public init(
|
||||
capturedAt: Date = Date(),
|
||||
modelId: String? = nil,
|
||||
backendLabel: String? = nil,
|
||||
diagnostics: LocalASRBiasDiagnostics,
|
||||
hotwordCount: Int = 0,
|
||||
promptBiasLength: Int = 0
|
||||
) {
|
||||
self.capturedAt = capturedAt
|
||||
self.modelId = modelId
|
||||
self.backendLabel = backendLabel
|
||||
self.diagnostics = diagnostics
|
||||
self.hotwordCount = hotwordCount
|
||||
self.promptBiasLength = promptBiasLength
|
||||
}
|
||||
}
|
||||
|
||||
public enum LocalASRBiasDiagnosticsStore {
|
||||
private static let defaultsKey = "mac.localASR.lastBiasDiagnostics"
|
||||
|
||||
public static func save(payload: LocalASRBiasPayload, modelId: String?, backendLabel: String?) {
|
||||
let snapshot = LocalASRBiasDiagnosticsSnapshot(
|
||||
modelId: modelId,
|
||||
backendLabel: backendLabel,
|
||||
diagnostics: payload.diagnostics,
|
||||
hotwordCount: payload.hardHotwords.count,
|
||||
promptBiasLength: payload.promptBias?.count ?? 0
|
||||
)
|
||||
guard let data = try? JSONEncoder().encode(snapshot) else { return }
|
||||
UserDefaults.standard.set(data, forKey: defaultsKey)
|
||||
}
|
||||
|
||||
public static func load() -> LocalASRBiasDiagnosticsSnapshot? {
|
||||
guard let data = UserDefaults.standard.data(forKey: defaultsKey) else { return nil }
|
||||
return try? JSONDecoder().decode(LocalASRBiasDiagnosticsSnapshot.self, from: data)
|
||||
}
|
||||
|
||||
public static func clear() {
|
||||
UserDefaults.standard.removeObject(forKey: defaultsKey)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
// LocalASRInstalledManifestIO.swift
|
||||
// OSGKeyboard · Shared
|
||||
|
||||
import Foundation
|
||||
|
||||
public enum LocalASRInstalledManifestIO {
|
||||
|
||||
public static func manifestURL(fileManager: FileManager = .default) -> URL {
|
||||
let appSupport = fileManager.urls(for: .applicationSupportDirectory, in: .userDomainMask).first!
|
||||
return appSupport
|
||||
.appendingPathComponent("OSGKeyboard/LocalASRModels/installed-manifest.json")
|
||||
}
|
||||
|
||||
public static func load(defaultModelId: String, fileManager: FileManager = .default) -> LocalASRInstalledManifest {
|
||||
let url = manifestURL(fileManager: fileManager)
|
||||
guard let data = try? Data(contentsOf: url),
|
||||
let manifest = try? JSONDecoder().decode(LocalASRInstalledManifest.self, from: data) else {
|
||||
return LocalASRInstalledManifest(selectedModelId: defaultModelId)
|
||||
}
|
||||
return manifest
|
||||
}
|
||||
|
||||
public static func save(_ manifest: LocalASRInstalledManifest, fileManager: FileManager = .default) throws {
|
||||
let url = manifestURL(fileManager: fileManager)
|
||||
try fileManager.createDirectory(at: url.deletingLastPathComponent(), withIntermediateDirectories: true)
|
||||
let encoder = JSONEncoder()
|
||||
encoder.outputFormatting = [.prettyPrinted, .sortedKeys]
|
||||
encoder.dateEncodingStrategy = .iso8601
|
||||
let data = try encoder.encode(manifest)
|
||||
try data.write(to: url, options: .atomic)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,157 @@
|
||||
// LocalASRModelDownloadClient.swift
|
||||
// OSGKeyboard · Shared
|
||||
//
|
||||
// URLSession download with byte-level progress and pause/resume (macOS local model installs).
|
||||
|
||||
import Foundation
|
||||
|
||||
#if os(macOS)
|
||||
|
||||
public struct LocalASRDownloadProgressUpdate: Sendable {
|
||||
public let bytesReceived: Int64
|
||||
public let bytesTotal: Int64
|
||||
|
||||
public var fraction: Double {
|
||||
guard bytesTotal > 0 else { return 0 }
|
||||
return min(1, max(0, Double(bytesReceived) / Double(bytesTotal)))
|
||||
}
|
||||
}
|
||||
|
||||
/// Controls an in-flight URLSession download; supports pause via resume data.
|
||||
public final class LocalASRModelDownloadController: NSObject, URLSessionDownloadDelegate, @unchecked Sendable {
|
||||
private let destinationURL: URL
|
||||
private let onProgress: @Sendable (LocalASRDownloadProgressUpdate) -> Void
|
||||
private lazy var delegateSession: URLSession = {
|
||||
URLSession(configuration: .default, delegate: self, delegateQueue: nil)
|
||||
}()
|
||||
|
||||
private var remoteURL: URL?
|
||||
private var task: URLSessionDownloadTask?
|
||||
private var completionContinuation: CheckedContinuation<Void, Error>?
|
||||
private var isPausing = false
|
||||
private var finished = false
|
||||
|
||||
init(
|
||||
destinationURL: URL,
|
||||
onProgress: @escaping @Sendable (LocalASRDownloadProgressUpdate) -> Void
|
||||
) {
|
||||
self.destinationURL = destinationURL
|
||||
self.onProgress = onProgress
|
||||
super.init()
|
||||
}
|
||||
|
||||
/// Runs until the archive is fully written to `destinationURL` (survives pause/resume).
|
||||
public func download(from remoteURL: URL) async throws {
|
||||
self.remoteURL = remoteURL
|
||||
try await withCheckedThrowingContinuation { (continuation: CheckedContinuation<Void, Error>) in
|
||||
completionContinuation = continuation
|
||||
startTask(resumeData: nil)
|
||||
}
|
||||
}
|
||||
|
||||
public func pause() async throws -> Data {
|
||||
guard task != nil, !finished else {
|
||||
throw LocalASRModelManagerError.downloadFailed("No active download to pause.")
|
||||
}
|
||||
return try await withCheckedThrowingContinuation { (continuation: CheckedContinuation<Data, Error>) in
|
||||
isPausing = true
|
||||
task?.cancel(byProducingResumeData: { [weak self] data in
|
||||
guard let self else { return }
|
||||
self.isPausing = false
|
||||
if let data {
|
||||
continuation.resume(returning: data)
|
||||
} else {
|
||||
continuation.resume(throwing: LocalASRModelManagerError.downloadFailed("Pause failed."))
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/// Continues a paused download; `download(from:)` must still be awaiting.
|
||||
public func resumeFromPause(_ resumeData: Data) {
|
||||
finished = false
|
||||
startTask(resumeData: resumeData)
|
||||
}
|
||||
|
||||
public func cancel() {
|
||||
finished = true
|
||||
task?.cancel()
|
||||
completionContinuation?.resume(throwing: CancellationError())
|
||||
completionContinuation = nil
|
||||
delegateSession.invalidateAndCancel()
|
||||
}
|
||||
|
||||
private func startTask(resumeData: Data?) {
|
||||
if let resumeData {
|
||||
task = delegateSession.downloadTask(withResumeData: resumeData)
|
||||
} else if let remoteURL {
|
||||
task = delegateSession.downloadTask(with: remoteURL)
|
||||
}
|
||||
task?.resume()
|
||||
}
|
||||
|
||||
// MARK: - URLSessionDownloadDelegate
|
||||
|
||||
public func urlSession(
|
||||
_ session: URLSession,
|
||||
downloadTask: URLSessionDownloadTask,
|
||||
didWriteData bytesWritten: Int64,
|
||||
totalBytesWritten: Int64,
|
||||
totalBytesExpectedToWrite: Int64
|
||||
) {
|
||||
onProgress(
|
||||
LocalASRDownloadProgressUpdate(
|
||||
bytesReceived: totalBytesWritten,
|
||||
bytesTotal: max(totalBytesExpectedToWrite, 1)
|
||||
)
|
||||
)
|
||||
}
|
||||
|
||||
public func urlSession(
|
||||
_ session: URLSession,
|
||||
downloadTask: URLSessionDownloadTask,
|
||||
didFinishDownloadingTo location: URL
|
||||
) {
|
||||
guard !finished else { return }
|
||||
finished = true
|
||||
do {
|
||||
let fm = FileManager.default
|
||||
if fm.fileExists(atPath: destinationURL.path) {
|
||||
try fm.removeItem(at: destinationURL)
|
||||
}
|
||||
try fm.moveItem(at: location, to: destinationURL)
|
||||
completionContinuation?.resume()
|
||||
} catch {
|
||||
completionContinuation?.resume(
|
||||
throwing: LocalASRModelManagerError.downloadFailed(error.localizedDescription)
|
||||
)
|
||||
}
|
||||
completionContinuation = nil
|
||||
session.finishTasksAndInvalidate()
|
||||
}
|
||||
|
||||
public func urlSession(_ session: URLSession, task: URLSessionTask, didCompleteWithError error: Error?) {
|
||||
guard !finished else { return }
|
||||
if isPausing { return }
|
||||
if let error {
|
||||
finished = true
|
||||
completionContinuation?.resume(
|
||||
throwing: LocalASRModelManagerError.downloadFailed(error.localizedDescription)
|
||||
)
|
||||
completionContinuation = nil
|
||||
session.finishTasksAndInvalidate()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public enum LocalASRModelDownloadClient {
|
||||
|
||||
public static func makeController(
|
||||
destinationURL: URL,
|
||||
onProgress: @escaping @Sendable (LocalASRDownloadProgressUpdate) -> Void
|
||||
) -> LocalASRModelDownloadController {
|
||||
LocalASRModelDownloadController(destinationURL: destinationURL, onProgress: onProgress)
|
||||
}
|
||||
}
|
||||
|
||||
#endif
|
||||
@@ -0,0 +1,138 @@
|
||||
// LocalASRModelInstallState.swift
|
||||
// OSGKeyboard · Shared
|
||||
|
||||
import Foundation
|
||||
|
||||
public enum LocalASRModelInstallState {
|
||||
|
||||
public static func rootDirectory(fileManager: FileManager = .default) -> URL {
|
||||
let appSupport = fileManager.urls(for: .applicationSupportDirectory, in: .userDomainMask).first!
|
||||
return appSupport.appendingPathComponent("OSGKeyboard/LocalASRModels", isDirectory: true)
|
||||
}
|
||||
|
||||
public static func installDirectory(for relativePath: String, fileManager: FileManager = .default) -> URL {
|
||||
rootDirectory(fileManager: fileManager).appendingPathComponent(relativePath, isDirectory: true)
|
||||
}
|
||||
|
||||
public static func isInstalled(
|
||||
_ model: LocalASRModelDefinition,
|
||||
manualMLXPath: String?,
|
||||
fileManager: FileManager = .default
|
||||
) -> Bool {
|
||||
switch model.installKind {
|
||||
case .manual:
|
||||
guard let required = model.requiredRelativeFiles, !required.isEmpty else { return false }
|
||||
let base = URL(fileURLWithPath: manualMLXPath ?? "", isDirectory: true)
|
||||
guard fileManager.fileExists(atPath: base.path) else { return false }
|
||||
return required.allSatisfy { fileManager.fileExists(atPath: base.appendingPathComponent($0).path) }
|
||||
case .archive:
|
||||
guard let relative = model.installRelativePath,
|
||||
let layout = model.layout,
|
||||
let baseName = model.archiveBaseName else { return false }
|
||||
let root = installDirectory(for: relative, fileManager: fileManager)
|
||||
.appendingPathComponent(baseName, isDirectory: true)
|
||||
return validateArchiveModel(at: root, model: model, layout: layout, fileManager: fileManager)
|
||||
case .runtime:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
public static func modelRootURL(
|
||||
_ model: LocalASRModelDefinition,
|
||||
fileManager: FileManager = .default
|
||||
) -> URL? {
|
||||
guard model.installKind == .archive,
|
||||
let relative = model.installRelativePath,
|
||||
let baseName = model.archiveBaseName else { return nil }
|
||||
return installDirectory(for: relative, fileManager: fileManager)
|
||||
.appendingPathComponent(baseName, isDirectory: true)
|
||||
}
|
||||
|
||||
public static func resolveRuntimeBinary(
|
||||
runtime: LocalASRRuntimeDefinition,
|
||||
fileManager: FileManager = .default
|
||||
) -> URL? {
|
||||
let root = installDirectory(for: runtime.installRelativePath, fileManager: fileManager)
|
||||
for candidate in runtime.binaryCandidates {
|
||||
let direct = root.appendingPathComponent(candidate)
|
||||
if fileManager.isExecutableFile(atPath: direct.path) {
|
||||
return direct
|
||||
}
|
||||
}
|
||||
for candidate in runtime.binaryCandidates {
|
||||
let name = (candidate as NSString).lastPathComponent
|
||||
if let found = findExecutable(named: name, under: root, fileManager: fileManager) {
|
||||
return found
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
public static func isRuntimeInstalled(
|
||||
_ runtime: LocalASRRuntimeDefinition,
|
||||
fileManager: FileManager = .default
|
||||
) -> Bool {
|
||||
resolveRuntimeBinary(runtime: runtime, fileManager: fileManager) != nil
|
||||
}
|
||||
|
||||
// MARK: - Private
|
||||
|
||||
private static func validateArchiveModel(
|
||||
at root: URL,
|
||||
model: LocalASRModelDefinition,
|
||||
layout: LocalASRModelLayout,
|
||||
fileManager: FileManager
|
||||
) -> Bool {
|
||||
switch model.backend {
|
||||
case .sherpaQwen3:
|
||||
guard let conv = layout.convFrontend,
|
||||
let encoder = layout.encoder,
|
||||
let decoder = layout.decoder,
|
||||
let tokenizer = layout.tokenizer else { return false }
|
||||
return fileManager.fileExists(atPath: root.appendingPathComponent(conv).path)
|
||||
&& fileManager.fileExists(atPath: root.appendingPathComponent(encoder).path)
|
||||
&& fileManager.fileExists(atPath: root.appendingPathComponent(decoder).path)
|
||||
&& fileManager.fileExists(atPath: root.appendingPathComponent(tokenizer, isDirectory: true).path)
|
||||
case .sherpaSenseVoice:
|
||||
guard let onnx = layout.senseVoiceModel,
|
||||
let tokens = layout.tokens else { return false }
|
||||
return fileManager.fileExists(atPath: root.appendingPathComponent(onnx).path)
|
||||
&& fileManager.fileExists(atPath: root.appendingPathComponent(tokens).path)
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
private static func findExecutable(
|
||||
named name: String,
|
||||
under root: URL,
|
||||
fileManager: FileManager
|
||||
) -> URL? {
|
||||
guard let enumerator = fileManager.enumerator(
|
||||
at: root,
|
||||
includingPropertiesForKeys: [.isExecutableKey],
|
||||
options: [.skipsHiddenFiles]
|
||||
) else { return nil }
|
||||
for case let url as URL in enumerator {
|
||||
guard url.lastPathComponent == name else { continue }
|
||||
if fileManager.isExecutableFile(atPath: url.path) {
|
||||
return url
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
public static func directoryByteCount(at url: URL, fileManager: FileManager = .default) -> Int64 {
|
||||
guard let enumerator = fileManager.enumerator(
|
||||
at: url,
|
||||
includingPropertiesForKeys: [.fileSizeKey],
|
||||
options: [.skipsHiddenFiles]
|
||||
) else { return 0 }
|
||||
var total: Int64 = 0
|
||||
for case let fileURL as URL in enumerator {
|
||||
let size = (try? fileURL.resourceValues(forKeys: [.fileSizeKey]).fileSize) ?? 0
|
||||
total += Int64(size)
|
||||
}
|
||||
return total
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,492 @@
|
||||
// LocalASRModelManager.swift
|
||||
// OSGKeyboard · Shared
|
||||
//
|
||||
// Installs local ASR model archives and Sherpa runtimes under Application Support.
|
||||
// Catalog is bundled; installed state is persisted in `installed-manifest.json`.
|
||||
|
||||
import Foundation
|
||||
|
||||
public struct LocalASRInstalledManifest: Codable, Sendable, Equatable {
|
||||
public var schemaVersion: Int
|
||||
public var selectedModelId: String
|
||||
public var installedModelIDs: [String]
|
||||
public var installedRuntimeIDs: [String]
|
||||
public var updatedAt: Date
|
||||
|
||||
public init(
|
||||
schemaVersion: Int = 1,
|
||||
selectedModelId: String,
|
||||
installedModelIDs: [String] = [],
|
||||
installedRuntimeIDs: [String] = [],
|
||||
updatedAt: Date = Date()
|
||||
) {
|
||||
self.schemaVersion = schemaVersion
|
||||
self.selectedModelId = selectedModelId
|
||||
self.installedModelIDs = installedModelIDs
|
||||
self.installedRuntimeIDs = installedRuntimeIDs
|
||||
self.updatedAt = updatedAt
|
||||
}
|
||||
}
|
||||
|
||||
public enum LocalASRModelInstallPhase: String, Sendable, Equatable {
|
||||
case idle
|
||||
case downloading
|
||||
case paused
|
||||
case extracting
|
||||
case validating
|
||||
case finalizing
|
||||
case failed
|
||||
case completed
|
||||
}
|
||||
|
||||
public struct LocalASRModelInstallProgress: Sendable, Equatable {
|
||||
public var phase: LocalASRModelInstallPhase
|
||||
public var fraction: Double
|
||||
public var message: String
|
||||
public var bytesReceived: Int64?
|
||||
public var bytesTotal: Int64?
|
||||
public var activeItemId: String?
|
||||
|
||||
public init(
|
||||
phase: LocalASRModelInstallPhase,
|
||||
fraction: Double,
|
||||
message: String,
|
||||
bytesReceived: Int64? = nil,
|
||||
bytesTotal: Int64? = nil,
|
||||
activeItemId: String? = nil
|
||||
) {
|
||||
self.phase = phase
|
||||
self.fraction = fraction
|
||||
self.message = message
|
||||
self.bytesReceived = bytesReceived
|
||||
self.bytesTotal = bytesTotal
|
||||
self.activeItemId = activeItemId
|
||||
}
|
||||
|
||||
public static let idle = LocalASRModelInstallProgress(phase: .idle, fraction: 0, message: "")
|
||||
}
|
||||
|
||||
public enum LocalASRModelManagerError: Error, LocalizedError {
|
||||
case downloadFailed(String)
|
||||
case extractFailed(String)
|
||||
case validationFailed(String)
|
||||
case runtimeMissing
|
||||
case binaryMissing
|
||||
|
||||
public var errorDescription: String? {
|
||||
switch self {
|
||||
case .downloadFailed(let detail): return "Download failed: \(detail)"
|
||||
case .extractFailed(let detail): return "Extract failed: \(detail)"
|
||||
case .validationFailed(let detail): return "Validation failed: \(detail)"
|
||||
case .runtimeMissing: return "Sherpa runtime is not installed."
|
||||
case .binaryMissing: return "Sherpa binary not found in runtime bundle."
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public actor LocalASRModelManager {
|
||||
|
||||
public static let shared = LocalASRModelManager()
|
||||
|
||||
private let fileManager = FileManager.default
|
||||
private var progress = LocalASRModelInstallProgress.idle
|
||||
#if os(macOS)
|
||||
private var activeDownloadController: LocalASRModelDownloadController?
|
||||
private var pausedResumeData: Data?
|
||||
#endif
|
||||
|
||||
private init() {}
|
||||
|
||||
public func currentProgress() -> LocalASRModelInstallProgress {
|
||||
progress
|
||||
}
|
||||
|
||||
#if os(macOS)
|
||||
public func pauseDownload() async throws {
|
||||
guard progress.phase == .downloading, let controller = activeDownloadController else { return }
|
||||
let resumeData = try await controller.pause()
|
||||
pausedResumeData = resumeData
|
||||
progress = LocalASRModelInstallProgress(
|
||||
phase: .paused,
|
||||
fraction: progress.fraction,
|
||||
message: progress.message,
|
||||
bytesReceived: progress.bytesReceived,
|
||||
bytesTotal: progress.bytesTotal,
|
||||
activeItemId: progress.activeItemId
|
||||
)
|
||||
}
|
||||
|
||||
public func resumeDownload() async throws {
|
||||
guard progress.phase == .paused,
|
||||
let resumeData = pausedResumeData,
|
||||
let controller = activeDownloadController else {
|
||||
throw LocalASRModelManagerError.downloadFailed("No paused download to resume.")
|
||||
}
|
||||
pausedResumeData = nil
|
||||
progress = LocalASRModelInstallProgress(
|
||||
phase: .downloading,
|
||||
fraction: progress.fraction,
|
||||
message: progress.message,
|
||||
bytesReceived: progress.bytesReceived,
|
||||
bytesTotal: progress.bytesTotal,
|
||||
activeItemId: progress.activeItemId
|
||||
)
|
||||
controller.resumeFromPause(resumeData)
|
||||
}
|
||||
|
||||
public func isDownloadPaused() -> Bool {
|
||||
progress.phase == .paused
|
||||
}
|
||||
#endif
|
||||
|
||||
public func rootDirectory() -> URL {
|
||||
let appSupport = fileManager.urls(for: .applicationSupportDirectory, in: .userDomainMask).first!
|
||||
return appSupport.appendingPathComponent("OSGKeyboard/LocalASRModels", isDirectory: true)
|
||||
}
|
||||
|
||||
public func manifestURL() -> URL {
|
||||
LocalASRInstalledManifestIO.manifestURL(fileManager: fileManager)
|
||||
}
|
||||
|
||||
public func loadManifest(defaultModelId: String) -> LocalASRInstalledManifest {
|
||||
LocalASRInstalledManifestIO.load(defaultModelId: defaultModelId, fileManager: fileManager)
|
||||
}
|
||||
|
||||
public func saveManifest(_ manifest: LocalASRInstalledManifest) throws {
|
||||
try LocalASRInstalledManifestIO.save(manifest, fileManager: fileManager)
|
||||
}
|
||||
|
||||
public func setSelectedModelId(_ modelId: String, catalog: LocalASRCatalogDocument) throws {
|
||||
var manifest = loadManifest(defaultModelId: catalog.defaultModelId)
|
||||
manifest.selectedModelId = modelId
|
||||
manifest.updatedAt = Date()
|
||||
try saveManifest(manifest)
|
||||
}
|
||||
|
||||
public func installDirectory(for relativePath: String) -> URL {
|
||||
rootDirectory().appendingPathComponent(relativePath, isDirectory: true)
|
||||
}
|
||||
|
||||
public func isModelInstalled(_ model: LocalASRModelDefinition, manualMLXPath: String?) -> Bool {
|
||||
LocalASRModelInstallState.isInstalled(model, manualMLXPath: manualMLXPath, fileManager: fileManager)
|
||||
}
|
||||
|
||||
public func isRuntimeInstalled(_ runtime: LocalASRRuntimeDefinition) -> Bool {
|
||||
LocalASRModelInstallState.isRuntimeInstalled(runtime, fileManager: fileManager)
|
||||
}
|
||||
|
||||
#if os(macOS)
|
||||
public func resolveRuntimeBinary(runtime: LocalASRRuntimeDefinition) -> URL? {
|
||||
LocalASRModelInstallState.resolveRuntimeBinary(runtime: runtime, fileManager: fileManager)
|
||||
}
|
||||
|
||||
public func installModel(
|
||||
_ model: LocalASRModelDefinition,
|
||||
catalog: LocalASRCatalogDocument
|
||||
) async throws {
|
||||
guard model.installKind == .archive,
|
||||
let relative = model.installRelativePath,
|
||||
let baseName = model.archiveBaseName,
|
||||
let sources = model.sources,
|
||||
!sources.isEmpty else {
|
||||
throw LocalASRModelManagerError.validationFailed("Model is not downloadable.")
|
||||
}
|
||||
|
||||
progress = LocalASRModelInstallProgress(
|
||||
phase: .downloading,
|
||||
fraction: 0.05,
|
||||
message: model.displayName,
|
||||
activeItemId: model.id
|
||||
)
|
||||
if model.backend == .sherpaQwen3 || model.backend == .sherpaSenseVoice {
|
||||
try await ensureRuntimeInstalled(catalog: catalog)
|
||||
}
|
||||
let sortedSources = sources.sorted { $0.priority < $1.priority }
|
||||
var lastError: Error?
|
||||
for source in sortedSources {
|
||||
do {
|
||||
try await installArchive(
|
||||
from: source.url,
|
||||
installRelativePath: relative,
|
||||
archiveBaseName: baseName,
|
||||
layoutModel: model,
|
||||
itemId: model.id,
|
||||
displayName: model.displayName
|
||||
)
|
||||
var manifest = loadManifest(defaultModelId: catalog.defaultModelId)
|
||||
if !manifest.installedModelIDs.contains(model.id) {
|
||||
manifest.installedModelIDs.append(model.id)
|
||||
}
|
||||
manifest.updatedAt = Date()
|
||||
try saveManifest(manifest)
|
||||
progress = LocalASRModelInstallProgress(
|
||||
phase: .completed,
|
||||
fraction: 1,
|
||||
message: model.displayName,
|
||||
activeItemId: model.id
|
||||
)
|
||||
return
|
||||
} catch {
|
||||
lastError = error
|
||||
}
|
||||
}
|
||||
progress = LocalASRModelInstallProgress(
|
||||
phase: .failed,
|
||||
fraction: 0,
|
||||
message: lastError?.localizedDescription ?? "Download failed"
|
||||
)
|
||||
throw lastError ?? LocalASRModelManagerError.downloadFailed("All mirrors failed")
|
||||
}
|
||||
|
||||
public func installRuntime(
|
||||
_ runtime: LocalASRRuntimeDefinition,
|
||||
catalog: LocalASRCatalogDocument
|
||||
) async throws {
|
||||
guard let source = runtime.sources.sorted(by: { $0.priority < $1.priority }).first else {
|
||||
throw LocalASRModelManagerError.downloadFailed("No runtime source configured.")
|
||||
}
|
||||
progress = LocalASRModelInstallProgress(
|
||||
phase: .downloading,
|
||||
fraction: 0.05,
|
||||
message: runtime.displayName,
|
||||
activeItemId: runtime.id
|
||||
)
|
||||
try await installArchive(
|
||||
from: source.url,
|
||||
installRelativePath: runtime.installRelativePath,
|
||||
archiveBaseName: runtime.installRelativePath.split(separator: "/").last.map(String.init) ?? runtime.id,
|
||||
layoutModel: nil,
|
||||
expectedBinaryCandidates: runtime.binaryCandidates,
|
||||
itemId: runtime.id,
|
||||
displayName: runtime.displayName
|
||||
)
|
||||
guard isRuntimeInstalled(runtime) else {
|
||||
throw LocalASRModelManagerError.binaryMissing
|
||||
}
|
||||
var manifest = loadManifest(defaultModelId: catalog.defaultModelId)
|
||||
if !manifest.installedRuntimeIDs.contains(runtime.id) {
|
||||
manifest.installedRuntimeIDs.append(runtime.id)
|
||||
}
|
||||
manifest.updatedAt = Date()
|
||||
try saveManifest(manifest)
|
||||
progress = LocalASRModelInstallProgress(phase: .completed, fraction: 1, message: runtime.displayName)
|
||||
}
|
||||
|
||||
public func modelRootURL(_ model: LocalASRModelDefinition) -> URL? {
|
||||
LocalASRModelInstallState.modelRootURL(model, fileManager: fileManager)
|
||||
}
|
||||
|
||||
public func installDirectoryURL(for model: LocalASRModelDefinition) -> URL? {
|
||||
guard let relative = model.installRelativePath else { return nil }
|
||||
return installDirectory(for: relative)
|
||||
}
|
||||
|
||||
public func deleteModel(
|
||||
_ model: LocalASRModelDefinition,
|
||||
catalog: LocalASRCatalogDocument
|
||||
) throws {
|
||||
guard model.installKind == .archive, let relative = model.installRelativePath else { return }
|
||||
let dir = installDirectory(for: relative)
|
||||
if fileManager.fileExists(atPath: dir.path) {
|
||||
try fileManager.removeItem(at: dir)
|
||||
}
|
||||
var manifest = loadManifest(defaultModelId: catalog.defaultModelId)
|
||||
manifest.installedModelIDs.removeAll { $0 == model.id }
|
||||
if manifest.selectedModelId == model.id {
|
||||
manifest.selectedModelId = catalog.defaultModelId
|
||||
UserDefaults.standard.set(catalog.defaultModelId, forKey: LocalASRPreferenceKeys.selectedModelId)
|
||||
}
|
||||
manifest.updatedAt = Date()
|
||||
try saveManifest(manifest)
|
||||
if progress.activeItemId == model.id {
|
||||
progress = .idle
|
||||
}
|
||||
}
|
||||
|
||||
public func deleteRuntime(
|
||||
_ runtime: LocalASRRuntimeDefinition,
|
||||
catalog: LocalASRCatalogDocument
|
||||
) throws {
|
||||
let dir = installDirectory(for: runtime.installRelativePath)
|
||||
if fileManager.fileExists(atPath: dir.path) {
|
||||
try fileManager.removeItem(at: dir)
|
||||
}
|
||||
var manifest = loadManifest(defaultModelId: catalog.defaultModelId)
|
||||
manifest.installedRuntimeIDs.removeAll { $0 == runtime.id }
|
||||
manifest.updatedAt = Date()
|
||||
try saveManifest(manifest)
|
||||
}
|
||||
|
||||
private func setProgress(_ update: LocalASRModelInstallProgress) {
|
||||
progress = update
|
||||
}
|
||||
|
||||
func updateDownloadProgress(
|
||||
itemId: String,
|
||||
displayName: String,
|
||||
update: LocalASRDownloadProgressUpdate
|
||||
) {
|
||||
reportDownloadProgress(itemId: itemId, displayName: displayName, update: update)
|
||||
}
|
||||
|
||||
private func reportDownloadProgress(
|
||||
itemId: String,
|
||||
displayName: String,
|
||||
update: LocalASRDownloadProgressUpdate
|
||||
) {
|
||||
// Download phase occupies 10%–55% of the overall install bar.
|
||||
let mapped = 0.10 + update.fraction * 0.45
|
||||
progress = LocalASRModelInstallProgress(
|
||||
phase: .downloading,
|
||||
fraction: mapped,
|
||||
message: displayName,
|
||||
bytesReceived: update.bytesReceived,
|
||||
bytesTotal: update.bytesTotal,
|
||||
activeItemId: itemId
|
||||
)
|
||||
}
|
||||
#endif
|
||||
|
||||
// MARK: - Private
|
||||
|
||||
#if os(macOS)
|
||||
private func installArchive(
|
||||
from urlString: String,
|
||||
installRelativePath: String,
|
||||
archiveBaseName: String,
|
||||
layoutModel: LocalASRModelDefinition?,
|
||||
expectedBinaryCandidates: [String]? = nil,
|
||||
itemId: String,
|
||||
displayName: String
|
||||
) async throws {
|
||||
guard let remoteURL = URL(string: urlString) else {
|
||||
throw LocalASRModelManagerError.downloadFailed("Invalid URL")
|
||||
}
|
||||
|
||||
let stagingRoot = rootDirectory().appendingPathComponent("staging/\(UUID().uuidString)", isDirectory: true)
|
||||
let destinationParent = installDirectory(for: installRelativePath)
|
||||
try fileManager.createDirectory(at: stagingRoot, withIntermediateDirectories: true)
|
||||
defer { try? fileManager.removeItem(at: stagingRoot) }
|
||||
|
||||
let archiveURL = stagingRoot.appendingPathComponent(remoteURL.lastPathComponent)
|
||||
progress = LocalASRModelInstallProgress(
|
||||
phase: .downloading,
|
||||
fraction: 0.10,
|
||||
message: displayName,
|
||||
activeItemId: itemId
|
||||
)
|
||||
|
||||
do {
|
||||
let controller = LocalASRModelDownloadClient.makeController(destinationURL: archiveURL) { update in
|
||||
Task {
|
||||
await LocalASRModelManager.shared.updateDownloadProgress(
|
||||
itemId: itemId,
|
||||
displayName: displayName,
|
||||
update: update
|
||||
)
|
||||
}
|
||||
}
|
||||
activeDownloadController = controller
|
||||
try await controller.download(from: remoteURL)
|
||||
activeDownloadController = nil
|
||||
pausedResumeData = nil
|
||||
} catch {
|
||||
activeDownloadController = nil
|
||||
pausedResumeData = nil
|
||||
throw LocalASRModelManagerError.downloadFailed(error.localizedDescription)
|
||||
}
|
||||
|
||||
progress = LocalASRModelInstallProgress(
|
||||
phase: .extracting,
|
||||
fraction: 0.58,
|
||||
message: displayName,
|
||||
activeItemId: itemId
|
||||
)
|
||||
try fileManager.createDirectory(at: destinationParent, withIntermediateDirectories: true)
|
||||
let extractOK = try await extractTarBz2(archiveURL: archiveURL, destination: destinationParent)
|
||||
guard extractOK else {
|
||||
throw LocalASRModelManagerError.extractFailed("tar extraction failed")
|
||||
}
|
||||
|
||||
progress = LocalASRModelInstallProgress(
|
||||
phase: .validating,
|
||||
fraction: 0.82,
|
||||
message: displayName,
|
||||
activeItemId: itemId
|
||||
)
|
||||
if let layoutModel {
|
||||
guard LocalASRModelInstallState.isInstalled(
|
||||
layoutModel,
|
||||
manualMLXPath: nil,
|
||||
fileManager: fileManager
|
||||
) else {
|
||||
throw LocalASRModelManagerError.validationFailed("Required model files missing after extract.")
|
||||
}
|
||||
}
|
||||
if let expectedBinaryCandidates {
|
||||
let runtimeRoot = destinationParent
|
||||
let found = expectedBinaryCandidates.contains { candidate in
|
||||
let direct = runtimeRoot.appendingPathComponent(candidate)
|
||||
if fileManager.isExecutableFile(atPath: direct.path) { return true }
|
||||
let name = (candidate as NSString).lastPathComponent
|
||||
return findExecutable(named: name, under: runtimeRoot) != nil
|
||||
}
|
||||
guard found else {
|
||||
throw LocalASRModelManagerError.binaryMissing
|
||||
}
|
||||
}
|
||||
|
||||
progress = LocalASRModelInstallProgress(
|
||||
phase: .finalizing,
|
||||
fraction: 0.95,
|
||||
message: displayName,
|
||||
activeItemId: itemId
|
||||
)
|
||||
try? fileManager.removeItem(at: archiveURL)
|
||||
}
|
||||
|
||||
public func ensureRuntimeInstalled(catalog: LocalASRCatalogDocument) async throws {
|
||||
guard let runtime = LocalASRModelCatalog.runtime(
|
||||
for: LocalASRModelCatalog.currentRuntimePlatform(),
|
||||
in: catalog
|
||||
) else {
|
||||
throw LocalASRModelManagerError.runtimeMissing
|
||||
}
|
||||
if isRuntimeInstalled(runtime) { return }
|
||||
try await installRuntime(runtime, catalog: catalog)
|
||||
}
|
||||
|
||||
private func findExecutable(named name: String, under root: URL) -> URL? {
|
||||
guard let enumerator = fileManager.enumerator(
|
||||
at: root,
|
||||
includingPropertiesForKeys: [.isExecutableKey],
|
||||
options: [.skipsHiddenFiles]
|
||||
) else { return nil }
|
||||
for case let url as URL in enumerator {
|
||||
guard url.lastPathComponent == name else { continue }
|
||||
if fileManager.isExecutableFile(atPath: url.path) {
|
||||
return url
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
private func extractTarBz2(archiveURL: URL, destination: URL) async throws -> Bool {
|
||||
try await withCheckedThrowingContinuation { continuation in
|
||||
let process = Process()
|
||||
process.executableURL = URL(fileURLWithPath: "/usr/bin/tar")
|
||||
process.arguments = ["-xjf", archiveURL.path, "-C", destination.path]
|
||||
process.standardOutput = FileHandle.nullDevice
|
||||
process.standardError = FileHandle.nullDevice
|
||||
process.terminationHandler = { proc in
|
||||
continuation.resume(returning: proc.terminationStatus == 0)
|
||||
}
|
||||
do {
|
||||
try process.run()
|
||||
} catch {
|
||||
continuation.resume(throwing: LocalASRModelManagerError.extractFailed(error.localizedDescription))
|
||||
}
|
||||
}
|
||||
}
|
||||
#endif
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
// LocalASRPreferenceKeys.swift
|
||||
// OSGKeyboard · Shared
|
||||
|
||||
import Foundation
|
||||
|
||||
enum LocalASRPreferenceKeys {
|
||||
static let selectedModelId = "mac.localASR.selectedModelId"
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
// LocalASRTranscriptCorrector.swift
|
||||
// OSGKeyboard · Shared
|
||||
//
|
||||
// Deterministic alias → canonical term replacement between raw ASR output
|
||||
// and the LLM polish step. Only applies whole-phrase matches.
|
||||
|
||||
import Foundation
|
||||
|
||||
public enum LocalASRTranscriptCorrector {
|
||||
|
||||
/// Applies high-confidence alias replacements (longest match first).
|
||||
public static func apply(
|
||||
_ text: String,
|
||||
pairs: [LocalASRCorrectionPair]
|
||||
) -> String {
|
||||
guard !text.isEmpty, !pairs.isEmpty else { return text }
|
||||
|
||||
let sorted = pairs.sorted { lhs, rhs in
|
||||
if lhs.alias.count != rhs.alias.count {
|
||||
return lhs.alias.count > rhs.alias.count
|
||||
}
|
||||
return lhs.alias.localizedCaseInsensitiveCompare(rhs.alias) == .orderedAscending
|
||||
}
|
||||
|
||||
var result = text
|
||||
for pair in sorted {
|
||||
result = replaceWholeMatches(
|
||||
in: result,
|
||||
alias: pair.alias,
|
||||
term: pair.term
|
||||
)
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
// MARK: - Private
|
||||
|
||||
private static func replaceWholeMatches(
|
||||
in text: String,
|
||||
alias: String,
|
||||
term: String
|
||||
) -> String {
|
||||
guard !alias.isEmpty, alias != term else { return text }
|
||||
|
||||
if alias.unicodeScalars.allSatisfy({ $0.isASCII }) {
|
||||
return replaceASCIIWord(in: text, alias: alias, term: term)
|
||||
}
|
||||
return text.replacingOccurrences(of: alias, with: term)
|
||||
}
|
||||
|
||||
private static func replaceASCIIWord(
|
||||
in text: String,
|
||||
alias: String,
|
||||
term: String
|
||||
) -> String {
|
||||
let escaped = NSRegularExpression.escapedPattern(for: alias)
|
||||
let pattern = "(?i)(?<![A-Za-z0-9_])\(escaped)(?![A-Za-z0-9_])"
|
||||
guard let regex = try? NSRegularExpression(pattern: pattern) else {
|
||||
return text
|
||||
}
|
||||
let range = NSRange(text.startIndex..<text.endIndex, in: text)
|
||||
return regex.stringByReplacingMatches(
|
||||
in: text,
|
||||
range: range,
|
||||
withTemplate: term
|
||||
)
|
||||
}
|
||||
}
|
||||
@@ -252,7 +252,10 @@ public actor PolishingService {
|
||||
providerId: String
|
||||
) -> String {
|
||||
let dictionary = store.personalDictionary
|
||||
let dictionaryBlock = dictionary.promptFragment()
|
||||
let dictionaryBlock = Self.mergedDictionaryBlock(
|
||||
dictionary: dictionary,
|
||||
supplement: context.dictionarySupplement
|
||||
)
|
||||
let contextGuideline = context.appContext.polishGuideline
|
||||
let intensityGuideline = context.intensity.promptGuideline
|
||||
let contract = Self.globalOutputContract(useChinese: shouldUseChineseGuidance(providerId: providerId))
|
||||
@@ -321,6 +324,17 @@ public actor PolishingService {
|
||||
}
|
||||
}
|
||||
|
||||
internal static func mergedDictionaryBlock(
|
||||
dictionary: PersonalDictionary,
|
||||
supplement: String?
|
||||
) -> String {
|
||||
let base = dictionary.promptFragment()
|
||||
let extra = supplement?.trimmingCharacters(in: .whitespacesAndNewlines) ?? ""
|
||||
if base.isEmpty { return extra }
|
||||
if extra.isEmpty { return base }
|
||||
return base + "\n" + extra
|
||||
}
|
||||
|
||||
private func shouldUseChineseGuidance(providerId: String) -> Bool {
|
||||
switch providerId {
|
||||
case "zhipu", "moonshot", "qwen", "deepseek":
|
||||
|
||||
@@ -157,8 +157,8 @@
|
||||
"mac.settings.recognition" = "RECOGNITION METHOD";
|
||||
"mac.settings.cloudEngine" = "Cloud Engine & AI Refinement";
|
||||
"mac.settings.cloudEngineDesc" = "Premium transcription via your provider's API, plus AI grammar and style polishing.";
|
||||
"mac.settings.localEngine" = "Local Recognition (Qwen3-ASR)";
|
||||
"mac.settings.localEngineDesc" = "On-device ASR with Qwen3-ASR 1.7B (MLX). High privacy, zero latency.";
|
||||
"mac.settings.localEngine" = "Local Recognition";
|
||||
"mac.settings.localEngineDesc" = "On-device transcription with a local model. High privacy, zero latency.";
|
||||
"mac.settings.localSpeechFallback" = "Local Recognition (Apple Speech)";
|
||||
"mac.settings.localSpeechFallbackDesc" = "On-device Apple Speech when the Qwen3 model is not installed.";
|
||||
"mac.settings.about" = "About";
|
||||
@@ -174,6 +174,9 @@
|
||||
"mac.settings.qwen3ModelDesc" = "Folder with config.json, model.safetensors, vocab.json, and merges.txt.";
|
||||
"mac.settings.qwen3Browse" = "Choose folder…";
|
||||
"mac.settings.qwen3Missing" = "Qwen3 model not found — using Apple Speech for now.";
|
||||
"mac.settings.mlxModelMissing" = "Select a Qwen3 MLX model folder below, or choose another installed model.";
|
||||
"mac.settings.selectedModelMissing" = "%@ is not installed — using Apple Speech for now.";
|
||||
"mac.settings.localModelFallbackApple" = "No local model is ready — using Apple Speech for now.";
|
||||
"mac.settings.accessibility" = "Accessibility";
|
||||
"mac.settings.accessibilityDesc" = "Required for global shortcut and auto-paste.";
|
||||
"mac.settings.openAccessibility" = "Open System Settings";
|
||||
@@ -188,6 +191,43 @@
|
||||
"mac.error.qwen3ModelMissing" = "Qwen3-ASR model not installed";
|
||||
"mac.error.qwen3LoadFailed" = "Failed to load Qwen3 model: %@";
|
||||
"mac.error.qwen3InferenceFailed" = "Qwen3 transcription failed: %@";
|
||||
"mac.localASR.models" = "Local ASR Models";
|
||||
"mac.localASR.modelsDesc" = "Sherpa models download directly. For MLX Qwen3, drop your converted weights into the folder opened by “Open folder”. All three share one storage directory.";
|
||||
"mac.localASR.download" = "Download";
|
||||
"mac.localASR.selectFolder" = "Choose folder";
|
||||
"mac.localASR.openFolder" = "Open folder";
|
||||
"mac.localASR.pause" = "Pause";
|
||||
"mac.localASR.resume" = "Resume";
|
||||
"mac.localASR.needsFolder" = "Folder required";
|
||||
"mac.localASR.installDone" = "Install completed.";
|
||||
"mac.localASR.installed" = "Installed";
|
||||
"mac.localASR.notInstalled" = "Not installed";
|
||||
"mac.localASR.hotwordsYes" = "Hotwords";
|
||||
"mac.localASR.hotwordsNo" = "No hotwords";
|
||||
"mac.localASR.catalogMissing" = "Local ASR catalog is missing from the app bundle.";
|
||||
"mac.localASR.diagnostics" = "Last Bias Diagnostics";
|
||||
"mac.localASR.diagnosticsDesc" = "Captured after your most recent local dictation.";
|
||||
"mac.localASR.diagEmpty" = "No local dictation yet.";
|
||||
"mac.localASR.diagBackend" = "Backend";
|
||||
"mac.localASR.diagUserTerms" = "User terms";
|
||||
"mac.localASR.diagBuiltinTerms" = "Builtin terms";
|
||||
"mac.localASR.diagHotwords" = "Hotwords sent";
|
||||
"mac.localASR.diagPrompt" = "Prompt chars";
|
||||
"mac.localASR.diagTruncated" = "Prompt truncated";
|
||||
"mac.localASR.delete" = "Delete";
|
||||
"mac.localASR.deleteDone" = "Model deleted.";
|
||||
"mac.localASR.redownload" = "Re-download";
|
||||
"mac.localASR.revealInFinder" = "Reveal in Finder";
|
||||
"mac.localASR.openStorage" = "Open model storage folder";
|
||||
"mac.localASR.runtime" = "Sherpa Runtime";
|
||||
"mac.localASR.runtimeDesc" = "Required for Sherpa Qwen3 and SenseVoice models. Installed automatically with those models.";
|
||||
"mac.localASR.phase.downloading" = "Downloading";
|
||||
"mac.localASR.phase.paused" = "Paused";
|
||||
"mac.localASR.phase.extracting" = "Extracting";
|
||||
"mac.localASR.phase.validating" = "Validating";
|
||||
"mac.localASR.phase.finalizing" = "Finalizing";
|
||||
"mac.localASR.phase.failed" = "Failed";
|
||||
"mac.localASR.phase.completed" = "Completed";
|
||||
"mac.error.accessibilityRequired" = "Enable Accessibility for OSGKeyboard in System Settings";
|
||||
"mac.foregroundApp" = "Front app: %@";
|
||||
"mac.sync.settingsTitle" = "iCloud Sync";
|
||||
|
||||
@@ -157,8 +157,8 @@
|
||||
"mac.settings.recognition" = "识别方式";
|
||||
"mac.settings.cloudEngine" = "云端引擎与 AI 润色";
|
||||
"mac.settings.cloudEngineDesc" = "通过服务商 API 进行高质量转写,并自动润色语法与风格。";
|
||||
"mac.settings.localEngine" = "本地识别(Qwen3-ASR)";
|
||||
"mac.settings.localEngineDesc" = "使用 Qwen3-ASR 1.7B(MLX)本地转写,高隐私、低延迟。";
|
||||
"mac.settings.localEngine" = "本地识别";
|
||||
"mac.settings.localEngineDesc" = "在本机使用本地模型转写,高隐私、低延迟。";
|
||||
"mac.settings.localSpeechFallback" = "本地识别(Apple Speech)";
|
||||
"mac.settings.localSpeechFallbackDesc" = "未安装 Qwen3 模型时,使用 Apple 本地语音识别。";
|
||||
"mac.settings.about" = "关于";
|
||||
@@ -174,6 +174,9 @@
|
||||
"mac.settings.qwen3ModelDesc" = "需包含 config.json、model.safetensors、vocab.json 与 merges.txt。";
|
||||
"mac.settings.qwen3Browse" = "选择文件夹…";
|
||||
"mac.settings.qwen3Missing" = "未找到 Qwen3 模型,暂时使用 Apple Speech。";
|
||||
"mac.settings.mlxModelMissing" = "请在下方选择 Qwen3 MLX 模型目录,或改用其他已安装的模型。";
|
||||
"mac.settings.selectedModelMissing" = "「%@」尚未安装,暂时使用 Apple Speech。";
|
||||
"mac.settings.localModelFallbackApple" = "没有可用的本地模型,暂时使用 Apple Speech。";
|
||||
"mac.settings.accessibility" = "辅助功能";
|
||||
"mac.settings.accessibilityDesc" = "全局快捷键与自动粘贴需要此权限。";
|
||||
"mac.settings.openAccessibility" = "打开系统设置";
|
||||
@@ -188,6 +191,43 @@
|
||||
"mac.error.qwen3ModelMissing" = "未安装 Qwen3-ASR 模型";
|
||||
"mac.error.qwen3LoadFailed" = "Qwen3 模型加载失败:%@";
|
||||
"mac.error.qwen3InferenceFailed" = "Qwen3 转写失败:%@";
|
||||
"mac.localASR.models" = "本地 ASR 模型";
|
||||
"mac.localASR.modelsDesc" = "Sherpa 模型可直接下载;MLX Qwen3 请将转换好的权重放入「打开目录」指向的文件夹。三个模型共用同一存储目录。";
|
||||
"mac.localASR.download" = "下载";
|
||||
"mac.localASR.selectFolder" = "选择目录";
|
||||
"mac.localASR.openFolder" = "打开目录";
|
||||
"mac.localASR.pause" = "暂停";
|
||||
"mac.localASR.resume" = "继续";
|
||||
"mac.localASR.needsFolder" = "需选择目录";
|
||||
"mac.localASR.installDone" = "安装完成。";
|
||||
"mac.localASR.installed" = "已安装";
|
||||
"mac.localASR.notInstalled" = "未安装";
|
||||
"mac.localASR.hotwordsYes" = "支持热词";
|
||||
"mac.localASR.hotwordsNo" = "无热词";
|
||||
"mac.localASR.catalogMissing" = "应用包内缺少本地 ASR 模型目录。";
|
||||
"mac.localASR.diagnostics" = "最近一次词库诊断";
|
||||
"mac.localASR.diagnosticsDesc" = "在上一轮本地听写后记录。";
|
||||
"mac.localASR.diagEmpty" = "尚无本地听写记录。";
|
||||
"mac.localASR.diagBackend" = "后端";
|
||||
"mac.localASR.diagUserTerms" = "用户词条";
|
||||
"mac.localASR.diagBuiltinTerms" = "内置词条";
|
||||
"mac.localASR.diagHotwords" = "热词数量";
|
||||
"mac.localASR.diagPrompt" = "Prompt 字符";
|
||||
"mac.localASR.diagTruncated" = "Prompt 已截断";
|
||||
"mac.localASR.delete" = "删除";
|
||||
"mac.localASR.deleteDone" = "模型已删除。";
|
||||
"mac.localASR.redownload" = "重新下载";
|
||||
"mac.localASR.revealInFinder" = "在 Finder 中显示";
|
||||
"mac.localASR.openStorage" = "打开模型存储目录";
|
||||
"mac.localASR.runtime" = "Sherpa 运行时";
|
||||
"mac.localASR.runtimeDesc" = "Sherpa Qwen3 与 SenseVoice 模型需要此运行时;下载上述模型时会自动安装。";
|
||||
"mac.localASR.phase.downloading" = "下载中";
|
||||
"mac.localASR.phase.paused" = "已暂停";
|
||||
"mac.localASR.phase.extracting" = "解压中";
|
||||
"mac.localASR.phase.validating" = "校验中";
|
||||
"mac.localASR.phase.finalizing" = "完成安装";
|
||||
"mac.localASR.phase.failed" = "失败";
|
||||
"mac.localASR.phase.completed" = "已完成";
|
||||
"mac.error.accessibilityRequired" = "请在系统设置中为 OSGKeyboard 启用辅助功能";
|
||||
"mac.foregroundApp" = "前台应用:%@";
|
||||
"mac.sync.settingsTitle" = "iCloud 同步";
|
||||
|
||||
@@ -141,6 +141,7 @@ final class FlowSessionBridgeTests: XCTestCase {
|
||||
|
||||
func testDarwinNotificationPostsWithoutCrashing() {
|
||||
FlowSessionDarwin.postSessionChanged()
|
||||
FlowSessionDarwin.postCommandChanged()
|
||||
FlowSessionDarwin.postHostReadyChanged()
|
||||
}
|
||||
|
||||
@@ -183,4 +184,112 @@ final class FlowSessionBridgeTests: XCTestCase {
|
||||
XCTAssertFalse(defaults.bool(forKey: FlowSessionKeys.flowHostReady))
|
||||
XCTAssertFalse(FlowSessionBridge.isHostReady(defaults: defaults))
|
||||
}
|
||||
|
||||
func testFlowCommandRoundTrip() {
|
||||
let defaults = makeDefaults()
|
||||
let sessionId = UUID()
|
||||
let utteranceId = UUID()
|
||||
let command = FlowCommand(
|
||||
sessionId: sessionId,
|
||||
utteranceId: utteranceId,
|
||||
commandSeq: 42,
|
||||
action: .startRecording,
|
||||
localeId: "zh-Hans",
|
||||
createdAt: 123
|
||||
)
|
||||
|
||||
FlowSessionBridge.writeCommand(command, defaults: defaults)
|
||||
|
||||
XCTAssertEqual(FlowSessionBridge.latestCommand(defaults: defaults), command)
|
||||
}
|
||||
|
||||
func testFlowResultRoundTripPreservesUtteranceIdentity() {
|
||||
let defaults = makeDefaults()
|
||||
let sessionId = UUID()
|
||||
let utteranceId = UUID()
|
||||
let result = FlowResult(
|
||||
sessionId: sessionId,
|
||||
utteranceId: utteranceId,
|
||||
commandSeq: 43,
|
||||
status: .final,
|
||||
text: "hello",
|
||||
warning: "raw fallback",
|
||||
createdAt: 124
|
||||
)
|
||||
|
||||
FlowSessionBridge.writeResult(result, defaults: defaults)
|
||||
|
||||
XCTAssertEqual(FlowSessionBridge.latestResult(defaults: defaults), result)
|
||||
FlowSessionBridge.clearResult(defaults: defaults)
|
||||
XCTAssertNil(FlowSessionBridge.latestResult(defaults: defaults))
|
||||
}
|
||||
|
||||
func testFlowAckRoundTrip() {
|
||||
let defaults = makeDefaults()
|
||||
let ack = FlowAck(
|
||||
sessionId: UUID(),
|
||||
utteranceId: UUID(),
|
||||
commandSeq: 44,
|
||||
consumedAt: 125
|
||||
)
|
||||
|
||||
FlowSessionBridge.writeAck(ack, defaults: defaults)
|
||||
|
||||
XCTAssertEqual(FlowSessionBridge.latestAck(defaults: defaults), ack)
|
||||
}
|
||||
|
||||
func testReadySnapshotDrivesHostReady() {
|
||||
let defaults = makeDefaults()
|
||||
let sessionId = UUID()
|
||||
let now = Date().timeIntervalSince1970
|
||||
FlowSessionBridge.markSessionActive(duration: 60, sessionId: sessionId, defaults: defaults)
|
||||
let snapshot = FlowReadySnapshot(
|
||||
sessionId: sessionId,
|
||||
ready: true,
|
||||
reason: .ready,
|
||||
heartbeatAt: now,
|
||||
readyAt: now,
|
||||
audioProofAt: now,
|
||||
engineMode: "local",
|
||||
localeId: "zh-Hans",
|
||||
sessionExpiresAt: now + 60
|
||||
)
|
||||
|
||||
FlowSessionBridge.writeReadySnapshot(snapshot, defaults: defaults)
|
||||
|
||||
XCTAssertEqual(FlowSessionBridge.readySnapshot(defaults: defaults), snapshot)
|
||||
XCTAssertTrue(FlowSessionBridge.isHostReady(defaults: defaults))
|
||||
}
|
||||
|
||||
func testClearFlowStateRemovesProtocolPayloads() {
|
||||
let defaults = makeDefaults()
|
||||
let sessionId = UUID()
|
||||
let utteranceId = UUID()
|
||||
FlowSessionBridge.writeCommand(
|
||||
FlowCommand(
|
||||
sessionId: sessionId,
|
||||
utteranceId: utteranceId,
|
||||
commandSeq: 1,
|
||||
action: .startRecording,
|
||||
localeId: "en-US"
|
||||
),
|
||||
defaults: defaults
|
||||
)
|
||||
FlowSessionBridge.writeResult(
|
||||
FlowResult(
|
||||
sessionId: sessionId,
|
||||
utteranceId: utteranceId,
|
||||
commandSeq: 1,
|
||||
status: .partial,
|
||||
text: "hello"
|
||||
),
|
||||
defaults: defaults
|
||||
)
|
||||
|
||||
FlowSessionBridge.clearFlowState(defaults: defaults)
|
||||
|
||||
XCTAssertNil(FlowSessionBridge.latestCommand(defaults: defaults))
|
||||
XCTAssertNil(FlowSessionBridge.latestResult(defaults: defaults))
|
||||
XCTAssertNil(FlowSessionBridge.readySnapshot(defaults: defaults))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,124 @@
|
||||
// LocalASRBiasAdapterTests.swift
|
||||
// OSGKeyboardTests
|
||||
|
||||
import XCTest
|
||||
@testable import OSGKeyboardShared
|
||||
|
||||
final class LocalASRBiasAdapterTests: XCTestCase {
|
||||
|
||||
private func makeFixtureLexicon() throws -> BuiltinLexiconIndex {
|
||||
let dir = FileManager.default.temporaryDirectory
|
||||
.appendingPathComponent("osg-phrases-\(UUID().uuidString)", isDirectory: true)
|
||||
try FileManager.default.createDirectory(at: dir, withIntermediateDirectories: true)
|
||||
let url = dir.appendingPathComponent("phrases.tsv")
|
||||
let tsv = """
|
||||
word\tpinyin\tsource\tweight
|
||||
SwiftUI\tswift ui\tcomputer_terms\t5
|
||||
Kubernetes\tku bo ne si\tcomputer_terms\t5
|
||||
一致性\tyi zhi xing\tcomputer_terms\t5
|
||||
"""
|
||||
try tsv.write(to: url, atomically: true, encoding: .utf8)
|
||||
addTeardownBlock {
|
||||
try? FileManager.default.removeItem(at: dir)
|
||||
}
|
||||
return BuiltinLexiconIndex(fixtureURL: url)
|
||||
}
|
||||
|
||||
func testAdaptBuildsPromptBiasForQwen3MLX() throws {
|
||||
let lexicon = try makeFixtureLexicon()
|
||||
var dict = PersonalDictionary.empty
|
||||
_ = dict.upsertManual(term: "Cursor")
|
||||
dict.updateAliases(for: dict.entries[0].id, aliases: ["cursor"])
|
||||
|
||||
let payload = LocalASRBiasAdapter.adapt(
|
||||
LocalASRBiasRequest(
|
||||
dictionary: dict,
|
||||
locale: Locale(identifier: "zh-CN"),
|
||||
capabilities: .qwen3MLX
|
||||
),
|
||||
lexicon: lexicon
|
||||
)
|
||||
|
||||
XCTAssertNotNil(payload.promptBias)
|
||||
XCTAssertTrue(payload.promptBias?.contains("Cursor") == true)
|
||||
XCTAssertTrue(payload.promptBias?.contains("SwiftUI") == true)
|
||||
XCTAssertEqual(payload.diagnostics.userTermCount, 2) // OSGKeyboard system + Cursor
|
||||
XCTAssertGreaterThan(payload.diagnostics.builtinTermCount, 0)
|
||||
}
|
||||
|
||||
func testAdaptProducesPolishFragmentWithoutUserDuplicates() throws {
|
||||
let lexicon = try makeFixtureLexicon()
|
||||
var dict = PersonalDictionary.empty
|
||||
_ = dict.upsertManual(term: "SwiftUI")
|
||||
|
||||
let payload = LocalASRBiasAdapter.adapt(
|
||||
LocalASRBiasRequest(
|
||||
dictionary: dict,
|
||||
locale: Locale(identifier: "zh-CN"),
|
||||
capabilities: .qwen3MLX
|
||||
),
|
||||
lexicon: lexicon
|
||||
)
|
||||
|
||||
XCTAssertFalse(payload.polishFragment.contains("SwiftUI"))
|
||||
XCTAssertTrue(payload.polishFragment.contains("Kubernetes"))
|
||||
}
|
||||
|
||||
func testCorrectionPairsFromAliases() {
|
||||
var dict = PersonalDictionary.empty
|
||||
_ = dict.upsertManual(term: "Kubernetes")
|
||||
dict.updateAliases(for: dict.entries[0].id, aliases: ["k8s"])
|
||||
|
||||
let payload = LocalASRBiasAdapter.adapt(
|
||||
LocalASRBiasRequest(
|
||||
dictionary: dict,
|
||||
locale: Locale(identifier: "zh-CN"),
|
||||
capabilities: .qwen3MLX
|
||||
),
|
||||
lexicon: BuiltinLexiconIndex.shared
|
||||
)
|
||||
|
||||
XCTAssertEqual(payload.correctionPairs.count, 1)
|
||||
XCTAssertEqual(payload.correctionPairs[0].alias, "k8s")
|
||||
XCTAssertEqual(payload.correctionPairs[0].term, "Kubernetes")
|
||||
}
|
||||
|
||||
func testTranscriptCorrectorReplacesASCIIAlias() {
|
||||
let pairs = [LocalASRCorrectionPair(alias: "k8s", term: "Kubernetes")]
|
||||
let result = LocalASRTranscriptCorrector.apply(
|
||||
"部署 k8s 集群",
|
||||
pairs: pairs
|
||||
)
|
||||
XCTAssertEqual(result, "部署 Kubernetes 集群")
|
||||
}
|
||||
|
||||
func testTranscriptCorrectorSkipsPartialASCIIMatch() {
|
||||
let pairs = [LocalASRCorrectionPair(alias: "k8s", term: "Kubernetes")]
|
||||
let result = LocalASRTranscriptCorrector.apply(
|
||||
"xk8s集群",
|
||||
pairs: pairs
|
||||
)
|
||||
XCTAssertEqual(result, "xk8s集群")
|
||||
}
|
||||
|
||||
func testBuiltinLexiconParsesTSV() {
|
||||
let terms = BuiltinLexiconIndex.parseTSV(
|
||||
"word\tpinyin\tsource\tweight\nFoo\tfoo\tcomputer_terms\t5\n"
|
||||
)
|
||||
XCTAssertEqual(terms.count, 1)
|
||||
XCTAssertEqual(terms[0].word, "Foo")
|
||||
XCTAssertEqual(terms[0].weight, 5)
|
||||
}
|
||||
|
||||
func testPolishingServiceMergesDictionarySupplement() {
|
||||
let dict = PersonalDictionary(entries: [
|
||||
PersonalDictionary.Entry(term: "Cursor", category: .productName, source: .manual),
|
||||
])
|
||||
let merged = PolishingService.mergedDictionaryBlock(
|
||||
dictionary: dict,
|
||||
supplement: "内置技术词汇参考:SwiftUI"
|
||||
)
|
||||
XCTAssertTrue(merged.contains("Cursor"))
|
||||
XCTAssertTrue(merged.contains("SwiftUI"))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,88 @@
|
||||
// LocalASRModelCatalogTests.swift
|
||||
// OSGKeyboardTests
|
||||
|
||||
import XCTest
|
||||
@testable import OSGKeyboardShared
|
||||
|
||||
final class LocalASRModelCatalogTests: XCTestCase {
|
||||
|
||||
func testBundledCatalogLoads() throws {
|
||||
let catalog = try LocalASRModelCatalog.loadBundled()
|
||||
XCTAssertEqual(catalog.schemaVersion, 1)
|
||||
XCTAssertFalse(catalog.models.isEmpty)
|
||||
XCTAssertTrue(catalog.models.contains { $0.id == "qwen3-mlx-1.7b" })
|
||||
XCTAssertTrue(catalog.models.contains { $0.id == "sherpa-qwen3-0.6b-int8" })
|
||||
}
|
||||
|
||||
func testCapabilitiesForSherpaQwen3() throws {
|
||||
let catalog = try LocalASRModelCatalog.loadBundled()
|
||||
let model = try XCTUnwrap(LocalASRModelCatalog.model("sherpa-qwen3-0.6b-int8", in: catalog))
|
||||
let caps = LocalASRModelCatalog.capabilities(for: model)
|
||||
XCTAssertEqual(caps.hotwordMode, .recognizerScoped)
|
||||
XCTAssertTrue(model.supportsHotwords)
|
||||
}
|
||||
|
||||
func testManifestRoundTrip() throws {
|
||||
let manifest = LocalASRInstalledManifest(
|
||||
selectedModelId: "sherpa-qwen3-0.6b-int8",
|
||||
installedModelIDs: ["sherpa-qwen3-0.6b-int8"]
|
||||
)
|
||||
let url = FileManager.default.temporaryDirectory
|
||||
.appendingPathComponent("manifest-\(UUID().uuidString).json")
|
||||
defer { try? FileManager.default.removeItem(at: url) }
|
||||
|
||||
let encoder = JSONEncoder()
|
||||
encoder.dateEncodingStrategy = .iso8601
|
||||
try encoder.encode(manifest).write(to: url)
|
||||
|
||||
let decoder = JSONDecoder()
|
||||
decoder.dateDecodingStrategy = .iso8601
|
||||
let loaded = try decoder.decode(LocalASRInstalledManifest.self, from: Data(contentsOf: url))
|
||||
XCTAssertEqual(loaded.selectedModelId, manifest.selectedModelId)
|
||||
XCTAssertEqual(loaded.installedModelIDs, manifest.installedModelIDs)
|
||||
}
|
||||
|
||||
func testBiasDiagnosticsStoreRoundTrip() {
|
||||
LocalASRBiasDiagnosticsStore.clear()
|
||||
let payload = LocalASRBiasPayload(
|
||||
hardHotwords: ["Cursor"],
|
||||
promptBias: "test",
|
||||
corpusContext: nil,
|
||||
polishFragment: "fragment",
|
||||
correctionPairs: [],
|
||||
diagnostics: LocalASRBiasDiagnostics(userTermCount: 2, builtinTermCount: 3)
|
||||
)
|
||||
LocalASRBiasDiagnosticsStore.save(
|
||||
payload: payload,
|
||||
modelId: "qwen3-mlx-1.7b",
|
||||
backendLabel: "MLX"
|
||||
)
|
||||
let snapshot = LocalASRBiasDiagnosticsStore.load()
|
||||
XCTAssertEqual(snapshot?.modelId, "qwen3-mlx-1.7b")
|
||||
XCTAssertEqual(snapshot?.diagnostics.userTermCount, 2)
|
||||
XCTAssertEqual(snapshot?.hotwordCount, 1)
|
||||
LocalASRBiasDiagnosticsStore.clear()
|
||||
}
|
||||
|
||||
func testSherpaAdapterProducesHardHotwords() throws {
|
||||
let fixtureURL = FileManager.default.temporaryDirectory
|
||||
.appendingPathComponent("phrases-\(UUID().uuidString).tsv")
|
||||
try "word\tpinyin\tsource\tweight\nSwiftUI\tswift ui\tcomputer_terms\t5\n"
|
||||
.write(to: fixtureURL, atomically: true, encoding: .utf8)
|
||||
defer { try? FileManager.default.removeItem(at: fixtureURL) }
|
||||
|
||||
var dict = PersonalDictionary.empty
|
||||
_ = dict.upsertManual(term: "Kubernetes")
|
||||
|
||||
let payload = LocalASRBiasAdapter.adapt(
|
||||
LocalASRBiasRequest(
|
||||
dictionary: dict,
|
||||
locale: Locale(identifier: "zh-CN"),
|
||||
capabilities: .sherpaQwen3
|
||||
),
|
||||
lexicon: BuiltinLexiconIndex(fixtureURL: fixtureURL)
|
||||
)
|
||||
XCTAssertFalse(payload.hardHotwords.isEmpty)
|
||||
XCTAssertTrue(payload.hardHotwords.contains("Kubernetes"))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,585 @@
|
||||
# OSGKeyboard 本地 ASR 技术架构
|
||||
|
||||
> **文档状态**:架构规划(非实现规格)
|
||||
> **适用范围**:macOS 本地听写;与 iOS 键盘扩展、云 ASR 路径的关系见各节说明。
|
||||
> **核心结论**:短期不换主模型,优先打通 **词库感知管道**;中期用 POC 验证 **Sherpa Qwen3 hard hotwords** 是否值得成为热词主线。
|
||||
|
||||
---
|
||||
|
||||
## 1. Executive Summary
|
||||
|
||||
OSGKeyboard 的本地 ASR 竞争力不来自单一模型,而来自:
|
||||
|
||||
1. **用户 PersonalDictionary**(term / aliases / iCloud)
|
||||
2. **内置技术词库**(`phrases.tsv` ≈ 1 万词,iOS 已用于 Apple CLM)
|
||||
3. **分层 bias**:ASR 偏置 → 后处理纠错 → Polish 保真
|
||||
4. **可替换的 Local ASR Provider**(Qwen3 MLX 主线,Sherpa / SenseVoice / Apple Speech 对照)
|
||||
|
||||
当前最大缺口:**macOS 本地路径未消费任何词库**;云路径已通过 `PersonalDictionary+ASRBias` 完整接线。
|
||||
|
||||
推荐路线:
|
||||
|
||||
| 阶段 | 动作 |
|
||||
|------|------|
|
||||
| **短期** | 保留 Qwen3 MLX;实现 `LocalASRBiasAdapter`;接 soft prompt + polish + aliases 后处理 |
|
||||
| **中期** | ModelScope 优先的本地模型 catalog;Sherpa Qwen3 hotwords POC |
|
||||
| **长期** | 按评测数据决定是否新增默认 provider 或保留 Qwen3 MLX |
|
||||
|
||||
---
|
||||
|
||||
## 2. 背景与问题定义
|
||||
|
||||
### 2.1 为什么本地 ASR 不能只讨论模型
|
||||
|
||||
语音输入的「专有名词准确率」由多层共同决定:
|
||||
|
||||
- **ASR 层**:听出 `Claude`、`SwiftUI`、`Qwen3-ASR`
|
||||
- **后处理层**:`克劳德` → `Claude`
|
||||
- **润色层**:保留品牌名、变量名,不擅自改写
|
||||
|
||||
闭源产品(Typeless 等)常把词典效果归因于云端 ASR;开源竞品(OpenLess、Typeflux、SayIt)表明:**词典必须按 backend 能力分层注入**,不能假设「一个 hotwords 数组走天下」。
|
||||
|
||||
### 2.2 OSG 相对竞品的结构性优势
|
||||
|
||||
| 能力 | OSGKeyboard | 典型开源竞品 |
|
||||
|------|-------------|----------------|
|
||||
| 用户词库 | `PersonalDictionary`(term + aliases + category + iCloud) | 多为 phrase-only |
|
||||
| 内置领域词库 | ~10k `phrases.tsv` + iOS CLM | OpenLess preset ~20 词;SayIt server hotwords.txt ~30 词 |
|
||||
| 云 ASR bias | 智谱 / 阿里 vocabulary / Whisper prompt | 单云或单 provider |
|
||||
| iOS 本地 CLM | `SFCustomLanguageModelData` | macOS 路径未等价 |
|
||||
|
||||
### 2.3 设计目标
|
||||
|
||||
- 离线、隐私友好的 macOS 本地听写
|
||||
- 复用 `PersonalDictionary` 与 `phrases.tsv` **源数据**(非 iOS `.bin` 直用)
|
||||
- Provider 可替换;能力矩阵诚实声明(尤其热词模式)
|
||||
- 模型下载可管理(**ModelScope 优先**,HF / GitHub 备用)
|
||||
- 可评测、可灰度、可回退
|
||||
|
||||
### 2.4 非目标
|
||||
|
||||
- 不立即将主路径切到 FunASR Python server 或 Sherpa
|
||||
- 不把 1 万词全量塞入 ASR prompt
|
||||
- 不把 Polish 当作唯一纠错层
|
||||
- 不承诺未 POC 验证的模型效果
|
||||
- 第一期不强制实现 Typeflux 式「自动词库学习」(仅作可选实验设计)
|
||||
|
||||
---
|
||||
|
||||
## 3. 当前架构(代码事实)
|
||||
|
||||
### 3.1 端到端数据流
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph macOS["macOS"]
|
||||
Rec["MacAudioRecorder"]
|
||||
Pipe["MacDictationPipeline"]
|
||||
Local["MacLocalASRService"]
|
||||
Cloud["CloudASRClient"]
|
||||
Polish["PolishingService"]
|
||||
Insert["MacTextInsertionService"]
|
||||
end
|
||||
Rec --> Pipe
|
||||
Pipe -->|engineMode local| Local
|
||||
Pipe -->|engineMode cloud| Cloud
|
||||
Local --> Polish
|
||||
Cloud --> Polish
|
||||
Polish --> Insert
|
||||
```
|
||||
|
||||
### 3.2 云路径 vs 本地路径
|
||||
|
||||
| 环节 | 云 ASR | 本地 ASR(当前) |
|
||||
|------|--------|------------------|
|
||||
| 入口 | `MacDictationPipeline.run` | 同左 |
|
||||
| ASR | `CloudASRClientFactory` + `dictionary: store.personalDictionary` | `MacLocalASRService.transcribe(samples, locale)` **无 dictionary** |
|
||||
| 词库 bias | `PersonalDictionary+ASRBias`(按 provider) | **无** |
|
||||
| 润色 | `PolishingService` + `promptFragment()` | 同左(仅用词典做 polish,不经 ASR) |
|
||||
| 默认模型 | 用户所选云 provider | Qwen3 MLX 1.7B;缺权重 → Apple Speech |
|
||||
|
||||
关键代码:
|
||||
|
||||
- [`OSGKeyboardMac/MacDictationPipeline.swift`](../OSGKeyboardMac/MacDictationPipeline.swift) — 本地分支未传 `personalDictionary`
|
||||
- [`OSGKeyboardMac/MacLocalASRService.swift`](../OSGKeyboardMac/MacLocalASRService.swift) — Qwen3 / Apple Speech 二选一
|
||||
- [`OSGKeyboardShared/Models/PersonalDictionary+ASRBias.swift`](../OSGKeyboardShared/Models/PersonalDictionary+ASRBias.swift) — 云侧 `asrHotwords` / `asrPromptBias` / 阿里热词表
|
||||
- [`OSGKeyboardShared/Services/PolishingService.swift`](../OSGKeyboardShared/Services/PolishingService.swift) — 润色层消费 `dictionary.promptFragment()`
|
||||
|
||||
### 3.3 iOS 词库资产(macOS 不可直接复用)
|
||||
|
||||
- **内置词库**:[`OSGKeyboard/Resources/CustomLanguageModel/v1/phrases.tsv`](../OSGKeyboard/Resources/CustomLanguageModel/v1/phrases.tsv)(约 10,301 行,含 `word` / `pinyin` / `source` / `weight`)
|
||||
- **Apple CLM**:[`OSGKeyboardShared/Services/CustomLanguageModelManager.swift`](../OSGKeyboardShared/Services/CustomLanguageModelManager.swift) → `SFSpeechLanguageModel.Configuration`,仅 iOS 26+ 中文本地 ASR 路径
|
||||
- **结论**:macOS 需从 TSV + `PersonalDictionary` **重新适配**为 Qwen3 prompt / Sherpa hotwords / polish fragment,不能加载 `OSGKeyboardCLM.bin`
|
||||
|
||||
### 3.4 本地模型现状
|
||||
|
||||
- 默认路径:`~/Library/Application Support/OSGKeyboard/models/qwen3-asr-1.7b-mlx`
|
||||
- 校验:`config.json`、`model.safetensors`、`vocab.json`、`merges.txt`
|
||||
- Qwen3 引擎:[`MacQwen3LocalASR.swift`](../OSGKeyboardMac/MacQwen3LocalASR.swift) / [`MacQwen3ASREngine.swift`](../OSGKeyboardMac/MacQwen3ASREngine.swift) — **无 prompt / hotword 入参**
|
||||
|
||||
---
|
||||
|
||||
## 4. 开源竞品源码观察
|
||||
|
||||
基于 GitHub topic `typeless-alternative` 及关联仓库(2026-03 快照)。
|
||||
|
||||
### 4.1 对比总表
|
||||
|
||||
| 项目 | 技术栈 | 本地 ASR | 热词 / 词库 | 可借鉴 | 不宜照搬 |
|
||||
|------|--------|----------|-------------|--------|----------|
|
||||
| **OpenLess** | Tauri/Rust | Qwen3 C 引擎、Apple Speech、Sherpa(Win) | 火山 `context.hotwords`;Whisper prompt;Polish hotword block;**本地 Qwen3 未接词典** | 多 provider + polish 双层 | 本地热词叙事过度乐观 |
|
||||
| **Typeflux** | Swift/macOS | SenseVoice、FunASR、Qwen3 Sherpa CLI、WhisperKit | `VocabularyStore` cap 500;Doubao hotwords;Whisper prompt;**Sherpa 本地无热词**;自动项目词 + 编辑后学习 | Swift 原生、词库排序、自动学习(作实验) | Sherpa 仅 CLI 离线,无 hotwords 接线 |
|
||||
| **SayIt** | Tauri + FastAPI | sherpa-onnx Rust:**Qwen3 recognizer 创建时写 hotwords** | 内置主题词 + 自定义;云 Qwen corpus;server `hotwords.txt` | **本地 Qwen3 hard hotwords 实证**;跨 provider `StartOptions.hotwords` | 服务端 vLLM 非 macOS 客户端主线 |
|
||||
| **VoiceSnap** | Go/Wails | SenseVoice + sherpa-onnx | **无个性化词库** | 离线体验:静音截断、剪贴板保护、填充词过滤 | 无万级词库场景 |
|
||||
| **OpenBroca** | Electron | Sherpa 等 | Dictionary hotword/replacement | **one-shot `recognize` first**;model catalog、sha256、selected model | Electron 栈 |
|
||||
|
||||
### 4.2 对 OSG 的启示
|
||||
|
||||
1. **学架构,不学「本地已完整支持热词」的 README 叙事**(OpenLess 本地 Qwen3 缺口与 OSG 类似)。
|
||||
2. **SayIt 证明**:sherpa-onnx `OfflineQwen3ASRModelConfig.hotwords` 可在 recognizer 创建时注入;热词变化需 **重建 recognizer**(缓存 key 含 hotwords 字符串)。
|
||||
3. **Typeflux 证明**:`activeTerms()` 上限 500 + 动态排序;但 Sherpa Qwen3/SenseVoice 命令行路径**未传** vocabulary prompt。
|
||||
4. **OpenBroca 证明**:runtime 不得静默选「目录里第一个模型」;须 `selectedModelId` + manifest。
|
||||
|
||||
---
|
||||
|
||||
## 5. 目标架构
|
||||
|
||||
### 5.1 管道总览
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
audioSamples["Audio Samples 16kHz"] --> macPipeline["MacDictationPipeline"]
|
||||
personalDict["PersonalDictionary"] --> biasAdapter["LocalASRBiasAdapter"]
|
||||
phrasesTSV["phrases.tsv Index"] --> biasAdapter
|
||||
runtimeCtx["Runtime Context locale app recentHits"] --> biasAdapter
|
||||
providerCap["Provider Capability"] --> biasAdapter
|
||||
|
||||
biasAdapter --> hotwords["hardHotwords"]
|
||||
biasAdapter --> promptBias["promptBias"]
|
||||
biasAdapter --> corpusText["corpusContext"]
|
||||
biasAdapter --> correctionPairs["correctionPairs"]
|
||||
biasAdapter --> polishFrag["polishFragment"]
|
||||
biasAdapter --> diag["diagnostics"]
|
||||
|
||||
macPipeline --> recognize["LocalASRProvider.recognize"]
|
||||
hotwords --> recognize
|
||||
promptBias --> recognize
|
||||
corpusText --> recognize
|
||||
|
||||
recognize --> rawTranscript["Raw Transcript"]
|
||||
rawTranscript --> correctionLayer["Correction Layer"]
|
||||
correctionPairs --> correctionLayer
|
||||
correctionLayer --> polishingService["PolishingService"]
|
||||
polishFrag --> polishingService
|
||||
polishingService --> finalText["Final Text"]
|
||||
```
|
||||
|
||||
### 5.2 三层职责边界
|
||||
|
||||
| 层 | 职责 | 禁止 |
|
||||
|----|------|------|
|
||||
| **ASR bias** | 提高听写阶段专有名词概率 | 承担全文语法润色 |
|
||||
| **Correction** | 高置信 `aliases → term` 替换 | 凭热词表改写普通句意 |
|
||||
| **Polish** | 标点、口语转书面、热词保真 | 单独承担全部专名纠错 |
|
||||
|
||||
---
|
||||
|
||||
## 6. Local ASR Provider 抽象
|
||||
|
||||
### 6.1 One-shot first
|
||||
|
||||
借鉴 OpenBroca:macOS 听写主路径为 **录完后一次性 `recognize`**;流式预览(`transcribe` / partial)为可选能力,非第一期必做。
|
||||
|
||||
建议协议(概念层):
|
||||
|
||||
```swift
|
||||
protocol LocalASRProvider {
|
||||
var capabilities: LocalASRCapabilities { get }
|
||||
func recognize(
|
||||
samples: [Float],
|
||||
sampleRate: Int,
|
||||
locale: Locale,
|
||||
bias: LocalASRBiasPayload?,
|
||||
options: LocalASRRecognizeOptions?
|
||||
) async throws -> LocalASRResult
|
||||
}
|
||||
```
|
||||
|
||||
### 6.2 能力矩阵(须诚实声明)
|
||||
|
||||
除 `supportsStreaming`、`maxHotwordCount` 外,**必须区分热词模式**:
|
||||
|
||||
| 字段 | 含义 |
|
||||
|------|------|
|
||||
| `hotwordMode` | `none` / `promptOnly` / `perRequest` / `recognizerScoped` / `cloudVocabulary` |
|
||||
| `hotwordStrength` | `weak` / `medium` / `strong`(产品文案用,非科学绝对值) |
|
||||
| `hotwordReloadCost` | `none` / `recognizerReload` / `modelReload` |
|
||||
| `maxPromptCharacters` | soft prompt 上限 |
|
||||
| `maxHotwordCount` | hard hotwords 上限 |
|
||||
| `supportsLanguageHint` | 是否接受 locale → language hint |
|
||||
|
||||
### 6.3 各 Backend 定位(规划)
|
||||
|
||||
| Provider | 角色 | hotwordMode(规划) | 备注 |
|
||||
|----------|------|---------------------|------|
|
||||
| **Qwen3 MLX** | 短期主线 | `promptOnly`(待接 `qwen_set_prompt` 等价 API) | 已有权重路径;改动面最小 |
|
||||
| **Sherpa Qwen3** | 中期 POC | `recognizerScoped` | SayIt 同款;热词变更加载成本 |
|
||||
| **SenseVoice** | 对照 | `none` 或弱 prompt | 速度/中文基线;非热词主线 |
|
||||
| **Apple Speech** | Fallback | `none`(macOS CLM 待验证) | 系统稳定 |
|
||||
| **Cloud ASR** | 质量上限 | 各云 `PersonalDictionary+ASRBias` | 非离线 |
|
||||
|
||||
**暂不主推**:Sherpa Paraformer 作为热词主线(官方不支持 Paraformer hotwords,与 transducer/Qwen3 不同)。
|
||||
|
||||
---
|
||||
|
||||
## 7. LocalASRBiasAdapter 设计
|
||||
|
||||
### 7.1 输入
|
||||
|
||||
| 输入 | 说明 |
|
||||
|------|------|
|
||||
| `PersonalDictionary.effectiveEntries` | 用户词;最高优先级 |
|
||||
| `BuiltinLexiconIndex` | 自 `phrases.tsv` 构建;按 weight / 场景筛选 |
|
||||
| `locale` | `store.localeId` |
|
||||
| `frontAppBundleId` | 可选;技术类 App 提升 IT 子集权重 |
|
||||
| `recentHitTerms` | 历史命中统计(若已有) |
|
||||
| `providerCapabilities` | 决定输出哪些字段、如何截断 |
|
||||
|
||||
### 7.2 输出 `LocalASRBiasPayload`
|
||||
|
||||
```swift
|
||||
struct LocalASRBiasPayload {
|
||||
var hardHotwords: [String] // Sherpa Qwen3、部分云 API
|
||||
var promptBias: String? // Qwen3 MLX、Whisper 系
|
||||
var corpusContext: String? // Qwen 云 corpus 风格(若将来统一)
|
||||
var polishFragment: String // PolishingService 追加块
|
||||
var correctionPairs: [(alias: String, term: String)]
|
||||
var diagnostics: BiasDiagnostics // 供设置页 / 调试
|
||||
}
|
||||
|
||||
struct BiasDiagnostics {
|
||||
var userTermCount: Int
|
||||
var builtinTermCount: Int
|
||||
var truncated: Bool
|
||||
var truncationReason: String?
|
||||
var selectedSources: [String] // e.g. user, builtin-it, builtin-top
|
||||
}
|
||||
```
|
||||
|
||||
### 7.3 优先级与截断
|
||||
|
||||
```
|
||||
用户高频 / 最近命中
|
||||
> PersonalDictionary(全部有效 term)
|
||||
> 当前 App 相关内置词(phrases 子集)
|
||||
> 高 weight 内置技术词(Top-N)
|
||||
> 其余内置词(仅 polish / 检索,不进 ASR)
|
||||
```
|
||||
|
||||
默认建议(可 POC 调参):
|
||||
|
||||
| 输出 | 默认上限 |
|
||||
|------|----------|
|
||||
| `hardHotwords` | 100(Qwen3 Sherpa);对齐 `asrHotwords(maxCount: 100)` |
|
||||
| `promptBias` | 800 字符;复用 `asrPromptBias(maxCharacters:)` 逻辑 |
|
||||
| ASR 层内置词 | 200–500;**不全量 1 万** |
|
||||
| `correctionPairs` | aliases 全量可进后处理,但仅 **整词 / 高置信** 替换 |
|
||||
|
||||
### 7.4 防污染规则
|
||||
|
||||
- 近静音、极短音频:减少或跳过内置词,保留用户词。
|
||||
- 用户词始终优先于内置词。
|
||||
- diagnostics 必须记录「为何丢弃」某批词(超 cap、provider 不支持、场景不匹配)。
|
||||
|
||||
### 7.5 与现有云代码复用
|
||||
|
||||
扩展 [`PersonalDictionary+ASRBias.swift`](../OSGKeyboardShared/Models/PersonalDictionary+ASRBias.swift) 为 **单一事实来源**,新增例如:
|
||||
|
||||
- `localPromptBias(maxCharacters:builtinTerms:)`
|
||||
- `correctionPairs()`
|
||||
- `rankedTermsForASR(limit:builtinBoost:)`
|
||||
|
||||
避免 macOS / iOS / Cloud 三套独立拼接逻辑。
|
||||
|
||||
---
|
||||
|
||||
## 8. 词库策略
|
||||
|
||||
### 8.1 PersonalDictionary
|
||||
|
||||
| 字段 | ASR | Correction | Polish |
|
||||
|------|-----|------------|--------|
|
||||
| `term` | hotword / prompt | 标准写法 | 必须保留 |
|
||||
| `aliases` | 可进 prompt 提示 | **主战场** | 语义纠错参考 |
|
||||
| `category` | 排序权重 | — | 分组展示 |
|
||||
| `usageCount` | 排序权重 | — | — |
|
||||
|
||||
iCloud:[`PersonalDictionaryCloudSync`](../OSGKeyboardShared/Services/PersonalDictionaryCloudSync/) 保证 Mac / iOS / Extension 一致;本地 ASR 只读 `AppGroupStore.personalDictionary`。
|
||||
|
||||
### 8.2 phrases.tsv 分层
|
||||
|
||||
**不全量进入 ASR prompt。**
|
||||
|
||||
| 层级 | 用途 | 规模建议 |
|
||||
|------|------|----------|
|
||||
| L1 ASR 高价值 | `weight >= 4` 或 curated IT 品牌缩写 | 200–500 |
|
||||
| L2 场景相关 | 按 `frontApp` / 用户最近命中动态加入 | +0–100 |
|
||||
| L3 全量索引 | 后处理模糊匹配、polish 检索 | ~10k |
|
||||
|
||||
TSV 列:`word`、`pinyin`、`source`、`weight` — 构建索引时保留 weight 用于排序。
|
||||
|
||||
### 8.3 iOS CLM 与 macOS 关系
|
||||
|
||||
- iOS:TSV → export script → `.bin` → `CustomLanguageModelManager`
|
||||
- macOS:TSV → `BuiltinLexiconIndex` → `LocalASRBiasAdapter` → Qwen3 / Sherpa / Polish
|
||||
- **同一 TSV 源**,两种消费格式;不尝试把 `.bin` 喂给 Sherpa/MLX
|
||||
|
||||
---
|
||||
|
||||
## 9. 自动词库学习(可选实验,非第一期)
|
||||
|
||||
借鉴 Typeflux `WorkflowController+AutomaticVocabulary`:
|
||||
|
||||
- 听写插入后,短时观察用户在前台可编辑框内的修改
|
||||
- LLM 或规则判断是否为「专名 / 品牌 / 大小写修正」
|
||||
- 候选进入 **待确认队列**,不直接写入 `PersonalDictionary`
|
||||
|
||||
**约束(必须写进隐私说明)**:
|
||||
|
||||
- 默认关闭
|
||||
- 不自动 iCloud 同步待确认项
|
||||
- 可一键清空、可审计来源
|
||||
- 拒绝:整句改写、纯语法修正、过短词条、编辑幅度过大
|
||||
|
||||
OSG 已有 `PersonalDictionary.Entry.Source.recentEdit` 与合并逻辑,可与之对齐而非新建平行存储。
|
||||
|
||||
---
|
||||
|
||||
## 10. 本地模型管理
|
||||
|
||||
### 10.1 原则
|
||||
|
||||
- **Catalog 与 Runtime 分离**:下载源只影响安装;推理只读本地 **已验证 manifest**
|
||||
- **Selected model 显式**:禁止「扫描目录用第一个 onnx」
|
||||
- **完整性**:sha256 或 size 校验 + staging 目录原子发布
|
||||
|
||||
### 10.2 存储布局(建议)
|
||||
|
||||
```
|
||||
~/Library/Application Support/OSGKeyboard/
|
||||
LocalASRModels/
|
||||
manifest.json # 已安装模型、版本、backend、capabilities
|
||||
qwen3-mlx-1.7b/ # 当前 MLX 布局(可与现路径兼容)
|
||||
sherpa-qwen3-0.6b/
|
||||
sherpa-sensevoice-small/
|
||||
```
|
||||
|
||||
### 10.3 Catalog 条目(概念)
|
||||
|
||||
```json
|
||||
{
|
||||
"modelId": "sherpa-qwen3-asr-0.6b-int8",
|
||||
"displayName": "Qwen3-ASR 0.6B (Sherpa)",
|
||||
"backend": "sherpaQwen3",
|
||||
"sizeBytes": 1200000000,
|
||||
"recommendedLocales": ["zh-CN", "en-US"],
|
||||
"supportsHotwords": true,
|
||||
"hotwordMode": "recognizerScoped",
|
||||
"sources": [
|
||||
{
|
||||
"type": "modelscope",
|
||||
"url": "https://www.modelscope.cn/api/v1/models/.../repo?Revision=master&FilePath=...",
|
||||
"sha256": "...",
|
||||
"priority": 1
|
||||
},
|
||||
{
|
||||
"type": "huggingface",
|
||||
"url": "https://huggingface.co/...",
|
||||
"priority": 2
|
||||
},
|
||||
{
|
||||
"type": "github",
|
||||
"url": "https://github.com/k2-fsa/sherpa-onnx/releases/download/...",
|
||||
"priority": 3
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### 10.4 ModelScope 策略
|
||||
|
||||
| 场景 | 策略 |
|
||||
|------|------|
|
||||
| 中国大陆用户默认 | **ModelScope 优先**(Qwen3-ASR、SenseVoice、FunASR 相关 ONNX) |
|
||||
| 国际 / ModelScope 失败 | Hugging Face → GitHub Releases |
|
||||
| 企业内网 | `custom` mirror URL(用户配置) |
|
||||
| 安装流程 | download → verify → extract → validate required files → rename staging → update manifest |
|
||||
| 失败 | 清理 staging / 临时文件;不留下半安装状态 |
|
||||
|
||||
MLX Qwen3 权重:可继续支持用户自选目录(现状),逐步纳入统一 catalog 的 `type: mlx` 条目。
|
||||
|
||||
### 10.5 UI / 设置需求(规划)
|
||||
|
||||
- 模型列表:体积、语言、安装状态、是否支持热词
|
||||
- 下载进度:phase(downloading / extracting / validating / finalizing)
|
||||
- 切换模型:仅允许 **installed + manifest 合法** 的项为默认
|
||||
- 诊断:当前 provider、capability、上次 bias diagnostics
|
||||
|
||||
---
|
||||
|
||||
## 11. 后端对比与决策矩阵
|
||||
|
||||
### 11.1 产品分层
|
||||
|
||||
```text
|
||||
主线: Qwen3 MLX + LocalASRBiasAdapter + Polish/Correction
|
||||
重点 POC: Sherpa Qwen3 hotwords
|
||||
对照: SenseVoice(速度)、Apple Speech(fallback)
|
||||
参考上限: Cloud ASR + PersonalDictionary
|
||||
团队部署: Qwen3-ASR vLLM(SayIt 式,非客户端主线)
|
||||
暂不主推: FunASR Paraformer hotwords、纯 OpenLess 本地词典叙事
|
||||
```
|
||||
|
||||
### 11.2 详细对比
|
||||
|
||||
| 维度 | Qwen3 MLX | Sherpa Qwen3 | SenseVoice | Apple Speech | Cloud |
|
||||
|------|-----------|--------------|------------|--------------|-------|
|
||||
| 离线 | ✅ | ✅ | ✅ | ✅ | ❌ |
|
||||
| 中英混合技术词 | 强(经验性) | 待 POC | 中 | 中 | 强 |
|
||||
| Hard hotwords | ❌→prompt | ✅ recognizerScoped | ❌ | ❌ | ✅ 因 provider 异 |
|
||||
| 实现成本 | 低(已有) | 高(runtime 体积) | 中 | 低 | 已有 |
|
||||
| 模型体积 | ~1.3GB+ | 类似 | ~350MB 级 | 0 | N/A |
|
||||
| 隐私 | 本地 | 本地 | 本地 | 本地 | 依配置 |
|
||||
|
||||
### 11.3 Sherpa POC 通过阈值(建议)
|
||||
|
||||
相对 **当前 Qwen3 MLX + 仅 polish** 基线:
|
||||
|
||||
| 指标 | 建议阈值 |
|
||||
|------|----------|
|
||||
| 用户热词召回率 | 提升 ≥ 20% |
|
||||
| 误触发率(未说热词却被改成热词) | ≤ 2% |
|
||||
| 30s 音频端到端延迟 | ≤ 基线 × 1.5 |
|
||||
| 内存峰值(8GB Mac 目标机) | 可接受且无 OOM |
|
||||
| 安装成功率 | 普通用户可完成 ModelScope/HF 下载 |
|
||||
|
||||
未达阈值:**保留 Qwen3 MLX 主线**,Sherpa 仅作高级选项。
|
||||
|
||||
---
|
||||
|
||||
## 12. POC 评测计划
|
||||
|
||||
### 12.1 测试集
|
||||
|
||||
| 类别 | 内容 | 目的 |
|
||||
|------|------|------|
|
||||
| A 普通中文 | 日常口语 50 句 | 基线 WER / 误触发 |
|
||||
| B 技术术语 | SwiftUI、Cursor、Qwen3-ASR 等 50 句 | 专名召回 |
|
||||
| C 用户词典 | 模拟 PersonalDictionary 20 词 × 多句 | 热词核心场景 |
|
||||
| D 长句润色 | 30s+ 口语 | polish 兜底 |
|
||||
| E 噪声 / 短句 | 低 SNR、<2s | 防污染规则 |
|
||||
|
||||
### 12.2 对照矩阵
|
||||
|
||||
| 配置 | 说明 |
|
||||
|------|------|
|
||||
| Baseline | Qwen3 MLX,无 bias |
|
||||
| B1 | Qwen3 MLX + promptBias |
|
||||
| B2 | B1 + polishFragment + correction |
|
||||
| POC1 | Sherpa Qwen3 + hardHotwords |
|
||||
| POC2 | SenseVoice,无 hotwords |
|
||||
| Ref | 云 ASR + PersonalDictionary |
|
||||
|
||||
### 12.3 指标
|
||||
|
||||
- Raw CER/WER(中文可用字错误率)
|
||||
- **Hotword recall**(用户词是否出现在 raw 或 final)
|
||||
- **False hotword rate**
|
||||
- Final accuracy(用户主观或编辑距离)
|
||||
- Latency:record end → text inserted
|
||||
- Memory / CPU、模型加载时间
|
||||
- 离线可靠性(无网络完成全流程)
|
||||
|
||||
---
|
||||
|
||||
## 13. 失败回退策略
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
start["recognize 开始"] --> qwen{"Qwen3 MLX 可用?"}
|
||||
qwen -->|是| qwenRun["Qwen3 + bias"]
|
||||
qwen -->|否| apple["Apple Speech"]
|
||||
qwenRun -->|失败| apple
|
||||
qwenRun -->|成功| post["Correction + Polish"]
|
||||
apple --> post
|
||||
post -->|Polish 失败| raw["返回 raw transcript"]
|
||||
post -->|成功| done["插入 final"]
|
||||
```
|
||||
|
||||
| 条件 | 行为 |
|
||||
|------|------|
|
||||
| Qwen3 权重缺失 | Apple Speech(现状) |
|
||||
| Qwen3 推理失败 | 可配置:重试一次 → Apple Speech |
|
||||
| Sherpa 未安装 | 不回退云;提示下载 |
|
||||
| 模型 manifest 损坏 | 标记 invalid,禁止设为默认 |
|
||||
| Polish 失败 | 使用 raw(现状) |
|
||||
| 用户禁用云 | 不静默切云 |
|
||||
|
||||
---
|
||||
|
||||
## 14. 分阶段落地路线
|
||||
|
||||
| Phase | 内容 | 交付物 |
|
||||
|-------|------|--------|
|
||||
| **1** | 本文档定稿;`LocalASRCapabilities` + `LocalASRBiasPayload` 类型设计 | 架构文档 + ADR 可选 |
|
||||
| **2** | `MacDictationPipeline` 接入 adapter;Qwen3 MLX `promptBias`;polish + correction | 实现 PR |
|
||||
| **3** | `BuiltinLexiconIndex`;Top-N;diagnostics UI | 实现 PR |
|
||||
| **4** | Model catalog + ModelScope 下载 + manifest | 实现 PR |
|
||||
| **5** | Sherpa Qwen3 POC + 评测报告 | 决策是否默认切换 |
|
||||
| **6** | 可选:自动词库学习实验(默认关) | 功能 flag |
|
||||
|
||||
---
|
||||
|
||||
## 15. 风险与待确认问题
|
||||
|
||||
| 风险 | 缓解 |
|
||||
|------|------|
|
||||
| Qwen3 prompt bias 过弱 | POC 对比 Sherpa hard hotwords;保留 correction + polish |
|
||||
| 热词过多污染识别 | cap + 场景筛选 + diagnostics |
|
||||
| aliases 后处理误改 | 整词边界、低置信跳过 |
|
||||
| Sherpa 分发体积 / 签名 / 公证 | 单独评估;可选按需下载 |
|
||||
| Apple Speech macOS CLM | 调研 macOS 26+ 是否可接 CLM;否则仅 fallback |
|
||||
| ModelScope API 变更 | 多 mirror;manifest 可更新 URL |
|
||||
| 自动学习隐私 | 默认关、本地、待确认 |
|
||||
| 低配 Mac 内存 | 单模型常驻策略;SenseVoice 作轻量选项 |
|
||||
|
||||
**待确认**:
|
||||
|
||||
1. MLX Swift API 是否暴露等价 `setPrompt`(对标 Open-Less/qwen-asr `qwen_set_prompt`)
|
||||
2. Sherpa-ONNX Swift/SPM 与 App Store 公证路径
|
||||
3. `phrases.tsv` Top-N 是否按 `weight` 静态裁剪即可,或需按 App 动态检索
|
||||
|
||||
---
|
||||
|
||||
## 16. 相关代码索引
|
||||
|
||||
| 主题 | 路径 |
|
||||
|------|------|
|
||||
| Mac 听写管道 | `OSGKeyboardMac/MacDictationPipeline.swift` |
|
||||
| 本地 ASR 入口 | `OSGKeyboardMac/MacLocalASRService.swift` |
|
||||
| Qwen3 MLX | `OSGKeyboardMac/MacQwen3LocalASR.swift`, `MacQwen3ASREngine.swift` |
|
||||
| Apple Speech fallback | `OSGKeyboardMac/MacSpeechLocalASR.swift` |
|
||||
| 用户词库 | `OSGKeyboardShared/Models/PersonalDictionary.swift` |
|
||||
| 云 bias | `OSGKeyboardShared/Models/PersonalDictionary+ASRBias.swift` |
|
||||
| 内置 TSV | `OSGKeyboard/Resources/CustomLanguageModel/v1/phrases.tsv` |
|
||||
| iOS CLM | `OSGKeyboardShared/Services/CustomLanguageModelManager.swift` |
|
||||
| 润色 | `OSGKeyboardShared/Services/PolishingService.swift` |
|
||||
| 文本插入 | `OSGKeyboardMac/MacTextInsertionService.swift` |
|
||||
|
||||
---
|
||||
|
||||
## 17. 修订记录
|
||||
|
||||
| 日期 | 说明 |
|
||||
|------|------|
|
||||
| 2026-03-31 | 初版:基于 OSG 代码审计 + typeless-alternative 竞品源码 + 计划评审(ModelScope、hotwordMode、回退策略) |
|
||||
+17
-3
@@ -50,8 +50,8 @@ settings:
|
||||
ASSETCATALOG_COMPILER_GENERATE_SWIFT_ASSET_SYMBOL_EXTENSIONS: YES
|
||||
STRING_CATALOG_GENERATE_SYMBOLS: YES
|
||||
CLANG_CXX_LANGUAGE_STANDARD: c++17
|
||||
MARKETING_VERSION: "0.5.0"
|
||||
CURRENT_PROJECT_VERSION: "18"
|
||||
MARKETING_VERSION: "0.5.2"
|
||||
CURRENT_PROJECT_VERSION: "19"
|
||||
# 签名配置来自 Signing.local.xcconfig(gitignored,不会被覆盖)
|
||||
|
||||
# 项目级签名 xcconfig,适用于所有 target
|
||||
@@ -327,6 +327,7 @@ targets:
|
||||
resources:
|
||||
- path: OSGKeyboardShared/Resources/CustomLanguageModel/v1/OSGKeyboardCLM.bin
|
||||
- path: OSGKeyboardShared/Resources/CustomLanguageModel/v1/compiled-manifest.json
|
||||
- path: OSGKeyboardShared/Resources/LocalASR/local-asr-catalog.json
|
||||
info:
|
||||
path: OSGKeyboardShared/Info.plist
|
||||
settings:
|
||||
@@ -429,10 +430,18 @@ targets:
|
||||
buildPhase: resources
|
||||
- path: OSGKeyboardShared/zh-Hans.lproj/Shared.strings
|
||||
buildPhase: resources
|
||||
- path: OSGKeyboard/Resources/CustomLanguageModel/v1/phrases.tsv
|
||||
buildPhase: resources
|
||||
- path: OSGKeyboardShared/Resources/LocalASR/local-asr-catalog.json
|
||||
buildPhase: resources
|
||||
entitlements:
|
||||
path: OSGKeyboardMac/OSGKeyboardMac.entitlements
|
||||
properties:
|
||||
com.apple.security.app-sandbox: true
|
||||
# Sandbox disabled on purpose: this menu-bar dictation utility uses the
|
||||
# Accessibility API to monitor a global hotkey and inject ⌘V into other
|
||||
# apps — both are forbidden by App Sandbox. Ship via Developer ID +
|
||||
# notarization (not Mac App Store / TestFlight, which force the sandbox).
|
||||
com.apple.security.app-sandbox: false
|
||||
com.apple.security.device.audio-input: true
|
||||
com.apple.security.network.client: true
|
||||
com.apple.developer.ubiquity-kvstore-identifier: $(TeamIdentifierPrefix)com.osgkeyboard.ios
|
||||
@@ -455,6 +464,9 @@ targets:
|
||||
ITSAppUsesNonExemptEncryption: false
|
||||
settings:
|
||||
base:
|
||||
PRODUCT_NAME: OSGKeyboard
|
||||
INFOPLIST_KEY_CFBundleDisplayName: OSGKeyboard
|
||||
INFOPLIST_KEY_CFBundleName: OSGKeyboard
|
||||
PRODUCT_BUNDLE_IDENTIFIER: com.osgkeyboard.mac
|
||||
MACOSX_DEPLOYMENT_TARGET: "15.0"
|
||||
ASSETCATALOG_COMPILER_APPICON_NAME: AppIcon
|
||||
@@ -462,6 +474,8 @@ targets:
|
||||
APPLICATION_EXTENSION_API_ONLY: NO
|
||||
CODE_SIGN_STYLE: Automatic
|
||||
DEVELOPMENT_TEAM: X329MZU23S
|
||||
# Required for Developer ID distribution + notarization (outside App Store).
|
||||
ENABLE_HARDENED_RUNTIME: YES
|
||||
dependencies:
|
||||
- sdk: Speech.framework
|
||||
- sdk: AVFoundation.framework
|
||||
|
||||
Reference in New Issue
Block a user