feat(macos): local ASR model manager, menu-bar polish, and release 0.5.2
Adds a bundled local ASR model catalog for the macOS app with one-click Sherpa Qwen3 / SenseVoice downloads (pause/resume, inline actions) and a shared model storage directory used by MLX Qwen3. Fixes the light-mode sidebar material and makes the menu-bar icon follow the system appearance with a refreshed status mark. Renames the built product to OSGKeyboard.app. Bumps version to 0.5.2 (build 19).
This commit is contained in:
@@ -7,20 +7,29 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||||||
|
|
||||||
## [Unreleased]
|
## [Unreleased]
|
||||||
|
|
||||||
|
## [0.5.2] - 2026-07-09
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
|
- **macOS local ASR models**: the desktop app ships a bundled model catalog with one-click download of Sherpa Qwen3 (hotwords) and SenseVoice models, plus a shared model storage directory; downloads show a circular progress ring with pause / resume, and each row has inline Download / Delete actions. / **macOS 本地 ASR 模型**:桌面 App 内置模型目录,可一键下载 Sherpa Qwen3(热词)与 SenseVoice 模型,并共用同一模型存储目录;下载显示带暂停 / 继续的环形进度,每行提供内联的下载 / 删除操作。
|
||||||
|
- **Shared model directory for MLX**: Qwen3-ASR MLX now uses a fixed subfolder inside the shared model storage — drop converted weights into the folder opened by "Open folder"; no per-model directory picker. / **MLX 共用模型目录**:Qwen3-ASR MLX 改用共享模型存储中的固定子目录——把转换好的权重放入「打开目录」指向的文件夹即可,不再逐模型选目录。
|
||||||
- **iCloud sync hardening**: per-field settings merge (`appSettings.v2`), per-device usage statistics (G-Counter), tombstoned dictionary/history merge, and a low-risk **Sync Now** action in Settings. / **iCloud 同步加固**:设置按字段合并(`appSettings.v2`)、统计按设备 G-Counter 累计、词库/历史带墓碑合并,并在设置中新增低风险的**立即同步**操作。
|
- **iCloud sync hardening**: per-field settings merge (`appSettings.v2`), per-device usage statistics (G-Counter), tombstoned dictionary/history merge, and a low-risk **Sync Now** action in Settings. / **iCloud 同步加固**:设置按字段合并(`appSettings.v2`)、统计按设备 G-Counter 累计、词库/历史带墓碑合并,并在设置中新增低风险的**立即同步**操作。
|
||||||
|
|
||||||
### Changed
|
### Changed
|
||||||
- **API key sync**: cloud provider API keys now replicate through **iCloud Keychain** when settings sync is on — never through iCloud KVS JSON. / **API 密钥同步**:开启设置同步后,云端服务商 API 密钥改由 **iCloud 钥匙串**复制,不再写入 iCloud KVS JSON。
|
- **API key sync**: cloud provider API keys now replicate through **iCloud Keychain** when settings sync is on — never through iCloud KVS JSON. / **API 密钥同步**:开启设置同步后,云端服务商 API 密钥改由 **iCloud 钥匙串**复制,不再写入 iCloud KVS JSON。
|
||||||
- **Speech history cap**: synced history limit is **300** entries (aligned with the sync payload). / **语音历史上限**:可同步历史上限为 **300** 条(与同步载荷一致)。
|
- **Speech history cap**: synced history limit is **300** entries (aligned with the sync payload). / **语音历史上限**:可同步历史上限为 **300** 条(与同步载荷一致)。
|
||||||
|
- **macOS app name**: the built product is now `OSGKeyboard.app` (was `OSGKeyboardMac.app`); Dock, About, and Finder all read **OSGKeyboard**. / **macOS 应用名称**:编译产物改为 `OSGKeyboard.app`(原 `OSGKeyboardMac.app`);Dock、关于窗口与 Finder 均显示 **OSGKeyboard**。
|
||||||
|
- **macOS local recognition label**: the Settings entry is now simply "Local Recognition" and no longer names a specific model. / **macOS 本地识别标签**:设置项改为「本地识别」,不再绑定具体模型名称。
|
||||||
|
|
||||||
### Fixed
|
### Fixed
|
||||||
|
- **macOS menu-bar icon in light mode**: the status-bar icon now follows the *system* menu-bar appearance, so forcing the app into Light while the system is Dark no longer renders an unreadable dark icon; a refreshed status mark is used. / **macOS 菜单栏图标(浅色模式)**:状态栏图标改为跟随*系统*菜单栏外观,App 强制浅色而系统为深色时不再出现看不清的深色图标;并更新了状态栏图标。
|
||||||
|
- **macOS light-mode sidebar**: restored the native translucent sidebar material so the light appearance matches system apps (e.g. System Settings, Notes) instead of a flat grey fill. / **macOS 浅色侧边栏**:恢复原生半透明侧栏材质,浅色外观与系统应用(如系统设置、备忘录)一致,不再是扁平灰底。
|
||||||
- **Settings sync wiping API keys**: pulling a legacy settings blob without API key fields no longer deletes local Keychain entries. / **设置同步清空 API 密钥**:拉取不含 API 密钥字段的旧版设置包时,不再删除本地 Keychain 项。
|
- **Settings sync wiping API keys**: pulling a legacy settings blob without API key fields no longer deletes local Keychain entries. / **设置同步清空 API 密钥**:拉取不含 API 密钥字段的旧版设置包时,不再删除本地 Keychain 项。
|
||||||
- **Cross-device settings conflicts**: changing different settings on two devices no longer lets one device's full blob overwrite the other's unrelated fields. / **跨设备设置冲突**:两台设备分别修改不同设置项时,不再因整包覆盖而冲掉对方未改动的字段。
|
- **Cross-device settings conflicts**: changing different settings on two devices no longer lets one device's full blob overwrite the other's unrelated fields. / **跨设备设置冲突**:两台设备分别修改不同设置项时,不再因整包覆盖而冲掉对方未改动的字段。
|
||||||
- **Usage statistics under-counting**: offline usage on multiple devices now sums correctly instead of taking per-field `max()`. / **使用统计少计**:多设备离线各自累计后合并为求和,不再对总量取 `max()`。
|
- **Usage statistics under-counting**: offline usage on multiple devices now sums correctly instead of taking per-field `max()`. / **使用统计少计**:多设备离线各自累计后合并为求和,不再对总量取 `max()`。
|
||||||
- **Dictionary/history resurrection**: deletes and "clear all" on one device propagate via tombstones so older remote entries cannot come back. / **词库/历史复活**:单设备删除或清空会通过墓碑传播,远端旧条目无法复活。
|
- **Dictionary/history resurrection**: deletes and "clear all" on one device propagate via tombstones so older remote entries cannot come back. / **词库/历史复活**:单设备删除或清空会通过墓碑传播,远端旧条目无法复活。
|
||||||
- **Flow false-ready mic state**: the keyboard mic now stays orange until the host app publishes a real ready contract (capture engine live + polling idle), not merely a fresh heartbeat; green tap-to-talk and jump-to-host behavior share the same `MicVoiceAvailability` gate, and orphaned `stopped` signals self-heal instead of hanging until timeout. / **Flow 伪就绪麦克风状态**:键盘麦克风在主 App 发布真实就绪合约(音频引擎在跑且轮询空闲)之前保持橙色,不再仅凭心跳误判;绿色「点按说话」与跳转主 App 共用同一 `MicVoiceAvailability` 闸门,孤立的 `stopped` 信号会自愈而不再长时间卡住。
|
- **Flow false-ready mic state**: the keyboard mic now stays orange until the host app publishes a real ready contract (capture engine live + polling idle), not merely a fresh heartbeat; green tap-to-talk and jump-to-host behavior share the same `MicVoiceAvailability` gate, and orphaned `stopped` signals self-heal instead of hanging until timeout. / **Flow 伪就绪麦克风状态**:键盘麦克风在主 App 发布真实就绪合约(音频引擎在跑且轮询空闲)之前保持橙色,不再仅凭心跳误判;绿色「点按说话」与跳转主 App 共用同一 `MicVoiceAvailability` 闸门,孤立的 `stopped` 信号会自愈而不再长时间卡住。
|
||||||
- **Flow mic stuck orange after ready**: a single stale cross-process heartbeat read no longer flips a healthy session into a sticky "session ended" error that forced the mic orange. The "session ended" hint now fires only when the (heartbeat-independent) session contract truly drops; a brief read jitter is smoothed by a ready grace window, and a lingering expired hint auto-recovers to green once the host is ready again. / **就绪后麦克风卡橙色**:单次跨进程心跳读数抖动不再把健康会话打成粘滞的「会话已结束」错误、强制麦克风变橙。「会话已结束」提示现仅在(不依赖心跳的)会话合约真正失效时触发;短暂读数抖动由就绪宽限期平滑,遗留的过期提示会在宿主重新就绪后自动恢复为绿色。
|
- **Flow mic stuck orange after ready**: a single stale cross-process heartbeat read no longer flips a healthy session into a sticky "session ended" error that forced the mic orange. The "session ended" hint now fires only when the (heartbeat-independent) session contract truly drops; a brief read jitter is smoothed by a ready grace window, and a lingering expired hint auto-recovers to green once the host is ready again. / **就绪后麦克风卡橙色**:单次跨进程心跳读数抖动不再把健康会话打成粘滞的「会话已结束」错误、强制麦克风变橙。「会话已结束」提示现仅在(不依赖心跳的)会话合约真正失效时触发;短暂读数抖动由就绪宽限期平滑,遗留的过期提示会在宿主重新就绪后自动恢复为绿色。
|
||||||
|
- **Orphaned Live Activity after force-quit**: force-quitting the app no longer leaves a stale OSGKeyboard status stuck on the Lock Screen / Dynamic Island. The `staleDate` is now ~45s (refreshed by the heartbeat while the session is alive) so the system reclaims a dead session's island on its own, and every app foreground now sweeps leftover Live Activities *before* trying to (re)start a session — so even a start that later fails (e.g. mic proof timeout) still clears the zombie island. / **强杀后遗留 Live Activity**:强制退出 App 不再在锁屏 / 灵动岛留下无法消失的 OSGKeyboard 状态。`staleDate` 缩短为约 45 秒(会话存活期间由心跳持续刷新),系统会自动回收已死会话的灵动岛;且每次 App 回到前台都会**先**清扫遗留的 Live Activity 再尝试(重新)启动会话——即便本次启动随后失败(如麦克风就绪超时),也不会留下僵尸灵动岛。
|
||||||
|
|
||||||
## [0.5.0] - 2026-07-07
|
## [0.5.0] - 2026-07-07
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,25 @@
|
|||||||
|
{
|
||||||
|
"images" : [
|
||||||
|
{
|
||||||
|
"filename" : "OSGStatusMark.png",
|
||||||
|
"idiom" : "universal",
|
||||||
|
"scale" : "1x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "universal",
|
||||||
|
"scale" : "2x"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"idiom" : "universal",
|
||||||
|
"scale" : "3x"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"info" : {
|
||||||
|
"author" : "xcode",
|
||||||
|
"version" : 1
|
||||||
|
},
|
||||||
|
"properties" : {
|
||||||
|
"preserves-vector-representation" : true,
|
||||||
|
"template-rendering-intent" : "template"
|
||||||
|
}
|
||||||
|
}
|
||||||
Binary file not shown.
|
After Width: | Height: | Size: 3.9 KiB |
@@ -10,12 +10,16 @@ import OSGKeyboardShared
|
|||||||
|
|
||||||
enum FlowLiveActivityController {
|
enum FlowLiveActivityController {
|
||||||
nonisolated(unsafe) private static var currentActivity: Activity<FlowActivityAttributes>?
|
nonisolated(unsafe) private static var currentActivity: Activity<FlowActivityAttributes>?
|
||||||
|
/// Last phase pushed to the Live Activity so `keepAlive()` can refresh the
|
||||||
|
/// `staleDate` without changing what the user sees.
|
||||||
|
nonisolated(unsafe) private static var currentPhase: FlowActivityAttributes.ContentState.Phase = .idle
|
||||||
|
|
||||||
/// If the host app is force-quit its `endSession()` never runs, orphaning
|
/// If the host app is force-quit its `endSession()` never runs, orphaning
|
||||||
/// the Live Activity. A `staleDate` lets the system grey it out and become
|
/// the Live Activity. A short `staleDate` lets the system grey it out and
|
||||||
/// willing to reclaim it without our process — refreshed on every update
|
/// reclaim it on its own within ~45s of the process dying. While the host is
|
||||||
/// so a genuinely active, in-use session never looks stale.
|
/// alive the heartbeat calls `keepAlive()` well inside this window, so a
|
||||||
private static let staleWindow: TimeInterval = 60 * 60
|
/// genuinely active session never looks stale.
|
||||||
|
private static let staleWindow: TimeInterval = 45
|
||||||
|
|
||||||
private static func freshContent(
|
private static func freshContent(
|
||||||
phase: FlowActivityAttributes.ContentState.Phase
|
phase: FlowActivityAttributes.ContentState.Phase
|
||||||
@@ -41,6 +45,7 @@ enum FlowLiveActivityController {
|
|||||||
}
|
}
|
||||||
|
|
||||||
do {
|
do {
|
||||||
|
currentPhase = .idle
|
||||||
currentActivity = try Activity.request(
|
currentActivity = try Activity.request(
|
||||||
attributes: FlowActivityAttributes(),
|
attributes: FlowActivityAttributes(),
|
||||||
content: freshContent(phase: .idle),
|
content: freshContent(phase: .idle),
|
||||||
@@ -54,14 +59,28 @@ enum FlowLiveActivityController {
|
|||||||
|
|
||||||
static func update(phase: FlowActivityAttributes.ContentState.Phase) {
|
static func update(phase: FlowActivityAttributes.ContentState.Phase) {
|
||||||
guard let activity = currentActivity else { return }
|
guard let activity = currentActivity else { return }
|
||||||
|
currentPhase = phase
|
||||||
let content = freshContent(phase: phase)
|
let content = freshContent(phase: phase)
|
||||||
Task {
|
Task {
|
||||||
await activity.update(content)
|
await activity.update(content)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Push a fresh `staleDate` without changing the visible phase. The host
|
||||||
|
/// heartbeat calls this well inside `staleWindow` so an in-use session
|
||||||
|
/// never looks stale; once the process dies the refreshes stop and the
|
||||||
|
/// system reclaims the orphaned Live Activity on its own.
|
||||||
|
static func keepAlive() {
|
||||||
|
guard let activity = currentActivity else { return }
|
||||||
|
let content = freshContent(phase: currentPhase)
|
||||||
|
Task {
|
||||||
|
await activity.update(content)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Dismiss the island presentation when the Flow session ends.
|
/// Dismiss the island presentation when the Flow session ends.
|
||||||
static func endSession() {
|
static func endSession() {
|
||||||
|
currentPhase = .idle
|
||||||
guard let activity = currentActivity else {
|
guard let activity = currentActivity else {
|
||||||
endStaleActivities()
|
endStaleActivities()
|
||||||
return
|
return
|
||||||
@@ -74,6 +93,18 @@ enum FlowLiveActivityController {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Clear Live Activities orphaned by a previous (force-quit) host process.
|
||||||
|
///
|
||||||
|
/// Safe to call on every app foreground: when this process already owns a
|
||||||
|
/// Live Activity (`currentActivity != nil`) we leave it alone so a healthy
|
||||||
|
/// running session is never torn down; we only sweep leftovers that belong
|
||||||
|
/// to a dead process. Call this *before* attempting to (re)start a session
|
||||||
|
/// so a failed start (e.g. mic timeout) still clears the stale island.
|
||||||
|
static func clearOrphanedActivities() {
|
||||||
|
guard currentActivity == nil else { return }
|
||||||
|
endStaleActivities()
|
||||||
|
}
|
||||||
|
|
||||||
/// Host relaunch can leave orphan activities; clear them before starting anew.
|
/// Host relaunch can leave orphan activities; clear them before starting anew.
|
||||||
private static func endStaleActivities() {
|
private static func endStaleActivities() {
|
||||||
let staleActivities = Activity<FlowActivityAttributes>.activities
|
let staleActivities = Activity<FlowActivityAttributes>.activities
|
||||||
|
|||||||
@@ -25,9 +25,7 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
private let capture = FlowContinuousCapture()
|
private let capture = FlowContinuousCapture()
|
||||||
private let store = AppGroupStore()
|
private let store = AppGroupStore()
|
||||||
/// Cloud-engine polish; local engine runs through built-in DeepSeek polish.
|
/// Cloud-engine polish; local engine runs through built-in DeepSeek polish.
|
||||||
private var polisher: PolishingService {
|
private let polisher = PolishingService()
|
||||||
PolishingService()
|
|
||||||
}
|
|
||||||
/// Cached ASR instance. v0.2.0: the only on-device backend is iOS
|
/// Cached ASR instance. v0.2.0: the only on-device backend is iOS
|
||||||
/// `SpeechAnalyzer`, which has no warm-up step — we can hand the
|
/// `SpeechAnalyzer`, which has no warm-up step — we can hand the
|
||||||
/// factory-built service straight back without going through the
|
/// factory-built service straight back without going through the
|
||||||
@@ -49,17 +47,26 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
private var expiryTask: Task<Void, Never>?
|
private var expiryTask: Task<Void, Never>?
|
||||||
private var levelTask: Task<Void, Never>?
|
private var levelTask: Task<Void, Never>?
|
||||||
private var startTask: Task<Void, Never>?
|
private var startTask: Task<Void, Never>?
|
||||||
|
private var commandObserver: FlowSessionDarwinObserver?
|
||||||
/// Last recording state the poll loop observed — logs only on transition.
|
/// Last recording state the poll loop observed — logs only on transition.
|
||||||
private var lastObservedRecordingState: FlowSessionKeys.RecordingState = .idle
|
private var lastObservedRecordingState: FlowSessionKeys.RecordingState = .idle
|
||||||
|
private var activeSessionId: UUID?
|
||||||
|
private var currentUtteranceId: UUID?
|
||||||
|
private var currentCommandSeq: Int64 = 0
|
||||||
|
private var lastHandledCommandSeq: Int64 = 0
|
||||||
private var isUtteranceRecording = false
|
private var isUtteranceRecording = false
|
||||||
/// True from `stopped` until the result/error is written back to App Group.
|
/// True from `stopped` until the result/error is written back to App Group.
|
||||||
private var isUtteranceProcessing = false
|
private var isUtteranceProcessing = false
|
||||||
private var finalizeTask: Task<Void, Never>?
|
private var finalizeTask: Task<Void, Never>?
|
||||||
private var asrTask: Task<Void, Never>?
|
private var asrTask: Task<Void, Never>?
|
||||||
|
private var utteranceSafetyTask: Task<Void, Never>?
|
||||||
private var chunkedPipeline: ChunkedUtterancePipeline?
|
private var chunkedPipeline: ChunkedUtterancePipeline?
|
||||||
private var currentPartial = ""
|
private var currentPartial = ""
|
||||||
private var lastFinal = ""
|
private var lastFinal = ""
|
||||||
private var chunkWarnings: [String] = []
|
private var chunkWarnings: [String] = []
|
||||||
|
private var lastReadyTraceSignature = ""
|
||||||
|
private var lastCommandFingerprint = ""
|
||||||
|
private var lastIgnoredCommandSignature = ""
|
||||||
/// Wall-clock span of the current mic-open utterance (excludes LLM polish).
|
/// Wall-clock span of the current mic-open utterance (excludes LLM polish).
|
||||||
private var utteranceRecordingStartedAt: Date?
|
private var utteranceRecordingStartedAt: Date?
|
||||||
/// True while the host app scene is `.active` — drives foreground renewal.
|
/// True while the host app scene is `.active` — drives foreground renewal.
|
||||||
@@ -67,7 +74,11 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
private var backgroundTaskID: UIBackgroundTaskIdentifier = .invalid
|
private var backgroundTaskID: UIBackgroundTaskIdentifier = .invalid
|
||||||
/// True while handling a keyboard-initiated `startflow` cold start.
|
/// True while handling a keyboard-initiated `startflow` cold start.
|
||||||
private var isColdStartHandoff = false
|
private var isColdStartHandoff = false
|
||||||
private static let coldStartAudioProofTimeout: TimeInterval = 2.5
|
private var coldStartRecoveryTask: Task<Void, Never>?
|
||||||
|
/// Initial proof window — cold mic sessions often need >2.5s after app switch.
|
||||||
|
private static let coldStartAudioProofTimeout: TimeInterval = 6
|
||||||
|
/// Extra window after the first timeout while the overlay shows a failure hint.
|
||||||
|
private static let coldStartRecoveryProofTimeout: TimeInterval = 12
|
||||||
|
|
||||||
init() {
|
init() {
|
||||||
// Sessions are (re)started explicitly on app foreground via
|
// Sessions are (re)started explicitly on app foreground via
|
||||||
@@ -83,6 +94,10 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
|
|
||||||
/// Starts a Flow session: permissions → continuous capture → App Group active.
|
/// Starts a Flow session: permissions → continuous capture → App Group active.
|
||||||
func startSession(duration: TimeInterval? = nil, coldStart: Bool = false) {
|
func startSession(duration: TimeInterval? = nil, coldStart: Bool = false) {
|
||||||
|
traceState(
|
||||||
|
"startSession.request",
|
||||||
|
extra: "coldStart=\(coldStart) duration=\(Int(duration ?? FlowSessionPolicy.sessionDuration()))"
|
||||||
|
)
|
||||||
guard AppGroup.isAvailable else {
|
guard AppGroup.isAvailable else {
|
||||||
debug("cannot start flow session: App Group unavailable")
|
debug("cannot start flow session: App Group unavailable")
|
||||||
return
|
return
|
||||||
@@ -106,7 +121,10 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
guard !isStarting else { return }
|
guard !isStarting else {
|
||||||
|
traceState("startSession.ignored", extra: "reason=alreadyStarting")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
startTask?.cancel()
|
startTask?.cancel()
|
||||||
startTask = Task { @MainActor [weak self] in
|
startTask = Task { @MainActor [weak self] in
|
||||||
@@ -148,6 +166,14 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
/// behind (its `endSession()` could not run at kill time).
|
/// behind (its `endSession()` could not run at kill time).
|
||||||
func activateOnForeground() {
|
func activateOnForeground() {
|
||||||
guard AppGroup.isAvailable else { return }
|
guard AppGroup.isAvailable else { return }
|
||||||
|
// Sweep any Live Activity a previous (force-quit) process left behind
|
||||||
|
// *before* we try to (re)start a session. Doing it here — rather than
|
||||||
|
// only inside `startSession()`'s success path — means a start that
|
||||||
|
// later fails (e.g. mic proof timeout) still clears the stale island
|
||||||
|
// instead of leaving a zombie on the lock screen / Dynamic Island.
|
||||||
|
// No-op when this process already owns a healthy Live Activity.
|
||||||
|
FlowLiveActivityController.clearOrphanedActivities()
|
||||||
|
|
||||||
guard AppPermissions.flowRequirementsMet else {
|
guard AppPermissions.flowRequirementsMet else {
|
||||||
sessionWarning = permissionWarningMessage()
|
sessionWarning = permissionWarningMessage()
|
||||||
FlowSessionBridge.setHostReady(false)
|
FlowSessionBridge.setHostReady(false)
|
||||||
@@ -161,7 +187,8 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func dismissColdStartOverlay() {
|
func dismissColdStartOverlay() {
|
||||||
guard coldStartContext?.state != .preparing else { return }
|
coldStartRecoveryTask?.cancel()
|
||||||
|
coldStartRecoveryTask = nil
|
||||||
if isActive {
|
if isActive {
|
||||||
refreshHostReady()
|
refreshHostReady()
|
||||||
}
|
}
|
||||||
@@ -189,8 +216,11 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
|
|
||||||
coldStartContext = nil
|
coldStartContext = nil
|
||||||
isColdStartHandoff = false
|
isColdStartHandoff = false
|
||||||
|
coldStartRecoveryTask?.cancel()
|
||||||
|
coldStartRecoveryTask = nil
|
||||||
startTask?.cancel()
|
startTask?.cancel()
|
||||||
startTask = nil
|
startTask = nil
|
||||||
|
commandObserver = nil
|
||||||
pollingTask?.cancel()
|
pollingTask?.cancel()
|
||||||
pollingTask = nil
|
pollingTask = nil
|
||||||
heartbeatTask?.cancel()
|
heartbeatTask?.cancel()
|
||||||
@@ -201,6 +231,8 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
levelTask = nil
|
levelTask = nil
|
||||||
finalizeTask?.cancel()
|
finalizeTask?.cancel()
|
||||||
finalizeTask = nil
|
finalizeTask = nil
|
||||||
|
utteranceSafetyTask?.cancel()
|
||||||
|
utteranceSafetyTask = nil
|
||||||
|
|
||||||
if isUtteranceRecording || isUtteranceProcessing {
|
if isUtteranceRecording || isUtteranceProcessing {
|
||||||
capture.cancelUtterance()
|
capture.cancelUtterance()
|
||||||
@@ -210,6 +242,10 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
}
|
}
|
||||||
asrTask = nil
|
asrTask = nil
|
||||||
chunkedPipeline = nil
|
chunkedPipeline = nil
|
||||||
|
activeSessionId = nil
|
||||||
|
currentUtteranceId = nil
|
||||||
|
currentCommandSeq = 0
|
||||||
|
lastHandledCommandSeq = 0
|
||||||
isUtteranceRecording = false
|
isUtteranceRecording = false
|
||||||
isUtteranceProcessing = false
|
isUtteranceProcessing = false
|
||||||
|
|
||||||
@@ -305,9 +341,11 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
guard isActive else { return }
|
guard isActive else { return }
|
||||||
|
|
||||||
if capture.running {
|
if capture.running {
|
||||||
capture.reassertIfRunning()
|
let reasserted = capture.reassertIfRunning()
|
||||||
if capture.engineHasRecentAudio() {
|
if reasserted, capture.engineHasRecentAudio() {
|
||||||
sessionWarning = nil
|
sessionWarning = nil
|
||||||
|
} else if !reasserted {
|
||||||
|
sessionWarning = AppL10n.string("flow.error.audioUnavailable")
|
||||||
}
|
}
|
||||||
refreshHostReady()
|
refreshHostReady()
|
||||||
return
|
return
|
||||||
@@ -329,22 +367,94 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
/// Publish whether the keyboard can start a new utterance without jumping to the host app.
|
/// Publish whether the keyboard can start a new utterance without jumping to the host app.
|
||||||
private func refreshHostReady() {
|
private func refreshHostReady() {
|
||||||
guard isActive else {
|
guard isActive else {
|
||||||
FlowSessionBridge.setHostReady(false)
|
FlowSessionBridge.writeReadySnapshot(
|
||||||
|
FlowReadySnapshot(
|
||||||
|
sessionId: activeSessionId,
|
||||||
|
ready: false,
|
||||||
|
reason: .noSession,
|
||||||
|
engineMode: store.engineMode,
|
||||||
|
localeId: store.localeId,
|
||||||
|
sessionExpiresAt: FlowSessionBridge.sessionExpiresAt()
|
||||||
|
)
|
||||||
|
)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
let pollingAlive = pollingTask != nil && pollingTask?.isCancelled != true
|
let pollingAlive = pollingTask != nil && pollingTask?.isCancelled != true
|
||||||
// Steady-state ready uses structural engine liveness. The stricter
|
let hasRecentAudio = capture.engineHasRecentAudio(maxAge: 2)
|
||||||
// "recent audio frame" proof is reserved for cold-start handoff only
|
|
||||||
// (`waitForAudioProof`) so brief UI-driven session hiccups do not
|
|
||||||
// drop the keyboard back to orange while the host app is foreground.
|
|
||||||
let canAcceptUtterance = capture.engineIsLive
|
let canAcceptUtterance = capture.engineIsLive
|
||||||
&& pollingAlive
|
&& pollingAlive
|
||||||
|
&& hasRecentAudio
|
||||||
&& !isUtteranceRecording
|
&& !isUtteranceRecording
|
||||||
&& !isUtteranceProcessing
|
&& !isUtteranceProcessing
|
||||||
&& sessionWarning == nil
|
&& sessionWarning == nil
|
||||||
|
|
||||||
FlowSessionBridge.setHostReady(canAcceptUtterance)
|
let reason: FlowReadySnapshot.Reason
|
||||||
|
if canAcceptUtterance {
|
||||||
|
reason = .ready
|
||||||
|
} else if sessionWarning != nil {
|
||||||
|
reason = .error
|
||||||
|
} else if isUtteranceRecording {
|
||||||
|
reason = .recording
|
||||||
|
} else if isUtteranceProcessing {
|
||||||
|
reason = .processing
|
||||||
|
} else if !capture.engineIsLive {
|
||||||
|
reason = .audioEngineNotLive
|
||||||
|
} else if !hasRecentAudio {
|
||||||
|
reason = .waitingForAudioProof
|
||||||
|
} else {
|
||||||
|
reason = .starting
|
||||||
|
}
|
||||||
|
|
||||||
|
let now = Date().timeIntervalSince1970
|
||||||
|
FlowSessionBridge.writeReadySnapshot(
|
||||||
|
FlowReadySnapshot(
|
||||||
|
sessionId: activeSessionId,
|
||||||
|
ready: canAcceptUtterance,
|
||||||
|
reason: reason,
|
||||||
|
heartbeatAt: now,
|
||||||
|
readyAt: canAcceptUtterance ? now : nil,
|
||||||
|
audioProofAt: hasRecentAudio ? now : nil,
|
||||||
|
engineMode: store.engineMode,
|
||||||
|
localeId: store.localeId,
|
||||||
|
busyUtteranceId: isUtteranceRecording || isUtteranceProcessing ? currentUtteranceId : nil,
|
||||||
|
sessionExpiresAt: FlowSessionBridge.sessionExpiresAt()
|
||||||
|
)
|
||||||
|
)
|
||||||
|
let signature = [
|
||||||
|
canAcceptUtterance ? "ready=1" : "ready=0",
|
||||||
|
"reason=\(reason.rawValue)",
|
||||||
|
capture.engineIsLive ? "engine=live" : "engine=dead",
|
||||||
|
hasRecentAudio ? "audio=fresh" : "audio=stale",
|
||||||
|
isUtteranceRecording ? "recording=1" : "recording=0",
|
||||||
|
isUtteranceProcessing ? "processing=1" : "processing=0",
|
||||||
|
sessionWarning == nil ? "warning=0" : "warning=1"
|
||||||
|
].joined(separator: "|")
|
||||||
|
if signature != lastReadyTraceSignature {
|
||||||
|
lastReadyTraceSignature = signature
|
||||||
|
traceState("hostReady.update", extra: signature)
|
||||||
|
}
|
||||||
|
reconcileColdStartOverlayIfRecovered()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// When the host contract turns green while the cold-start overlay still
|
||||||
|
/// shows a stale preparing/failed snapshot, heal automatically.
|
||||||
|
private func reconcileColdStartOverlayIfRecovered() {
|
||||||
|
guard isColdStartHandoff, isActive else { return }
|
||||||
|
guard FlowSessionBridge.isHostReady() else { return }
|
||||||
|
guard let context = coldStartContext else { return }
|
||||||
|
|
||||||
|
switch context.state {
|
||||||
|
case .preparing:
|
||||||
|
presentColdStartReadyOverlay()
|
||||||
|
case .failed:
|
||||||
|
sessionWarning = nil
|
||||||
|
coldStartRecoveryTask?.cancel()
|
||||||
|
coldStartRecoveryTask = nil
|
||||||
|
dismissColdStartOverlay()
|
||||||
|
case .ready:
|
||||||
|
break
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Home preview field gained focus while this app is the Flow host.
|
/// Home preview field gained focus while this app is the Flow host.
|
||||||
@@ -378,12 +488,14 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
// MARK: - Session start
|
// MARK: - Session start
|
||||||
|
|
||||||
private func startSessionAsync(duration: TimeInterval?) async {
|
private func startSessionAsync(duration: TimeInterval?) async {
|
||||||
|
traceState("startSessionAsync.begin")
|
||||||
isStarting = true
|
isStarting = true
|
||||||
sessionWarning = nil
|
sessionWarning = nil
|
||||||
defer { isStarting = false }
|
defer { isStarting = false }
|
||||||
|
|
||||||
guard AppPermissions.flowRequirementsMet else {
|
guard AppPermissions.flowRequirementsMet else {
|
||||||
sessionWarning = permissionWarningMessage()
|
sessionWarning = permissionWarningMessage()
|
||||||
|
traceState("startSessionAsync.blocked", extra: "reason=permissions")
|
||||||
FlowSessionBridge.setHostReady(false)
|
FlowSessionBridge.setHostReady(false)
|
||||||
if isColdStartHandoff {
|
if isColdStartHandoff {
|
||||||
showColdStartPermissionFailure()
|
showColdStartPermissionFailure()
|
||||||
@@ -397,6 +509,7 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
} catch {
|
} catch {
|
||||||
let message = (error as? LocalizedError)?.errorDescription ?? error.localizedDescription
|
let message = (error as? LocalizedError)?.errorDescription ?? error.localizedDescription
|
||||||
sessionWarning = message
|
sessionWarning = message
|
||||||
|
traceState("startSessionAsync.failed", extra: "reason=captureStart error=\(message)")
|
||||||
FlowSessionBridge.setHostReady(false)
|
FlowSessionBridge.setHostReady(false)
|
||||||
if isColdStartHandoff {
|
if isColdStartHandoff {
|
||||||
showColdStartAudioFailure(message: message)
|
showColdStartAudioFailure(message: message)
|
||||||
@@ -408,23 +521,36 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
guard await waitForAudioProof() else {
|
guard await waitForAudioProof() else {
|
||||||
let message = AppL10n.string("flow.coldStart.error.audioTimeout")
|
let message = AppL10n.string("flow.coldStart.error.audioTimeout")
|
||||||
sessionWarning = message
|
sessionWarning = message
|
||||||
capture.stop()
|
traceState("startSessionAsync.failed", extra: "reason=audioProofTimeout")
|
||||||
FlowSessionBridge.setHostReady(false)
|
FlowSessionBridge.setHostReady(false)
|
||||||
if isColdStartHandoff {
|
if isColdStartHandoff {
|
||||||
showColdStartAudioFailure(message: message)
|
showColdStartAudioFailure(message: message)
|
||||||
|
scheduleColdStartRecovery(duration: duration)
|
||||||
|
} else {
|
||||||
|
capture.stop()
|
||||||
}
|
}
|
||||||
debug("continuous capture did not produce audio frames before timeout")
|
debug("continuous capture did not produce audio frames before timeout")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
activateFlowSessionAfterAudioProof(duration: duration)
|
||||||
|
traceState("startSessionAsync.ready")
|
||||||
|
debug("Flow session started (\(Int(duration ?? FlowSessionPolicy.sessionDuration()))s inactivity window), continuous capture running")
|
||||||
|
}
|
||||||
|
|
||||||
|
private func activateFlowSessionAfterAudioProof(duration: TimeInterval?) {
|
||||||
let resolvedDuration = duration ?? FlowSessionPolicy.sessionDuration()
|
let resolvedDuration = duration ?? FlowSessionPolicy.sessionDuration()
|
||||||
FlowSessionBridge.markSessionActive(duration: resolvedDuration)
|
let sessionId = activeSessionId ?? UUID()
|
||||||
|
activeSessionId = sessionId
|
||||||
|
lastHandledCommandSeq = 0
|
||||||
|
FlowSessionBridge.markSessionActive(duration: resolvedDuration, sessionId: sessionId)
|
||||||
FlowSessionDarwin.postSessionChanged()
|
FlowSessionDarwin.postSessionChanged()
|
||||||
isActive = true
|
isActive = true
|
||||||
ScreenWakeLock.acquire()
|
ScreenWakeLock.acquire()
|
||||||
sessionExpiresAt = Date().addingTimeInterval(resolvedDuration)
|
sessionExpiresAt = Date().addingTimeInterval(resolvedDuration)
|
||||||
|
|
||||||
startHeartbeat()
|
startHeartbeat()
|
||||||
|
startCommandObserver()
|
||||||
startPolling()
|
startPolling()
|
||||||
startLevelPublishing()
|
startLevelPublishing()
|
||||||
scheduleExpiry(after: resolvedDuration)
|
scheduleExpiry(after: resolvedDuration)
|
||||||
@@ -434,7 +560,7 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
FlowLiveActivityController.startSession()
|
FlowLiveActivityController.startSession()
|
||||||
|
|
||||||
refreshHostReady()
|
refreshHostReady()
|
||||||
debug("Flow session started (\(Int(resolvedDuration))s inactivity window), continuous capture running")
|
traceState("activateFlowSessionAfterAudioProof.done")
|
||||||
}
|
}
|
||||||
|
|
||||||
private func prepareExistingSessionForColdStartReturn() async {
|
private func prepareExistingSessionForColdStartReturn() async {
|
||||||
@@ -445,6 +571,7 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
sessionWarning = message
|
sessionWarning = message
|
||||||
FlowSessionBridge.setHostReady(false)
|
FlowSessionBridge.setHostReady(false)
|
||||||
showColdStartAudioFailure(message: message)
|
showColdStartAudioFailure(message: message)
|
||||||
|
scheduleColdStartRecovery(duration: nil)
|
||||||
debug("existing session failed cold-start audio proof")
|
debug("existing session failed cold-start audio proof")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -466,25 +593,54 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
let message = AppL10n.string("flow.coldStart.error.audioTimeout")
|
let message = AppL10n.string("flow.coldStart.error.audioTimeout")
|
||||||
sessionWarning = message
|
sessionWarning = message
|
||||||
showColdStartAudioFailure(message: message)
|
showColdStartAudioFailure(message: message)
|
||||||
|
scheduleColdStartRecovery(duration: nil)
|
||||||
debug("cold-start blocked: host ready contract not published")
|
debug("cold-start blocked: host ready contract not published")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
let hostEntry = HostReturnService.pendingHostEntry()
|
presentColdStartReadyOverlay()
|
||||||
let skipSwitch = FlowSessionPolicy.skipAppSwitch()
|
}
|
||||||
coldStartContext = FlowColdStartContext(hostEntry: hostEntry, state: .ready)
|
|
||||||
|
|
||||||
if skipSwitch, hostEntry != nil {
|
private func presentColdStartReadyOverlay() {
|
||||||
Task { @MainActor [weak self] in
|
let hostEntry = HostReturnService.pendingHostEntry()
|
||||||
try? await Task.sleep(nanoseconds: 450_000_000)
|
coldStartContext = FlowColdStartContext(hostEntry: hostEntry, state: .ready)
|
||||||
guard let self, self.coldStartContext?.state == .ready else { return }
|
scheduleAutoReturnToHostIfNeeded(hostEntry: hostEntry)
|
||||||
if HostReturnService.openPendingHostIfPossible() {
|
}
|
||||||
self.dismissColdStartOverlay()
|
|
||||||
}
|
private func scheduleAutoReturnToHostIfNeeded(hostEntry: HostAppEntry?) {
|
||||||
|
let skipSwitch = FlowSessionPolicy.skipAppSwitch()
|
||||||
|
guard skipSwitch, hostEntry != nil else { return }
|
||||||
|
Task { @MainActor [weak self] in
|
||||||
|
try? await Task.sleep(nanoseconds: 450_000_000)
|
||||||
|
guard let self, self.coldStartContext?.state == .ready else { return }
|
||||||
|
if HostReturnService.openPendingHostIfPossible() {
|
||||||
|
self.dismissColdStartOverlay()
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Keeps proving mic readiness after the first timeout instead of tearing
|
||||||
|
/// capture down — many handoffs become ready a few seconds later.
|
||||||
|
private func scheduleColdStartRecovery(duration: TimeInterval?) {
|
||||||
|
coldStartRecoveryTask?.cancel()
|
||||||
|
coldStartRecoveryTask = Task { @MainActor [weak self] in
|
||||||
|
guard let self else { return }
|
||||||
|
let recovered = await self.capture.awaitAudioFlowing(
|
||||||
|
timeout: Self.coldStartRecoveryProofTimeout
|
||||||
|
)
|
||||||
|
guard !Task.isCancelled else { return }
|
||||||
|
guard self.isColdStartHandoff else { return }
|
||||||
|
guard recovered else { return }
|
||||||
|
|
||||||
|
self.sessionWarning = nil
|
||||||
|
self.traceState("coldStartRecovery.recovered")
|
||||||
|
if !self.isActive {
|
||||||
|
self.activateFlowSessionAfterAudioProof(duration: duration)
|
||||||
|
}
|
||||||
|
self.refreshHostReady()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
private func showColdStartPreparing() {
|
private func showColdStartPreparing() {
|
||||||
coldStartContext = FlowColdStartContext(
|
coldStartContext = FlowColdStartContext(
|
||||||
hostEntry: HostReturnService.pendingHostEntry(),
|
hostEntry: HostReturnService.pendingHostEntry(),
|
||||||
@@ -546,6 +702,14 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
|
|
||||||
// MARK: - Polling
|
// MARK: - Polling
|
||||||
|
|
||||||
|
private func startCommandObserver() {
|
||||||
|
commandObserver = FlowSessionDarwinObserver(
|
||||||
|
notificationName: FlowSessionDarwin.commandNotificationName
|
||||||
|
) { [weak self] in
|
||||||
|
self?.handleKeyboardSignal()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
private func startPolling() {
|
private func startPolling() {
|
||||||
pollingTask?.cancel()
|
pollingTask?.cancel()
|
||||||
lastObservedRecordingState = FlowSessionBridge.recordingState()
|
lastObservedRecordingState = FlowSessionBridge.recordingState()
|
||||||
@@ -556,46 +720,122 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
pollingTask = Task { @MainActor [weak self] in
|
pollingTask = Task { @MainActor [weak self] in
|
||||||
while !Task.isCancelled {
|
while !Task.isCancelled {
|
||||||
self?.handleKeyboardSignal()
|
self?.handleKeyboardSignal()
|
||||||
try? await Task.sleep(nanoseconds: 50_000_000)
|
try? await Task.sleep(nanoseconds: 500_000_000)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
private func handleKeyboardSignal() {
|
private func handleKeyboardSignal() {
|
||||||
let signal = FlowSessionBridge.recordingState()
|
guard let command = FlowSessionBridge.latestCommand() else {
|
||||||
if signal != lastObservedRecordingState {
|
lastCommandFingerprint = ""
|
||||||
// The single most important cross-process signal: proves whether the
|
return
|
||||||
// host actually SEES the keyboard's recording state writes.
|
|
||||||
FlowDiagnostics.log(
|
|
||||||
"poll observed recordingState \(lastObservedRecordingState.rawValue) → \(signal.rawValue) " +
|
|
||||||
"[rec=\(isUtteranceRecording) proc=\(isUtteranceProcessing) fg=\(isAppForeground)]"
|
|
||||||
)
|
|
||||||
lastObservedRecordingState = signal
|
|
||||||
}
|
}
|
||||||
switch signal {
|
let fingerprint = "\(command.sessionId.uuidString)|\(command.utteranceId.uuidString)|\(command.action.rawValue)|\(command.commandSeq)"
|
||||||
case .recording:
|
guard fingerprint != lastCommandFingerprint else { return }
|
||||||
|
lastCommandFingerprint = fingerprint
|
||||||
|
handleFlowCommand(command)
|
||||||
|
}
|
||||||
|
|
||||||
|
private func handleFlowCommand(_ command: FlowCommand) {
|
||||||
|
guard let activeSessionId, command.sessionId == activeSessionId else {
|
||||||
|
traceIgnoredCommand(
|
||||||
|
reason: "staleSession",
|
||||||
|
command: command,
|
||||||
|
detail: "commandSession=\(command.sessionId)"
|
||||||
|
)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
guard command.commandSeq > lastHandledCommandSeq else {
|
||||||
|
traceIgnoredCommand(
|
||||||
|
reason: "seqNotIncreasing",
|
||||||
|
command: command,
|
||||||
|
detail: "last=\(lastHandledCommandSeq)"
|
||||||
|
)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
lastHandledCommandSeq = command.commandSeq
|
||||||
|
lastIgnoredCommandSignature = ""
|
||||||
|
|
||||||
|
FlowDiagnostics.log(
|
||||||
|
"command \(command.action.rawValue) seq=\(command.commandSeq) utterance=\(command.utteranceId)"
|
||||||
|
)
|
||||||
|
|
||||||
|
switch command.action {
|
||||||
|
case .startRecording:
|
||||||
guard !isUtteranceRecording, !isUtteranceProcessing else { return }
|
guard !isUtteranceRecording, !isUtteranceProcessing else { return }
|
||||||
beginUtterance()
|
beginUtterance(utteranceId: command.utteranceId, commandSeq: command.commandSeq)
|
||||||
case .stopped:
|
case .stopRecording:
|
||||||
|
guard currentUtteranceId == command.utteranceId else { return }
|
||||||
if isUtteranceRecording {
|
if isUtteranceRecording {
|
||||||
endUtterance()
|
endUtterance()
|
||||||
} else if !isUtteranceProcessing {
|
} else if !isUtteranceProcessing {
|
||||||
FlowSessionBridge.setRecordingState(.idle)
|
storeCurrentError(
|
||||||
FlowSessionBridge.storeTranscriptionError(
|
|
||||||
AppL10n.string("flow.error.recognitionInterrupted"),
|
AppL10n.string("flow.error.recognitionInterrupted"),
|
||||||
kind: .recognitionInterrupted
|
kind: .recognitionInterrupted
|
||||||
)
|
)
|
||||||
debug("stopped without active utterance — notified keyboard")
|
debug("stop command without active utterance — notified keyboard")
|
||||||
}
|
}
|
||||||
case .aborted:
|
case .abort:
|
||||||
|
guard currentUtteranceId == command.utteranceId else { return }
|
||||||
abortUtterance()
|
abortUtterance()
|
||||||
case .idle, .processing:
|
|
||||||
break
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
private func beginUtterance() {
|
private func storeCurrentPartial(_ text: String) {
|
||||||
guard capture.engineIsLive else {
|
guard let activeSessionId, let currentUtteranceId else { return }
|
||||||
|
let trimmed = text.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||||
|
guard !trimmed.isEmpty else { return }
|
||||||
|
FlowSessionBridge.writeResult(
|
||||||
|
FlowResult(
|
||||||
|
sessionId: activeSessionId,
|
||||||
|
utteranceId: currentUtteranceId,
|
||||||
|
commandSeq: currentCommandSeq,
|
||||||
|
status: .partial,
|
||||||
|
text: trimmed
|
||||||
|
)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
private func storeCurrentFinal(_ text: String, warning: String? = nil) {
|
||||||
|
let trimmed = text.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||||
|
guard !trimmed.isEmpty else {
|
||||||
|
storeCurrentError(AppL10n.string("flow.error.noSpeech"), kind: .noSpeech)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
guard let activeSessionId, let currentUtteranceId else { return }
|
||||||
|
FlowSessionBridge.writeResult(
|
||||||
|
FlowResult(
|
||||||
|
sessionId: activeSessionId,
|
||||||
|
utteranceId: currentUtteranceId,
|
||||||
|
commandSeq: currentCommandSeq,
|
||||||
|
status: .final,
|
||||||
|
text: trimmed,
|
||||||
|
warning: warning
|
||||||
|
)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
private func storeCurrentError(
|
||||||
|
_ message: String,
|
||||||
|
kind: FlowSessionKeys.TranscriptionErrorKind = .generic,
|
||||||
|
status: FlowResult.Status = .error
|
||||||
|
) {
|
||||||
|
guard let activeSessionId, let currentUtteranceId else { return }
|
||||||
|
FlowSessionBridge.writeResult(
|
||||||
|
FlowResult(
|
||||||
|
sessionId: activeSessionId,
|
||||||
|
utteranceId: currentUtteranceId,
|
||||||
|
commandSeq: currentCommandSeq,
|
||||||
|
status: status,
|
||||||
|
text: message,
|
||||||
|
errorKind: kind
|
||||||
|
)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
private func beginUtterance(utteranceId: UUID? = nil, commandSeq: Int64 = 0) {
|
||||||
|
guard capture.engineHasRecentAudio(maxAge: 2) else {
|
||||||
|
traceState("beginUtterance.blocked", extra: "reason=audioNotRecent")
|
||||||
failUtterance(
|
failUtterance(
|
||||||
message: AppL10n.string("flow.error.audioUnavailable"),
|
message: AppL10n.string("flow.error.audioUnavailable"),
|
||||||
kind: .audioUnavailable
|
kind: .audioUnavailable
|
||||||
@@ -603,13 +843,25 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
guard !isUtteranceProcessing else {
|
guard !isUtteranceProcessing else {
|
||||||
|
traceState("beginUtterance.ignored", extra: "reason=processing")
|
||||||
debug("beginUtterance ignored — previous utterance still processing")
|
debug("beginUtterance ignored — previous utterance still processing")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
bindSessionASRIfNeeded()
|
||||||
|
let expectedEngine = store.engineMode
|
||||||
|
if sessionASREngineMode != expectedEngine {
|
||||||
|
traceState(
|
||||||
|
"beginUtterance.rebindMismatch",
|
||||||
|
extra: "expectedEngine=\(expectedEngine) boundEngine=\(sessionASREngineMode ?? "nil")"
|
||||||
|
)
|
||||||
|
bindSessionASRIfNeeded(force: true)
|
||||||
|
}
|
||||||
|
|
||||||
// Usually already warm from session start; refresh without blocking the mic gate.
|
// Usually already warm from session start; refresh without blocking the mic gate.
|
||||||
scheduleASRWarmup()
|
scheduleASRWarmup()
|
||||||
|
|
||||||
|
currentUtteranceId = utteranceId ?? UUID()
|
||||||
|
currentCommandSeq = commandSeq
|
||||||
currentPartial = ""
|
currentPartial = ""
|
||||||
lastFinal = ""
|
lastFinal = ""
|
||||||
chunkWarnings = []
|
chunkWarnings = []
|
||||||
@@ -617,6 +869,7 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
let localeId = store.localeId
|
let localeId = store.localeId
|
||||||
FlowSessionBridge.setTranscriptionLanguage(localeId)
|
FlowSessionBridge.setTranscriptionLanguage(localeId)
|
||||||
FlowSessionBridge.clearPendingTranscription()
|
FlowSessionBridge.clearPendingTranscription()
|
||||||
|
FlowSessionBridge.clearResult()
|
||||||
|
|
||||||
let locale = SpeechLocaleResolver.resolve(localeId)
|
let locale = SpeechLocaleResolver.resolve(localeId)
|
||||||
let stream = capture.beginUtterance()
|
let stream = capture.beginUtterance()
|
||||||
@@ -625,6 +878,7 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
|
|
||||||
isUtteranceRecording = true
|
isUtteranceRecording = true
|
||||||
utteranceRecordingStartedAt = Date()
|
utteranceRecordingStartedAt = Date()
|
||||||
|
startUtteranceSafetyTimer()
|
||||||
refreshHostReady()
|
refreshHostReady()
|
||||||
FlowLiveActivityController.update(phase: .recording)
|
FlowLiveActivityController.update(phase: .recording)
|
||||||
FlowDiagnostics.log(
|
FlowDiagnostics.log(
|
||||||
@@ -637,8 +891,9 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
asrTask = Task.detached(priority: .userInitiated) { [weak manager = self] in
|
asrTask = Task.detached(priority: .userInitiated) { [weak manager = self] in
|
||||||
let outcome = await pipeline.transcribe(stream: stream) { partial in
|
let outcome = await pipeline.transcribe(stream: stream) { partial in
|
||||||
Task { @MainActor in
|
Task { @MainActor in
|
||||||
manager?.currentPartial = partial
|
guard let manager else { return }
|
||||||
FlowSessionBridge.storeTranscriptionPartial(partial)
|
manager.currentPartial = partial
|
||||||
|
manager.storeCurrentPartial(partial)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
// Re-bind `manager` inside the `@MainActor` block so the
|
// Re-bind `manager` inside the `@MainActor` block so the
|
||||||
@@ -672,14 +927,33 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
debug("utterance recording started")
|
debug("utterance recording started")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private func startUtteranceSafetyTimer() {
|
||||||
|
utteranceSafetyTask?.cancel()
|
||||||
|
let utteranceId = currentUtteranceId
|
||||||
|
utteranceSafetyTask = Task { @MainActor [weak self] in
|
||||||
|
let timeout = FlowSessionKeys.maxUtteranceDuration + 10
|
||||||
|
try? await Task.sleep(nanoseconds: UInt64(timeout * 1_000_000_000))
|
||||||
|
guard let self, !Task.isCancelled else { return }
|
||||||
|
guard self.isUtteranceRecording, self.currentUtteranceId == utteranceId else { return }
|
||||||
|
self.storeCurrentError(
|
||||||
|
AppL10n.string("flow.error.recognitionInterrupted"),
|
||||||
|
kind: .recognitionInterrupted,
|
||||||
|
status: .timeout
|
||||||
|
)
|
||||||
|
self.abortUtterance()
|
||||||
|
self.debug("utterance safety timer aborted stale recording")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
private func endUtterance() {
|
private func endUtterance() {
|
||||||
guard isUtteranceRecording else { return }
|
guard isUtteranceRecording else { return }
|
||||||
|
|
||||||
// Close the mic gate first, then mark processing before dropping the
|
// Close the mic gate first, then mark processing before dropping the
|
||||||
// recording flag so the poll loop cannot start a second utterance.
|
// recording flag so the poll loop cannot start a second utterance.
|
||||||
FlowSessionBridge.setRecordingState(.processing)
|
|
||||||
isUtteranceRecording = false
|
isUtteranceRecording = false
|
||||||
isUtteranceProcessing = true
|
isUtteranceProcessing = true
|
||||||
|
utteranceSafetyTask?.cancel()
|
||||||
|
utteranceSafetyTask = nil
|
||||||
refreshHostReady()
|
refreshHostReady()
|
||||||
FlowLiveActivityController.update(phase: .processing)
|
FlowLiveActivityController.update(phase: .processing)
|
||||||
|
|
||||||
@@ -699,6 +973,8 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
isUtteranceRecording = false
|
isUtteranceRecording = false
|
||||||
isUtteranceProcessing = false
|
isUtteranceProcessing = false
|
||||||
utteranceRecordingStartedAt = nil
|
utteranceRecordingStartedAt = nil
|
||||||
|
utteranceSafetyTask?.cancel()
|
||||||
|
utteranceSafetyTask = nil
|
||||||
finalizeTask?.cancel()
|
finalizeTask?.cancel()
|
||||||
finalizeTask = nil
|
finalizeTask = nil
|
||||||
asrTask?.cancel()
|
asrTask?.cancel()
|
||||||
@@ -709,8 +985,8 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
currentPartial = ""
|
currentPartial = ""
|
||||||
lastFinal = ""
|
lastFinal = ""
|
||||||
chunkWarnings = []
|
chunkWarnings = []
|
||||||
FlowSessionBridge.storeTranscriptionPartial("")
|
currentUtteranceId = nil
|
||||||
FlowSessionBridge.setRecordingState(.idle)
|
currentCommandSeq = 0
|
||||||
FlowLiveActivityController.update(phase: .idle)
|
FlowLiveActivityController.update(phase: .idle)
|
||||||
refreshHostReady()
|
refreshHostReady()
|
||||||
debug("utterance aborted")
|
debug("utterance aborted")
|
||||||
@@ -723,6 +999,8 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
isUtteranceRecording = false
|
isUtteranceRecording = false
|
||||||
isUtteranceProcessing = false
|
isUtteranceProcessing = false
|
||||||
utteranceRecordingStartedAt = nil
|
utteranceRecordingStartedAt = nil
|
||||||
|
utteranceSafetyTask?.cancel()
|
||||||
|
utteranceSafetyTask = nil
|
||||||
finalizeTask?.cancel()
|
finalizeTask?.cancel()
|
||||||
finalizeTask = nil
|
finalizeTask = nil
|
||||||
asrTask?.cancel()
|
asrTask?.cancel()
|
||||||
@@ -733,9 +1011,9 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
currentPartial = ""
|
currentPartial = ""
|
||||||
lastFinal = ""
|
lastFinal = ""
|
||||||
chunkWarnings = []
|
chunkWarnings = []
|
||||||
FlowSessionBridge.storeTranscriptionPartial("")
|
storeCurrentError(message, kind: kind)
|
||||||
FlowSessionBridge.storeTranscriptionError(message, kind: kind)
|
currentUtteranceId = nil
|
||||||
FlowSessionBridge.setRecordingState(.idle)
|
currentCommandSeq = 0
|
||||||
FlowLiveActivityController.update(phase: .idle)
|
FlowLiveActivityController.update(phase: .idle)
|
||||||
refreshHostReady()
|
refreshHostReady()
|
||||||
debug("utterance failed: \(message)")
|
debug("utterance failed: \(message)")
|
||||||
@@ -747,28 +1025,36 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
) {
|
) {
|
||||||
isUtteranceProcessing = false
|
isUtteranceProcessing = false
|
||||||
utteranceRecordingStartedAt = nil
|
utteranceRecordingStartedAt = nil
|
||||||
|
utteranceSafetyTask?.cancel()
|
||||||
|
utteranceSafetyTask = nil
|
||||||
finalizeTask?.cancel()
|
finalizeTask?.cancel()
|
||||||
finalizeTask = nil
|
finalizeTask = nil
|
||||||
chunkedPipeline = nil
|
chunkedPipeline = nil
|
||||||
currentPartial = ""
|
currentPartial = ""
|
||||||
lastFinal = ""
|
lastFinal = ""
|
||||||
chunkWarnings = []
|
chunkWarnings = []
|
||||||
FlowSessionBridge.storeTranscriptionPartial("")
|
storeCurrentError(message, kind: kind)
|
||||||
FlowSessionBridge.storeTranscriptionError(message, kind: kind)
|
currentUtteranceId = nil
|
||||||
FlowSessionBridge.setRecordingState(.idle)
|
currentCommandSeq = 0
|
||||||
FlowLiveActivityController.update(phase: .idle)
|
FlowLiveActivityController.update(phase: .idle)
|
||||||
refreshHostReady()
|
refreshHostReady()
|
||||||
debug("utterance processing failed: \(message)")
|
debug("utterance processing failed: \(message)")
|
||||||
}
|
}
|
||||||
|
|
||||||
private func finalizeUtterance() async {
|
private func finalizeUtterance() async {
|
||||||
|
let finalizeSessionId = activeSessionId
|
||||||
|
let finalizeUtteranceId = currentUtteranceId
|
||||||
let pipelineStarted = Date()
|
let pipelineStarted = Date()
|
||||||
defer {
|
defer {
|
||||||
isUtteranceProcessing = false
|
if activeSessionId == finalizeSessionId,
|
||||||
FlowSessionBridge.setRecordingState(.idle)
|
currentUtteranceId == finalizeUtteranceId {
|
||||||
FlowLiveActivityController.update(phase: .idle)
|
isUtteranceProcessing = false
|
||||||
touchSessionActivity()
|
FlowLiveActivityController.update(phase: .idle)
|
||||||
refreshHostReady()
|
touchSessionActivity()
|
||||||
|
currentUtteranceId = nil
|
||||||
|
currentCommandSeq = 0
|
||||||
|
refreshHostReady()
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
let asrWait = asrWaitTimeout()
|
let asrWait = asrWaitTimeout()
|
||||||
@@ -783,9 +1069,10 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
try? await Task.sleep(nanoseconds: 100_000_000)
|
try? await Task.sleep(nanoseconds: 100_000_000)
|
||||||
}
|
}
|
||||||
|
|
||||||
if lastFinal.isEmpty, let asrTask {
|
if lastFinal.isEmpty {
|
||||||
FlowDiagnostics.log("ASR wait elapsed — awaiting asrTask completion")
|
FlowDiagnostics.log("ASR wait elapsed — cancelling ASR task and using best available transcript")
|
||||||
_ = await asrTask.value
|
asrTask?.cancel()
|
||||||
|
Task { await chunkedPipeline?.cancel() }
|
||||||
}
|
}
|
||||||
|
|
||||||
let asrElapsed = Date().timeIntervalSince(pipelineStarted)
|
let asrElapsed = Date().timeIntervalSince(pipelineStarted)
|
||||||
@@ -803,10 +1090,7 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
(asrTask?.isCancelled == true) ? .recognitionInterrupted : .noSpeech
|
(asrTask?.isCancelled == true) ? .recognitionInterrupted : .noSpeech
|
||||||
FlowDiagnostics.log("finalize failed: empty transcript after \(String(format: "%.1f", asrElapsed))s")
|
FlowDiagnostics.log("finalize failed: empty transcript after \(String(format: "%.1f", asrElapsed))s")
|
||||||
utteranceRecordingStartedAt = nil
|
utteranceRecordingStartedAt = nil
|
||||||
FlowSessionBridge.storeTranscriptionError(
|
storeCurrentError(AppL10n.string(key), kind: kind)
|
||||||
AppL10n.string(key),
|
|
||||||
kind: kind
|
|
||||||
)
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -832,7 +1116,7 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
providerIdOverride: pipelineStore.polishProviderIdOverride
|
providerIdOverride: pipelineStore.polishProviderIdOverride
|
||||||
)
|
)
|
||||||
delivered = polished
|
delivered = polished
|
||||||
FlowSessionBridge.storeTranscriptionResult(polished, polishWarning: chunkNote)
|
storeCurrentFinal(polished, warning: chunkNote)
|
||||||
FlowDiagnostics.log(
|
FlowDiagnostics.log(
|
||||||
"polish done in \(String(format: "%.1f", Date().timeIntervalSince(polishStarted)))s " +
|
"polish done in \(String(format: "%.1f", Date().timeIntervalSince(polishStarted)))s " +
|
||||||
"total=\(String(format: "%.1f", Date().timeIntervalSince(pipelineStarted)))s"
|
"total=\(String(format: "%.1f", Date().timeIntervalSince(pipelineStarted)))s"
|
||||||
@@ -854,7 +1138,7 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
"\(error.localizedDescription)"
|
"\(error.localizedDescription)"
|
||||||
)
|
)
|
||||||
delivered = fallback.text
|
delivered = fallback.text
|
||||||
FlowSessionBridge.storeTranscriptionResult(fallback.text, polishWarning: fallback.polishWarning)
|
storeCurrentFinal(fallback.text, warning: fallback.polishWarning)
|
||||||
}
|
}
|
||||||
|
|
||||||
SpeechHistoryStore.shared.recordUtterance(
|
SpeechHistoryStore.shared.recordUtterance(
|
||||||
@@ -867,7 +1151,6 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
currentPartial = ""
|
currentPartial = ""
|
||||||
lastFinal = ""
|
lastFinal = ""
|
||||||
chunkWarnings = []
|
chunkWarnings = []
|
||||||
FlowSessionBridge.storeTranscriptionPartial("")
|
|
||||||
chunkedPipeline = nil
|
chunkedPipeline = nil
|
||||||
debug("utterance finalized length=\(text.count)")
|
debug("utterance finalized length=\(text.count)")
|
||||||
}
|
}
|
||||||
@@ -964,6 +1247,12 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
heartbeatTask?.cancel()
|
heartbeatTask?.cancel()
|
||||||
FlowSessionBridge.writeHeartbeat()
|
FlowSessionBridge.writeHeartbeat()
|
||||||
heartbeatTask = Task { @MainActor [weak self] in
|
heartbeatTask = Task { @MainActor [weak self] in
|
||||||
|
// Refresh the Live Activity `staleDate` every N heartbeat ticks
|
||||||
|
// (1 Hz) — well inside `FlowLiveActivityController.staleWindow` so a
|
||||||
|
// live session never looks stale, while a force-quit stops these
|
||||||
|
// refreshes and lets the system reclaim the orphaned island.
|
||||||
|
let liveActivityKeepAliveEveryTicks = 15
|
||||||
|
var tick = 0
|
||||||
while !Task.isCancelled {
|
while !Task.isCancelled {
|
||||||
guard let self else { break }
|
guard let self else { break }
|
||||||
if self.isActive, !self.capture.engineIsLive {
|
if self.isActive, !self.capture.engineIsLive {
|
||||||
@@ -971,6 +1260,10 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
}
|
}
|
||||||
FlowSessionBridge.writeHeartbeat()
|
FlowSessionBridge.writeHeartbeat()
|
||||||
self.refreshHostReady()
|
self.refreshHostReady()
|
||||||
|
tick += 1
|
||||||
|
if tick % liveActivityKeepAliveEveryTicks == 0 {
|
||||||
|
FlowLiveActivityController.keepAlive()
|
||||||
|
}
|
||||||
try? await Task.sleep(nanoseconds: 1_000_000_000)
|
try? await Task.sleep(nanoseconds: 1_000_000_000)
|
||||||
guard self.isActive else { break }
|
guard self.isActive else { break }
|
||||||
}
|
}
|
||||||
@@ -989,4 +1282,43 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
private func debug(_ message: String) {
|
private func debug(_ message: String) {
|
||||||
FlowDiagnostics.log(message)
|
FlowDiagnostics.log(message)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private func traceIgnoredCommand(reason: String, command: FlowCommand, detail: String) {
|
||||||
|
let signature = "\(reason)|\(command.action.rawValue)|\(command.commandSeq)|\(command.sessionId.uuidString)|\(command.utteranceId.uuidString)|\(detail)"
|
||||||
|
guard signature != lastIgnoredCommandSignature else { return }
|
||||||
|
lastIgnoredCommandSignature = signature
|
||||||
|
traceState(
|
||||||
|
"command.ignored",
|
||||||
|
extra: "reason=\(reason) action=\(command.action.rawValue) seq=\(command.commandSeq) \(detail)"
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
private func traceState(_ event: String, extra: String? = nil) {
|
||||||
|
let staleness = FlowSessionBridge.heartbeatStaleness().map { String(format: "%.1f", $0) } ?? "nil"
|
||||||
|
let sessionId = activeSessionId?.uuidString ?? "nil"
|
||||||
|
let utteranceId = currentUtteranceId?.uuidString ?? "nil"
|
||||||
|
let summary = [
|
||||||
|
"event=\(event)",
|
||||||
|
"active=\(isActive)",
|
||||||
|
"starting=\(isStarting)",
|
||||||
|
"coldStart=\(isColdStartHandoff)",
|
||||||
|
"sessionId=\(sessionId)",
|
||||||
|
"utteranceId=\(utteranceId)",
|
||||||
|
"cmdSeq=\(currentCommandSeq)",
|
||||||
|
"lastCmd=\(lastHandledCommandSeq)",
|
||||||
|
"recording=\(isUtteranceRecording)",
|
||||||
|
"processing=\(isUtteranceProcessing)",
|
||||||
|
"storeEngine=\(store.engineMode)",
|
||||||
|
"boundEngine=\(sessionASREngineMode ?? "nil")",
|
||||||
|
"engineLive=\(capture.engineIsLive)",
|
||||||
|
"hostReady=\(FlowSessionBridge.isHostReady())",
|
||||||
|
"sessionActive=\(FlowSessionBridge.isSessionActive())",
|
||||||
|
"heartbeatStaleness=\(staleness)"
|
||||||
|
].joined(separator: " ")
|
||||||
|
if let extra, !extra.isEmpty {
|
||||||
|
debug("[trace] \(summary) \(extra)")
|
||||||
|
} else {
|
||||||
|
debug("[trace] \(summary)")
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -38,6 +38,17 @@ struct FlowColdStartOverlay: View {
|
|||||||
/// Fraction of the screen height the bottom gradient occupies.
|
/// Fraction of the screen height the bottom gradient occupies.
|
||||||
private let gradientHeightFraction: CGFloat = 0.50
|
private let gradientHeightFraction: CGFloat = 0.50
|
||||||
|
|
||||||
|
/// Ready and failure states dismiss on blank tap; preparing stays
|
||||||
|
/// informational only (no accidental dismiss while proving audio).
|
||||||
|
private var allowsBlankTapDismiss: Bool {
|
||||||
|
switch context.state {
|
||||||
|
case .ready, .failed:
|
||||||
|
return true
|
||||||
|
case .preparing:
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
var body: some View {
|
var body: some View {
|
||||||
GeometryReader { geo in
|
GeometryReader { geo in
|
||||||
ZStack(alignment: .bottom) {
|
ZStack(alignment: .bottom) {
|
||||||
@@ -56,6 +67,16 @@ struct FlowColdStartOverlay: View {
|
|||||||
.frame(maxWidth: .infinity, alignment: .bottom)
|
.frame(maxWidth: .infinity, alignment: .bottom)
|
||||||
.allowsHitTesting(false)
|
.allowsHitTesting(false)
|
||||||
|
|
||||||
|
if allowsBlankTapDismiss {
|
||||||
|
// Captures taps on empty overlay space (dismiss) and blocks
|
||||||
|
// pass-through to the host shell underneath. Action buttons in
|
||||||
|
// `content` sit above this layer and remain tappable.
|
||||||
|
Color.clear
|
||||||
|
.contentShape(Rectangle())
|
||||||
|
.onTapGesture(perform: onDismiss)
|
||||||
|
.ignoresSafeArea()
|
||||||
|
}
|
||||||
|
|
||||||
VStack(spacing: Spacing.lg) {
|
VStack(spacing: Spacing.lg) {
|
||||||
content
|
content
|
||||||
.padding(.horizontal, Spacing.xl)
|
.padding(.horizontal, Spacing.xl)
|
||||||
@@ -63,14 +84,6 @@ struct FlowColdStartOverlay: View {
|
|||||||
homeIndicator
|
homeIndicator
|
||||||
.padding(.bottom, max(geo.safeAreaInsets.bottom, Spacing.sm))
|
.padding(.bottom, max(geo.safeAreaInsets.bottom, Spacing.sm))
|
||||||
}
|
}
|
||||||
.allowsHitTesting(false)
|
|
||||||
|
|
||||||
if context.state == .ready {
|
|
||||||
Color.clear
|
|
||||||
.contentShape(Rectangle())
|
|
||||||
.onTapGesture(perform: onDismiss)
|
|
||||||
.ignoresSafeArea()
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
.frame(maxWidth: .infinity, maxHeight: .infinity, alignment: .bottom)
|
.frame(maxWidth: .infinity, maxHeight: .infinity, alignment: .bottom)
|
||||||
.ignoresSafeArea()
|
.ignoresSafeArea()
|
||||||
|
|||||||
@@ -42,6 +42,10 @@ final class KeyboardFlowCoordinator {
|
|||||||
private static let hostReadyGrace: TimeInterval = 4
|
private static let hostReadyGrace: TimeInterval = 4
|
||||||
private var flowSessionMonitorTask: Task<Void, Never>?
|
private var flowSessionMonitorTask: Task<Void, Never>?
|
||||||
private var isAwaitingFlowResult = false
|
private var isAwaitingFlowResult = false
|
||||||
|
private var activeSessionId: UUID?
|
||||||
|
private var currentUtteranceId: UUID?
|
||||||
|
private var currentCommandSeq: Int64 = 0
|
||||||
|
private var lastAvailabilityTraceSignature = ""
|
||||||
|
|
||||||
init(
|
init(
|
||||||
state: KeyboardState,
|
state: KeyboardState,
|
||||||
@@ -133,7 +137,9 @@ final class KeyboardFlowCoordinator {
|
|||||||
|
|
||||||
private func recomputeMicVoiceAvailability() {
|
private func recomputeMicVoiceAvailability() {
|
||||||
FlowSessionBridge.reloadFromDisk()
|
FlowSessionBridge.reloadFromDisk()
|
||||||
let hostReady = FlowSessionBridge.isHostReady()
|
let readySnapshot = FlowSessionBridge.readySnapshot()
|
||||||
|
activeSessionId = readySnapshot?.sessionId ?? activeSessionId
|
||||||
|
let hostReady = readySnapshot?.ready == true && FlowSessionBridge.isHostReady()
|
||||||
let now = Date().timeIntervalSince1970
|
let now = Date().timeIntervalSince1970
|
||||||
if hostReady { lastHostReadyAt = now }
|
if hostReady { lastHostReadyAt = now }
|
||||||
// Grace window: the host was ready very recently, so treat a momentary
|
// Grace window: the host was ready very recently, so treat a momentary
|
||||||
@@ -145,7 +151,7 @@ final class KeyboardFlowCoordinator {
|
|||||||
let hostWarming = !hostReady
|
let hostWarming = !hostReady
|
||||||
&& FlowSessionBridge.isSessionActive()
|
&& FlowSessionBridge.isSessionActive()
|
||||||
&& (FlowSessionBridge.isHostReachable() || isPendingFlowStart || withinReadyGrace)
|
&& (FlowSessionBridge.isHostReachable() || isPendingFlowStart || withinReadyGrace)
|
||||||
state.flowSessionActive = hostReady
|
state.flowSessionActive = FlowSessionBridge.isSessionActive()
|
||||||
state.micVoiceAvailability = MicVoiceAvailabilityResolver.resolve(
|
state.micVoiceAvailability = MicVoiceAvailabilityResolver.resolve(
|
||||||
phase: state.phase,
|
phase: state.phase,
|
||||||
micDisabled: state.micDisabled,
|
micDisabled: state.micDisabled,
|
||||||
@@ -154,6 +160,20 @@ final class KeyboardFlowCoordinator {
|
|||||||
hostReady: hostReady,
|
hostReady: hostReady,
|
||||||
isPreparingSession: isPendingFlowStart || hostWarming
|
isPreparingSession: isPendingFlowStart || hostWarming
|
||||||
)
|
)
|
||||||
|
let signature = [
|
||||||
|
"phase=\(String(describing: state.phase))",
|
||||||
|
"availability=\(String(describing: state.micVoiceAvailability))",
|
||||||
|
hostReady ? "hostReady=1" : "hostReady=0",
|
||||||
|
state.flowSessionActive ? "sessionActive=1" : "sessionActive=0",
|
||||||
|
isPendingFlowStart ? "pending=1" : "pending=0",
|
||||||
|
isFlowRecording ? "recording=1" : "recording=0",
|
||||||
|
isAwaitingFlowResult ? "awaiting=1" : "awaiting=0",
|
||||||
|
readySnapshot?.reason.rawValue ?? "snapshot=nil"
|
||||||
|
].joined(separator: "|")
|
||||||
|
if signature != lastAvailabilityTraceSignature {
|
||||||
|
lastAvailabilityTraceSignature = signature
|
||||||
|
traceState("availability.update", extra: signature)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Session is live but the ready contract has not landed yet — poll
|
/// Session is live but the ready contract has not landed yet — poll
|
||||||
@@ -228,7 +248,8 @@ final class KeyboardFlowCoordinator {
|
|||||||
scheduleAutoClearError()
|
scheduleAutoClearError()
|
||||||
recomputeMicVoiceAvailability()
|
recomputeMicVoiceAvailability()
|
||||||
case .unavailable(.preparingSession):
|
case .unavailable(.preparingSession):
|
||||||
return
|
detectAndStoreAppContext()
|
||||||
|
beginFlowStart()
|
||||||
case .unavailable(.hostNotReady):
|
case .unavailable(.hostNotReady):
|
||||||
detectAndStoreAppContext()
|
detectAndStoreAppContext()
|
||||||
beginFlowStart()
|
beginFlowStart()
|
||||||
@@ -247,8 +268,8 @@ final class KeyboardFlowCoordinator {
|
|||||||
isFlowRecording = false
|
isFlowRecording = false
|
||||||
stopUtteranceCountdown()
|
stopUtteranceCountdown()
|
||||||
ExtensionScreenWakeLock.release()
|
ExtensionScreenWakeLock.release()
|
||||||
FlowSessionBridge.setRecordingState(.stopped)
|
writeCommand(.stopRecording)
|
||||||
debug("pressEnded wrote .stopped (readback=\(FlowSessionBridge.recordingState().rawValue))")
|
debug("pressEnded wrote stop command")
|
||||||
state.phase = .processing
|
state.phase = .processing
|
||||||
state.lastTranscript = ExtL10n.string("keyboard.flow.transcribing")
|
state.lastTranscript = ExtL10n.string("keyboard.flow.transcribing")
|
||||||
recomputeMicVoiceAvailability()
|
recomputeMicVoiceAvailability()
|
||||||
@@ -256,7 +277,10 @@ final class KeyboardFlowCoordinator {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func beginFlowStart() {
|
func beginFlowStart() {
|
||||||
guard !isPendingFlowStart else { return }
|
guard !isPendingFlowStart else {
|
||||||
|
traceState("beginFlowStart.ignored", extra: "reason=pendingAlreadyTrue")
|
||||||
|
return
|
||||||
|
}
|
||||||
isPendingFlowStart = true
|
isPendingFlowStart = true
|
||||||
isFlowRecording = false
|
isFlowRecording = false
|
||||||
flowStartDeadline = Date().timeIntervalSince1970 + FlowWatchdog.startTimeout
|
flowStartDeadline = Date().timeIntervalSince1970 + FlowWatchdog.startTimeout
|
||||||
@@ -264,11 +288,11 @@ final class KeyboardFlowCoordinator {
|
|||||||
recomputeMicVoiceAvailability()
|
recomputeMicVoiceAvailability()
|
||||||
openHostApp("startflow")
|
openHostApp("startflow")
|
||||||
startFlowStartWatchdog()
|
startFlowStartWatchdog()
|
||||||
debug("beginFlowStart")
|
traceState("beginFlowStart.started")
|
||||||
}
|
}
|
||||||
|
|
||||||
func handleHostAppOpenResult(path: String, success: Bool) {
|
func handleHostAppOpenResult(path: String, success: Bool) {
|
||||||
debug("openHostApp path=\(path) success=\(success)")
|
traceState("openHostApp.result", extra: "path=\(path) success=\(success)")
|
||||||
guard !success else { return }
|
guard !success else { return }
|
||||||
|
|
||||||
// The open genuinely failed (iOS blocked it / no Full Access). Don't
|
// The open genuinely failed (iOS blocked it / no Full Access). Don't
|
||||||
@@ -278,6 +302,7 @@ final class KeyboardFlowCoordinator {
|
|||||||
isPendingFlowStart = false
|
isPendingFlowStart = false
|
||||||
flowStartDeadline = 0
|
flowStartDeadline = 0
|
||||||
stopFlowWatchdog()
|
stopFlowWatchdog()
|
||||||
|
traceState("openHostApp.failed", extra: "path=startflow cancelPending=1")
|
||||||
showManualOpenHint(path: "startflow")
|
showManualOpenHint(path: "startflow")
|
||||||
recomputeMicVoiceAvailability()
|
recomputeMicVoiceAvailability()
|
||||||
return
|
return
|
||||||
@@ -290,9 +315,10 @@ final class KeyboardFlowCoordinator {
|
|||||||
guard !isAwaitingFlowResult else { return }
|
guard !isAwaitingFlowResult else { return }
|
||||||
if isFlowRecording || isPendingFlowStart {
|
if isFlowRecording || isPendingFlowStart {
|
||||||
if isFlowRecording {
|
if isFlowRecording {
|
||||||
FlowSessionBridge.setRecordingState(.aborted)
|
writeCommand(.abort)
|
||||||
ExtensionScreenWakeLock.release()
|
ExtensionScreenWakeLock.release()
|
||||||
}
|
}
|
||||||
|
currentUtteranceId = nil
|
||||||
isFlowRecording = false
|
isFlowRecording = false
|
||||||
isPendingFlowStart = false
|
isPendingFlowStart = false
|
||||||
stopUtteranceCountdown()
|
stopUtteranceCountdown()
|
||||||
@@ -304,17 +330,56 @@ final class KeyboardFlowCoordinator {
|
|||||||
|
|
||||||
// MARK: - Private
|
// MARK: - Private
|
||||||
|
|
||||||
|
private func nextCommandSeq() -> Int64 {
|
||||||
|
let millis = Int64(Date().timeIntervalSince1970 * 1_000)
|
||||||
|
currentCommandSeq = max(currentCommandSeq + 1, millis)
|
||||||
|
return currentCommandSeq
|
||||||
|
}
|
||||||
|
|
||||||
|
private func writeCommand(_ action: FlowCommand.Action) {
|
||||||
|
guard let activeSessionId, let currentUtteranceId else { return }
|
||||||
|
let command = FlowCommand(
|
||||||
|
sessionId: activeSessionId,
|
||||||
|
utteranceId: currentUtteranceId,
|
||||||
|
commandSeq: nextCommandSeq(),
|
||||||
|
action: action,
|
||||||
|
localeId: state.localeId
|
||||||
|
)
|
||||||
|
FlowSessionBridge.writeCommand(command)
|
||||||
|
debug(
|
||||||
|
"command \(action.rawValue) seq=\(command.commandSeq) " +
|
||||||
|
"utterance=\(currentUtteranceId.uuidString)"
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
private func consumePendingFlowDeliveryIfNeeded() {
|
private func consumePendingFlowDeliveryIfNeeded() {
|
||||||
if isAwaitingFlowResult {
|
if isAwaitingFlowResult {
|
||||||
if let delivery = FlowSessionBridge.consumeTranscriptionDelivery() {
|
if let result = matchingResult(), result.status == .final, let text = result.text, !text.isEmpty {
|
||||||
isAwaitingFlowResult = false
|
isAwaitingFlowResult = false
|
||||||
stopFlowWatchdog()
|
stopFlowWatchdog()
|
||||||
textInserter.handleFlowTranscript(delivery)
|
FlowSessionBridge.writeAck(
|
||||||
|
FlowAck(
|
||||||
|
sessionId: result.sessionId,
|
||||||
|
utteranceId: result.utteranceId,
|
||||||
|
commandSeq: result.commandSeq
|
||||||
|
)
|
||||||
|
)
|
||||||
|
FlowSessionBridge.clearResult()
|
||||||
|
currentUtteranceId = nil
|
||||||
|
textInserter.handleFlowTranscript(
|
||||||
|
TranscriptionDelivery(text: text, polishWarning: result.warning)
|
||||||
|
)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
if let error = FlowSessionBridge.consumeTranscriptionError() {
|
if let result = matchingResult(), isTerminalFailure(result) {
|
||||||
isAwaitingFlowResult = false
|
isAwaitingFlowResult = false
|
||||||
stopFlowWatchdog()
|
stopFlowWatchdog()
|
||||||
|
FlowSessionBridge.clearResult()
|
||||||
|
currentUtteranceId = nil
|
||||||
|
let error = FlowTranscriptionError(
|
||||||
|
message: result.text ?? ExtL10n.string("keyboard.flow.resultTimeout"),
|
||||||
|
kind: result.errorKind ?? .generic
|
||||||
|
)
|
||||||
state.phase = .error(
|
state.phase = .error(
|
||||||
.fromFlowTranscription(error),
|
.fromFlowTranscription(error),
|
||||||
message: error.message
|
message: error.message
|
||||||
@@ -330,6 +395,20 @@ final class KeyboardFlowCoordinator {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private func matchingResult() -> FlowResult? {
|
||||||
|
guard let result = FlowSessionBridge.latestResult() else { return nil }
|
||||||
|
guard let activeSessionId, let currentUtteranceId else { return nil }
|
||||||
|
guard result.sessionId == activeSessionId,
|
||||||
|
result.utteranceId == currentUtteranceId else {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
private func isTerminalFailure(_ result: FlowResult) -> Bool {
|
||||||
|
result.status == .error || result.status == .timeout || result.status == .aborted
|
||||||
|
}
|
||||||
|
|
||||||
/// When the host process died mid-utterance, abort local recording / waiting
|
/// When the host process died mid-utterance, abort local recording / waiting
|
||||||
/// so the user is not stuck until the long result watchdog fires.
|
/// so the user is not stuck until the long result watchdog fires.
|
||||||
private func recoverFromDeadHostIfNeeded() {
|
private func recoverFromDeadHostIfNeeded() {
|
||||||
@@ -339,7 +418,8 @@ final class KeyboardFlowCoordinator {
|
|||||||
isFlowRecording = false
|
isFlowRecording = false
|
||||||
stopUtteranceCountdown()
|
stopUtteranceCountdown()
|
||||||
ExtensionScreenWakeLock.release()
|
ExtensionScreenWakeLock.release()
|
||||||
FlowSessionBridge.setRecordingState(.aborted)
|
writeCommand(.abort)
|
||||||
|
currentUtteranceId = nil
|
||||||
stopFlowWatchdog()
|
stopFlowWatchdog()
|
||||||
state.level = 0
|
state.level = 0
|
||||||
state.phase = .idle
|
state.phase = .idle
|
||||||
@@ -355,12 +435,14 @@ final class KeyboardFlowCoordinator {
|
|||||||
}
|
}
|
||||||
|
|
||||||
private func failHostDisconnected() {
|
private func failHostDisconnected() {
|
||||||
|
traceState("hostDisconnected.fail")
|
||||||
isAwaitingFlowResult = false
|
isAwaitingFlowResult = false
|
||||||
isFlowRecording = false
|
isFlowRecording = false
|
||||||
isPendingFlowStart = false
|
isPendingFlowStart = false
|
||||||
stopUtteranceCountdown()
|
stopUtteranceCountdown()
|
||||||
ExtensionScreenWakeLock.release()
|
ExtensionScreenWakeLock.release()
|
||||||
FlowSessionBridge.setRecordingState(.aborted)
|
writeCommand(.abort)
|
||||||
|
currentUtteranceId = nil
|
||||||
stopFlowWatchdog()
|
stopFlowWatchdog()
|
||||||
state.level = 0
|
state.level = 0
|
||||||
let message = ExtL10n.string("keyboard.flow.hostDisconnected")
|
let message = ExtL10n.string("keyboard.flow.hostDisconnected")
|
||||||
@@ -396,6 +478,7 @@ final class KeyboardFlowCoordinator {
|
|||||||
private func startFlowRecording() {
|
private func startFlowRecording() {
|
||||||
recomputeMicVoiceAvailability()
|
recomputeMicVoiceAvailability()
|
||||||
guard state.micVoiceAvailability.isReady else {
|
guard state.micVoiceAvailability.isReady else {
|
||||||
|
traceState("startFlowRecording.blocked", extra: "availability=\(String(describing: state.micVoiceAvailability))")
|
||||||
beginFlowStart()
|
beginFlowStart()
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -403,8 +486,14 @@ final class KeyboardFlowCoordinator {
|
|||||||
flowStartDeadline = 0
|
flowStartDeadline = 0
|
||||||
stopFlowWatchdog()
|
stopFlowWatchdog()
|
||||||
|
|
||||||
FlowSessionBridge.setTranscriptionLanguage(state.localeId)
|
guard let sessionId = FlowSessionBridge.readySnapshot()?.sessionId else {
|
||||||
FlowSessionBridge.setRecordingState(.recording)
|
traceState("startFlowRecording.blocked", extra: "reason=missingSessionIdInReadySnapshot")
|
||||||
|
beginFlowStart()
|
||||||
|
return
|
||||||
|
}
|
||||||
|
activeSessionId = sessionId
|
||||||
|
currentUtteranceId = UUID()
|
||||||
|
writeCommand(.startRecording)
|
||||||
isFlowRecording = true
|
isFlowRecording = true
|
||||||
state.lastTranscript = ""
|
state.lastTranscript = ""
|
||||||
state.phase = .recording
|
state.phase = .recording
|
||||||
@@ -414,9 +503,7 @@ final class KeyboardFlowCoordinator {
|
|||||||
}
|
}
|
||||||
startUtteranceCountdown()
|
startUtteranceCountdown()
|
||||||
startFlowLevelWatchdog()
|
startFlowLevelWatchdog()
|
||||||
// Read back in-process to confirm the write landed before we rely on
|
traceState("startFlowRecording.started")
|
||||||
// the host polling it out cross-process.
|
|
||||||
debug("startFlowRecording wrote .recording (readback=\(FlowSessionBridge.recordingState().rawValue))")
|
|
||||||
}
|
}
|
||||||
|
|
||||||
private func startUtteranceCountdown() {
|
private func startUtteranceCountdown() {
|
||||||
@@ -450,6 +537,7 @@ final class KeyboardFlowCoordinator {
|
|||||||
state.phase = .idle
|
state.phase = .idle
|
||||||
state.lastTranscript = ""
|
state.lastTranscript = ""
|
||||||
recomputeMicVoiceAvailability()
|
recomputeMicVoiceAvailability()
|
||||||
|
traceState("pendingStart.cancelledByUser")
|
||||||
}
|
}
|
||||||
|
|
||||||
private func startFlowStartWatchdog() {
|
private func startFlowStartWatchdog() {
|
||||||
@@ -465,6 +553,7 @@ final class KeyboardFlowCoordinator {
|
|||||||
if self.flowStartDeadline > 0, now > self.flowStartDeadline {
|
if self.flowStartDeadline > 0, now > self.flowStartDeadline {
|
||||||
self.isPendingFlowStart = false
|
self.isPendingFlowStart = false
|
||||||
self.flowStartDeadline = 0
|
self.flowStartDeadline = 0
|
||||||
|
self.traceState("startWatchdog.timeout")
|
||||||
self.showManualOpenHint(path: "startflow")
|
self.showManualOpenHint(path: "startflow")
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -480,7 +569,7 @@ final class KeyboardFlowCoordinator {
|
|||||||
state.lastTranscript = ""
|
state.lastTranscript = ""
|
||||||
refreshSessionState()
|
refreshSessionState()
|
||||||
startFlowRecording()
|
startFlowRecording()
|
||||||
debug("completeFlowStartHandoff → auto startFlowRecording")
|
traceState("completeFlowStartHandoff.done")
|
||||||
}
|
}
|
||||||
|
|
||||||
private func startFlowLevelWatchdog() {
|
private func startFlowLevelWatchdog() {
|
||||||
@@ -492,6 +581,12 @@ final class KeyboardFlowCoordinator {
|
|||||||
self.state.level = Double(peak)
|
self.state.level = Double(peak)
|
||||||
}
|
}
|
||||||
self.refreshFlowPartialIfNeeded()
|
self.refreshFlowPartialIfNeeded()
|
||||||
|
let staleness = FlowSessionBridge.heartbeatStaleness() ?? .infinity
|
||||||
|
if staleness > 5 {
|
||||||
|
self.debug("levelWatchdog: host heartbeat stale while recording")
|
||||||
|
self.failHostDisconnected()
|
||||||
|
return
|
||||||
|
}
|
||||||
try? await Task.sleep(nanoseconds: FlowWatchdog.pollIntervalNs)
|
try? await Task.sleep(nanoseconds: FlowWatchdog.pollIntervalNs)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -501,7 +596,10 @@ final class KeyboardFlowCoordinator {
|
|||||||
guard isFlowRecording || isAwaitingFlowResult else { return }
|
guard isFlowRecording || isAwaitingFlowResult else { return }
|
||||||
switch state.phase {
|
switch state.phase {
|
||||||
case .recording, .processing:
|
case .recording, .processing:
|
||||||
if let partial = FlowSessionBridge.transcriptionPartial() {
|
if let result = matchingResult(),
|
||||||
|
result.status == .partial,
|
||||||
|
let partial = result.text,
|
||||||
|
!partial.isEmpty {
|
||||||
state.lastTranscript = partial
|
state.lastTranscript = partial
|
||||||
}
|
}
|
||||||
default:
|
default:
|
||||||
@@ -517,16 +615,33 @@ final class KeyboardFlowCoordinator {
|
|||||||
debug("resultWatchdog started timeout=\(Int(resultTimeout))s engine=\(state.engineMode)")
|
debug("resultWatchdog started timeout=\(Int(resultTimeout))s engine=\(state.engineMode)")
|
||||||
flowWatchdogTask = Task { @MainActor [weak self] in
|
flowWatchdogTask = Task { @MainActor [weak self] in
|
||||||
while let self, !Task.isCancelled {
|
while let self, !Task.isCancelled {
|
||||||
if let delivery = FlowSessionBridge.consumeTranscriptionDelivery() {
|
if let result = self.matchingResult(), result.status == .final, let text = result.text, !text.isEmpty {
|
||||||
self.isAwaitingFlowResult = false
|
self.isAwaitingFlowResult = false
|
||||||
self.stopFlowWatchdog()
|
self.stopFlowWatchdog()
|
||||||
self.debug("resultWatchdog consumed delivery len=\(delivery.text.count)")
|
FlowSessionBridge.writeAck(
|
||||||
self.textInserter.handleFlowTranscript(delivery)
|
FlowAck(
|
||||||
|
sessionId: result.sessionId,
|
||||||
|
utteranceId: result.utteranceId,
|
||||||
|
commandSeq: result.commandSeq
|
||||||
|
)
|
||||||
|
)
|
||||||
|
FlowSessionBridge.clearResult()
|
||||||
|
self.currentUtteranceId = nil
|
||||||
|
self.debug("resultWatchdog consumed delivery len=\(text.count)")
|
||||||
|
self.textInserter.handleFlowTranscript(
|
||||||
|
TranscriptionDelivery(text: text, polishWarning: result.warning)
|
||||||
|
)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
if let error = FlowSessionBridge.consumeTranscriptionError() {
|
if let result = self.matchingResult(), self.isTerminalFailure(result) {
|
||||||
self.isAwaitingFlowResult = false
|
self.isAwaitingFlowResult = false
|
||||||
self.stopFlowWatchdog()
|
self.stopFlowWatchdog()
|
||||||
|
FlowSessionBridge.clearResult()
|
||||||
|
self.currentUtteranceId = nil
|
||||||
|
let error = FlowTranscriptionError(
|
||||||
|
message: result.text ?? ExtL10n.string("keyboard.flow.resultTimeout"),
|
||||||
|
kind: result.errorKind ?? .generic
|
||||||
|
)
|
||||||
self.debug("resultWatchdog consumed error kind=\(error.kind.rawValue)")
|
self.debug("resultWatchdog consumed error kind=\(error.kind.rawValue)")
|
||||||
self.state.phase = .error(
|
self.state.phase = .error(
|
||||||
.fromFlowTranscription(error),
|
.fromFlowTranscription(error),
|
||||||
@@ -539,6 +654,11 @@ final class KeyboardFlowCoordinator {
|
|||||||
self.refreshFlowPartialIfNeeded()
|
self.refreshFlowPartialIfNeeded()
|
||||||
let now = Date().timeIntervalSince1970
|
let now = Date().timeIntervalSince1970
|
||||||
let staleness = FlowSessionBridge.heartbeatStaleness() ?? .infinity
|
let staleness = FlowSessionBridge.heartbeatStaleness() ?? .infinity
|
||||||
|
if self.isFlowRecording, staleness > 5 {
|
||||||
|
self.debug("level/result watchdog: host heartbeat stale while recording")
|
||||||
|
self.failHostDisconnected()
|
||||||
|
return
|
||||||
|
}
|
||||||
if staleness > FlowSessionKeys.heartbeatZombieInterval {
|
if staleness > FlowSessionKeys.heartbeatZombieInterval {
|
||||||
self.debug("resultWatchdog: host heartbeat zombie (staleness=\(String(format: "%.1f", staleness))s)")
|
self.debug("resultWatchdog: host heartbeat zombie (staleness=\(String(format: "%.1f", staleness))s)")
|
||||||
self.failHostDisconnected()
|
self.failHostDisconnected()
|
||||||
@@ -553,6 +673,7 @@ final class KeyboardFlowCoordinator {
|
|||||||
if now - startedAt > resultTimeout {
|
if now - startedAt > resultTimeout {
|
||||||
self.isAwaitingFlowResult = false
|
self.isAwaitingFlowResult = false
|
||||||
self.stopFlowWatchdog()
|
self.stopFlowWatchdog()
|
||||||
|
self.currentUtteranceId = nil
|
||||||
self.debug("resultWatchdog TIMEOUT after \(Int(resultTimeout))s — no result from host")
|
self.debug("resultWatchdog TIMEOUT after \(Int(resultTimeout))s — no result from host")
|
||||||
let msg = ExtL10n.string("keyboard.flow.resultTimeout")
|
let msg = ExtL10n.string("keyboard.flow.resultTimeout")
|
||||||
self.state.phase = .error(.flowResultTimeout, message: msg)
|
self.state.phase = .error(.flowResultTimeout, message: msg)
|
||||||
@@ -573,4 +694,29 @@ final class KeyboardFlowCoordinator {
|
|||||||
private func debug(_ message: String) {
|
private func debug(_ message: String) {
|
||||||
OSGLog.keyboardExt.info("\(message, privacy: .public)")
|
OSGLog.keyboardExt.info("\(message, privacy: .public)")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private func traceState(_ event: String, extra: String? = nil) {
|
||||||
|
let staleness = FlowSessionBridge.heartbeatStaleness().map { String(format: "%.1f", $0) } ?? "nil"
|
||||||
|
let sessionId = activeSessionId?.uuidString ?? "nil"
|
||||||
|
let utteranceId = currentUtteranceId?.uuidString ?? "nil"
|
||||||
|
let summary = [
|
||||||
|
"event=\(event)",
|
||||||
|
"phase=\(String(describing: state.phase))",
|
||||||
|
"availability=\(String(describing: state.micVoiceAvailability))",
|
||||||
|
"pending=\(isPendingFlowStart)",
|
||||||
|
"recording=\(isFlowRecording)",
|
||||||
|
"awaiting=\(isAwaitingFlowResult)",
|
||||||
|
"sessionId=\(sessionId)",
|
||||||
|
"utteranceId=\(utteranceId)",
|
||||||
|
"cmdSeq=\(currentCommandSeq)",
|
||||||
|
"sessionActive=\(FlowSessionBridge.isSessionActive())",
|
||||||
|
"hostReady=\(FlowSessionBridge.isHostReady())",
|
||||||
|
"heartbeatStaleness=\(staleness)"
|
||||||
|
].joined(separator: " ")
|
||||||
|
if let extra, !extra.isEmpty {
|
||||||
|
debug("[trace] \(summary) \(extra)")
|
||||||
|
} else {
|
||||||
|
debug("[trace] \(summary)")
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -52,7 +52,7 @@ private struct FlowLiveActivityLockScreenView: View {
|
|||||||
|
|
||||||
var body: some View {
|
var body: some View {
|
||||||
HStack(spacing: 12) {
|
HStack(spacing: 12) {
|
||||||
FlowLiveActivityBrandMark(height: 18)
|
FlowLiveActivityBrandMark(height: 13)
|
||||||
VStack(alignment: .leading, spacing: 4) {
|
VStack(alignment: .leading, spacing: 4) {
|
||||||
Text("OSGKeyboard")
|
Text("OSGKeyboard")
|
||||||
.font(.headline)
|
.font(.headline)
|
||||||
@@ -99,9 +99,10 @@ private struct FlowLiveActivityTrailingGlyph: View {
|
|||||||
.foregroundStyle(.red)
|
.foregroundStyle(.red)
|
||||||
.symbolEffect(.variableColor.iterative, options: .repeating)
|
.symbolEffect(.variableColor.iterative, options: .repeating)
|
||||||
case .processing:
|
case .processing:
|
||||||
ProgressView()
|
Image(systemName: "ellipsis")
|
||||||
.progressViewStyle(.circular)
|
.font(.title3.weight(.semibold))
|
||||||
.tint(.white)
|
.foregroundStyle(.white)
|
||||||
|
.symbolEffect(.variableColor.iterative, options: .repeating)
|
||||||
case .idle:
|
case .idle:
|
||||||
// Session ready but NOT listening — avoid a mic glyph so users
|
// Session ready but NOT listening — avoid a mic glyph so users
|
||||||
// don't think the keyboard is recording in the background.
|
// don't think the keyboard is recording in the background.
|
||||||
|
|||||||
@@ -183,37 +183,48 @@ struct BottomDictationBar: View {
|
|||||||
.fixedSize()
|
.fixedSize()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// 麦克风按钮始终居中固定:录音时的波形放进按钮内部,
|
||||||
|
// “按停止”提示作为浮层显示在按钮上方,二者均不参与布局,
|
||||||
|
// 因此按下 Option 触发录音时按钮位置不会发生偏移。
|
||||||
private var recordControl: some View {
|
private var recordControl: some View {
|
||||||
HStack(spacing: Spacing.sm) {
|
recordButton
|
||||||
if viewModel.isRecording {
|
.overlay(alignment: .top) {
|
||||||
MiniWaveform(level: viewModel.audioLevel)
|
if viewModel.isRecording {
|
||||||
|
Text(MacL10n.string("mac.record.pressStop", language: lang))
|
||||||
|
.font(TypeStyle.caption)
|
||||||
|
.foregroundStyle(palette.textTertiary)
|
||||||
|
.fixedSize()
|
||||||
|
.offset(y: -22)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
Button(action: viewModel.toggleRecording) {
|
}
|
||||||
ZStack {
|
|
||||||
Circle()
|
private var recordButton: some View {
|
||||||
.fill(viewModel.isRecording ? palette.recordRed : palette.accent)
|
Button(action: viewModel.toggleRecording) {
|
||||||
.frame(width: 52, height: 52)
|
ZStack {
|
||||||
.macGlassSurface(in: Circle(), fillOpacity: 0.2)
|
Circle()
|
||||||
.shadow(
|
.fill(viewModel.isRecording ? palette.recordRed : palette.accent)
|
||||||
color: (viewModel.isRecording ? palette.recordRed : palette.accent).opacity(0.5),
|
.frame(width: 52, height: 52)
|
||||||
radius: pulse ? 14 : 6
|
.macGlassSurface(in: Circle(), fillOpacity: 0.2)
|
||||||
)
|
.shadow(
|
||||||
Image(systemName: viewModel.isRecording ? "stop.fill" : "mic.fill")
|
color: (viewModel.isRecording ? palette.recordRed : palette.accent).opacity(0.5),
|
||||||
|
radius: pulse ? 14 : 6
|
||||||
|
)
|
||||||
|
if viewModel.isRecording {
|
||||||
|
// 与 iOS 一致:录音时在红色按钮内部显示实时波形
|
||||||
|
MiniWaveform(level: viewModel.audioLevel, barCount: 4, tint: palette.textOnAccent)
|
||||||
|
} else {
|
||||||
|
Image(systemName: "mic.fill")
|
||||||
.font(.system(size: 20, weight: .bold))
|
.font(.system(size: 20, weight: .bold))
|
||||||
.foregroundStyle(palette.textOnAccent)
|
.foregroundStyle(palette.textOnAccent)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
.buttonStyle(.plain)
|
}
|
||||||
.disabled(viewModel.isProcessing)
|
.buttonStyle(.plain)
|
||||||
.onAppear {
|
.disabled(viewModel.isProcessing)
|
||||||
withAnimation(.easeInOut(duration: 1.1).repeatForever(autoreverses: true)) {
|
.onAppear {
|
||||||
pulse = true
|
withAnimation(.easeInOut(duration: 1.1).repeatForever(autoreverses: true)) {
|
||||||
}
|
pulse = true
|
||||||
}
|
|
||||||
if viewModel.isRecording {
|
|
||||||
Text(MacL10n.string("mac.record.pressStop", language: lang))
|
|
||||||
.font(TypeStyle.caption)
|
|
||||||
.foregroundStyle(palette.textTertiary)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -31,9 +31,30 @@ enum MacDictationPipeline {
|
|||||||
|
|
||||||
let locale = Locale(identifier: store.localeId.isEmpty ? "zh-CN" : store.localeId)
|
let locale = Locale(identifier: store.localeId.isEmpty ? "zh-CN" : store.localeId)
|
||||||
let raw: String
|
let raw: String
|
||||||
|
var localBias: LocalASRBiasPayload?
|
||||||
|
|
||||||
if store.engineMode == "local" {
|
if store.engineMode == "local" {
|
||||||
raw = try await MacLocalASRService.transcribe(samples: samples, locale: locale)
|
MacAppContextService.captureAndPersist(to: store)
|
||||||
|
let capabilities = MacLocalASRService.currentCapabilities()
|
||||||
|
let bias = LocalASRBiasAdapter.adapt(
|
||||||
|
LocalASRBiasRequest(
|
||||||
|
dictionary: store.personalDictionary,
|
||||||
|
locale: locale,
|
||||||
|
frontAppBundleId: MacAppContextService.frontmostBundleIdentifier(),
|
||||||
|
capabilities: capabilities
|
||||||
|
)
|
||||||
|
)
|
||||||
|
localBias = bias
|
||||||
|
LocalASRBiasDiagnosticsStore.save(
|
||||||
|
payload: bias,
|
||||||
|
modelId: MacLocalASRService.selectedModelDefinition()?.id,
|
||||||
|
backendLabel: MacLocalASRService.currentBackendLabel()
|
||||||
|
)
|
||||||
|
raw = try await MacLocalASRService.transcribe(
|
||||||
|
samples: samples,
|
||||||
|
locale: locale,
|
||||||
|
bias: bias
|
||||||
|
)
|
||||||
} else {
|
} else {
|
||||||
let strategy = CloudASRModelCatalog.strategy(for: store.providerId)
|
let strategy = CloudASRModelCatalog.strategy(for: store.providerId)
|
||||||
guard strategy != .localFallback else { throw MacDictationError.providerHasNoCloudASR }
|
guard strategy != .localFallback else { throw MacDictationError.providerHasNoCloudASR }
|
||||||
@@ -51,13 +72,33 @@ enum MacDictationPipeline {
|
|||||||
let trimmed = raw.trimmingCharacters(in: .whitespacesAndNewlines)
|
let trimmed = raw.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||||
guard !trimmed.isEmpty else { throw MacDictationError.emptyTranscript }
|
guard !trimmed.isEmpty else { throw MacDictationError.emptyTranscript }
|
||||||
|
|
||||||
|
let postASR: String
|
||||||
|
if let localBias, !localBias.correctionPairs.isEmpty {
|
||||||
|
postASR = LocalASRTranscriptCorrector.apply(trimmed, pairs: localBias.correctionPairs)
|
||||||
|
} else {
|
||||||
|
postASR = trimmed
|
||||||
|
}
|
||||||
|
|
||||||
|
let polishContext: PolishContext?
|
||||||
|
if let supplement = localBias?.polishFragment.trimmingCharacters(in: .whitespacesAndNewlines),
|
||||||
|
!supplement.isEmpty {
|
||||||
|
polishContext = PolishContext(
|
||||||
|
appContext: store.detectedAppContext?.context ?? .unknown,
|
||||||
|
intensity: store.polishIntensity,
|
||||||
|
dictionarySupplement: supplement
|
||||||
|
)
|
||||||
|
} else {
|
||||||
|
polishContext = nil
|
||||||
|
}
|
||||||
|
|
||||||
if let polished = try? await PolishingService(store: store).polish(
|
if let polished = try? await PolishingService(store: store).polish(
|
||||||
trimmed,
|
postASR,
|
||||||
mode: store.polishModeForPipeline
|
mode: store.polishModeForPipeline,
|
||||||
|
context: polishContext
|
||||||
),
|
),
|
||||||
!polished.isEmpty {
|
!polished.isEmpty {
|
||||||
return polished
|
return polished
|
||||||
}
|
}
|
||||||
return trimmed
|
return postASR
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -107,8 +107,9 @@ final class MacDictationViewModel: ObservableObject {
|
|||||||
/// Pre-load MLX weights + Metal shaders so the first dictation is fast.
|
/// Pre-load MLX weights + Metal shaders so the first dictation is fast.
|
||||||
func warmUpQwen3IfNeeded() {
|
func warmUpQwen3IfNeeded() {
|
||||||
guard config.engineMode == "local",
|
guard config.engineMode == "local",
|
||||||
MacLocalASRPreferences.backend == .qwen3MLX,
|
let model = MacLocalASRService.selectedModelDefinition(),
|
||||||
MacLocalASRPreferences.qwen3ModelIsInstalled() else { return }
|
model.backend == .mlx,
|
||||||
|
MacLocalASRService.isModelInstalled(model) else { return }
|
||||||
let path = MacLocalASRPreferences.qwen3ModelPath
|
let path = MacLocalASRPreferences.qwen3ModelPath
|
||||||
Task.detached(priority: .utility) {
|
Task.detached(priority: .utility) {
|
||||||
_ = try? await MacQwen3ASREngine.shared.prepareIfNeeded(modelPath: path)
|
_ = try? await MacQwen3ASREngine.shared.prepareIfNeeded(modelPath: path)
|
||||||
@@ -156,6 +157,39 @@ final class MacDictationViewModel: ObservableObject {
|
|||||||
return "\(seconds)s"
|
return "\(seconds)s"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
var localModelReady: Bool {
|
||||||
|
_ = localModelRevision
|
||||||
|
if let model = MacLocalASRService.selectedModelDefinition() {
|
||||||
|
return MacLocalASRService.isModelInstalled(model)
|
||||||
|
}
|
||||||
|
return MacLocalASRPreferences.qwen3ModelIsInstalled()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Context-aware warning when local engine is selected but the active model is not ready.
|
||||||
|
var localModelWarningMessage: String? {
|
||||||
|
_ = localModelRevision
|
||||||
|
guard config.engineMode == "local" else { return nil }
|
||||||
|
if localModelReady { return nil }
|
||||||
|
guard let model = MacLocalASRService.selectedModelDefinition() else {
|
||||||
|
return MacL10n.string("mac.settings.localModelFallbackApple", language: config.uiLanguage)
|
||||||
|
}
|
||||||
|
if model.installKind == .manual {
|
||||||
|
return MacL10n.string("mac.settings.mlxModelMissing", language: config.uiLanguage)
|
||||||
|
}
|
||||||
|
return MacL10n.format(
|
||||||
|
"mac.settings.selectedModelMissing",
|
||||||
|
language: config.uiLanguage,
|
||||||
|
model.displayName
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
@Published private(set) var localModelRevision = 0
|
||||||
|
|
||||||
|
func bumpLocalModelRevision() {
|
||||||
|
localModelRevision += 1
|
||||||
|
objectWillChange.send()
|
||||||
|
}
|
||||||
|
|
||||||
var qwen3ModelInstalled: Bool {
|
var qwen3ModelInstalled: Bool {
|
||||||
MacLocalASRPreferences.qwen3ModelIsInstalled()
|
MacLocalASRPreferences.qwen3ModelIsInstalled()
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,455 @@
|
|||||||
|
// MacLocalASRModelSettingsView.swift
|
||||||
|
// OSGKeyboard · Mac
|
||||||
|
//
|
||||||
|
// Local ASR model catalog, download progress, MLX path, and bias diagnostics.
|
||||||
|
|
||||||
|
import AppKit
|
||||||
|
import SwiftUI
|
||||||
|
|
||||||
|
@MainActor
|
||||||
|
final class MacLocalASRModelSettingsViewModel: ObservableObject {
|
||||||
|
@Published var catalog: LocalASRCatalogDocument?
|
||||||
|
@Published var selectedModelId: String = MacLocalASRPreferences.selectedModelId
|
||||||
|
@Published var installProgress = LocalASRModelInstallProgress.idle
|
||||||
|
@Published var diagnosticsSnapshot = LocalASRBiasDiagnosticsStore.load()
|
||||||
|
@Published var statusMessage = ""
|
||||||
|
@Published var isInstalling = false
|
||||||
|
@Published var isDownloadPaused = false
|
||||||
|
|
||||||
|
var onLocalModelStateChanged: (() -> Void)?
|
||||||
|
|
||||||
|
private let manager = LocalASRModelManager.shared
|
||||||
|
private var progressPollTask: Task<Void, Never>?
|
||||||
|
|
||||||
|
deinit {
|
||||||
|
progressPollTask?.cancel()
|
||||||
|
}
|
||||||
|
|
||||||
|
func reload() {
|
||||||
|
catalog = try? LocalASRModelCatalog.loadBundled()
|
||||||
|
if let catalog {
|
||||||
|
let manifest = LocalASRInstalledManifestIO.load(defaultModelId: catalog.defaultModelId)
|
||||||
|
selectedModelId = manifest.selectedModelId.isEmpty
|
||||||
|
? MacLocalASRPreferences.selectedModelId
|
||||||
|
: manifest.selectedModelId
|
||||||
|
}
|
||||||
|
diagnosticsSnapshot = LocalASRBiasDiagnosticsStore.load()
|
||||||
|
onLocalModelStateChanged?()
|
||||||
|
}
|
||||||
|
|
||||||
|
func isInstalled(_ model: LocalASRModelDefinition) -> Bool {
|
||||||
|
MacLocalASRService.isModelInstalled(model)
|
||||||
|
}
|
||||||
|
|
||||||
|
func isInstallingModel(_ model: LocalASRModelDefinition) -> Bool {
|
||||||
|
isInstalling && installProgress.activeItemId == model.id
|
||||||
|
}
|
||||||
|
|
||||||
|
func installedDiskUsage(_ model: LocalASRModelDefinition) -> String? {
|
||||||
|
guard model.installKind == .archive,
|
||||||
|
let relative = model.installRelativePath,
|
||||||
|
isInstalled(model) else { return nil }
|
||||||
|
let dir = LocalASRModelInstallState.installDirectory(for: relative)
|
||||||
|
let bytes = LocalASRModelInstallState.directoryByteCount(at: dir)
|
||||||
|
guard bytes > 0 else { return nil }
|
||||||
|
return ByteCountFormatter.string(fromByteCount: bytes, countStyle: .file)
|
||||||
|
}
|
||||||
|
|
||||||
|
func currentRuntime(in catalog: LocalASRCatalogDocument) -> LocalASRRuntimeDefinition? {
|
||||||
|
LocalASRModelCatalog.runtime(for: LocalASRModelCatalog.currentRuntimePlatform(), in: catalog)
|
||||||
|
}
|
||||||
|
|
||||||
|
func isRuntimeInstalled(_ runtime: LocalASRRuntimeDefinition) -> Bool {
|
||||||
|
LocalASRModelInstallState.isRuntimeInstalled(runtime)
|
||||||
|
}
|
||||||
|
|
||||||
|
func selectModel(_ modelId: String) {
|
||||||
|
guard let catalog, !isInstalling else { return }
|
||||||
|
selectedModelId = modelId
|
||||||
|
MacLocalASRPreferences.selectedModelId = modelId
|
||||||
|
var manifest = LocalASRInstalledManifestIO.load(defaultModelId: catalog.defaultModelId)
|
||||||
|
manifest.selectedModelId = modelId
|
||||||
|
manifest.updatedAt = Date()
|
||||||
|
try? LocalASRInstalledManifestIO.save(manifest)
|
||||||
|
onLocalModelStateChanged?()
|
||||||
|
}
|
||||||
|
|
||||||
|
func installModel(_ model: LocalASRModelDefinition) {
|
||||||
|
guard let catalog, !isInstalling else { return }
|
||||||
|
statusMessage = ""
|
||||||
|
isInstalling = true
|
||||||
|
isDownloadPaused = false
|
||||||
|
startProgressPolling()
|
||||||
|
Task {
|
||||||
|
do {
|
||||||
|
try await manager.installModel(model, catalog: catalog)
|
||||||
|
installProgress = await manager.currentProgress()
|
||||||
|
selectModel(model.id)
|
||||||
|
statusMessage = MacL10n.string("mac.localASR.installDone")
|
||||||
|
} catch {
|
||||||
|
installProgress = await manager.currentProgress()
|
||||||
|
statusMessage = error.localizedDescription
|
||||||
|
}
|
||||||
|
isInstalling = false
|
||||||
|
isDownloadPaused = false
|
||||||
|
stopProgressPolling()
|
||||||
|
reload()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func pauseDownload() {
|
||||||
|
Task {
|
||||||
|
do {
|
||||||
|
try await manager.pauseDownload()
|
||||||
|
isDownloadPaused = true
|
||||||
|
installProgress = await manager.currentProgress()
|
||||||
|
} catch {
|
||||||
|
statusMessage = error.localizedDescription
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func resumeDownload() {
|
||||||
|
Task {
|
||||||
|
do {
|
||||||
|
try await manager.resumeDownload()
|
||||||
|
isDownloadPaused = false
|
||||||
|
installProgress = await manager.currentProgress()
|
||||||
|
} catch {
|
||||||
|
statusMessage = error.localizedDescription
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func deleteModel(_ model: LocalASRModelDefinition) {
|
||||||
|
guard let catalog, !isInstalling else { return }
|
||||||
|
Task {
|
||||||
|
do {
|
||||||
|
try await manager.deleteModel(model, catalog: catalog)
|
||||||
|
statusMessage = MacL10n.string("mac.localASR.deleteDone")
|
||||||
|
reload()
|
||||||
|
} catch {
|
||||||
|
statusMessage = error.localizedDescription
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func revealModelInFinder(_ model: LocalASRModelDefinition) {
|
||||||
|
guard let relative = model.installRelativePath else { return }
|
||||||
|
let url = LocalASRModelInstallState.installDirectory(for: relative)
|
||||||
|
NSWorkspace.shared.activateFileViewerSelecting([url])
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Opens (creating if needed) the model's shared subfolder so the user can
|
||||||
|
/// drop in manually-converted weights (used by the MLX model).
|
||||||
|
func revealModelFolder(_ model: LocalASRModelDefinition) {
|
||||||
|
guard let relative = model.installRelativePath else { return }
|
||||||
|
let url = LocalASRModelInstallState.installDirectory(for: relative)
|
||||||
|
try? FileManager.default.createDirectory(at: url, withIntermediateDirectories: true)
|
||||||
|
NSWorkspace.shared.open(url)
|
||||||
|
}
|
||||||
|
|
||||||
|
func revealStorageRoot() {
|
||||||
|
let url = LocalASRModelInstallState.rootDirectory()
|
||||||
|
try? FileManager.default.createDirectory(at: url, withIntermediateDirectories: true)
|
||||||
|
NSWorkspace.shared.open(url)
|
||||||
|
}
|
||||||
|
|
||||||
|
func progressLabel(for progress: LocalASRModelInstallProgress, language: AppUILanguage) -> String {
|
||||||
|
let phaseKey: String
|
||||||
|
switch progress.phase {
|
||||||
|
case .downloading: phaseKey = "mac.localASR.phase.downloading"
|
||||||
|
case .paused: phaseKey = "mac.localASR.phase.paused"
|
||||||
|
case .extracting: phaseKey = "mac.localASR.phase.extracting"
|
||||||
|
case .validating: phaseKey = "mac.localASR.phase.validating"
|
||||||
|
case .finalizing: phaseKey = "mac.localASR.phase.finalizing"
|
||||||
|
case .failed: phaseKey = "mac.localASR.phase.failed"
|
||||||
|
case .completed: phaseKey = "mac.localASR.phase.completed"
|
||||||
|
case .idle: return progress.message
|
||||||
|
}
|
||||||
|
let phase = MacL10n.string(phaseKey, language: language)
|
||||||
|
if let received = progress.bytesReceived, let total = progress.bytesTotal, total > 0 {
|
||||||
|
let recv = ByteCountFormatter.string(fromByteCount: received, countStyle: .file)
|
||||||
|
let tot = ByteCountFormatter.string(fromByteCount: total, countStyle: .file)
|
||||||
|
return "\(phase) · \(progress.message) (\(recv) / \(tot))"
|
||||||
|
}
|
||||||
|
return "\(phase) · \(progress.message)"
|
||||||
|
}
|
||||||
|
|
||||||
|
private func startProgressPolling() {
|
||||||
|
progressPollTask?.cancel()
|
||||||
|
progressPollTask = Task { [weak self] in
|
||||||
|
while !Task.isCancelled {
|
||||||
|
guard let self else { return }
|
||||||
|
let current = await manager.currentProgress()
|
||||||
|
await MainActor.run {
|
||||||
|
self.installProgress = current
|
||||||
|
self.isDownloadPaused = current.phase == .paused
|
||||||
|
}
|
||||||
|
try? await Task.sleep(for: .milliseconds(120))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private func stopProgressPolling() {
|
||||||
|
progressPollTask?.cancel()
|
||||||
|
progressPollTask = nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func formattedSize(_ bytes: Int) -> String {
|
||||||
|
ByteCountFormatter.string(fromByteCount: Int64(bytes), countStyle: .file)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
struct MacLocalASRModelSettingsView: View {
|
||||||
|
@ObservedObject var viewModel: MacDictationViewModel
|
||||||
|
@StateObject private var modelVM = MacLocalASRModelSettingsViewModel()
|
||||||
|
@Environment(\.themePalette) private var palette
|
||||||
|
|
||||||
|
private var lang: AppUILanguage { viewModel.config.uiLanguage }
|
||||||
|
|
||||||
|
var body: some View {
|
||||||
|
Group {
|
||||||
|
if let catalog = modelVM.catalog {
|
||||||
|
modelPickerSection(catalog: catalog)
|
||||||
|
runtimeSection(catalog: catalog)
|
||||||
|
} else {
|
||||||
|
Text(MacL10n.string("mac.localASR.catalogMissing", language: lang))
|
||||||
|
.foregroundStyle(palette.textSecondary)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
.onAppear {
|
||||||
|
modelVM.onLocalModelStateChanged = { viewModel.bumpLocalModelRevision() }
|
||||||
|
modelVM.reload()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private func modelPickerSection(catalog: LocalASRCatalogDocument) -> some View {
|
||||||
|
Section {
|
||||||
|
ForEach(catalog.models) { model in
|
||||||
|
modelRow(model)
|
||||||
|
}
|
||||||
|
|
||||||
|
if modelVM.isInstalling,
|
||||||
|
modelVM.installProgress.phase == .extracting
|
||||||
|
|| modelVM.installProgress.phase == .validating
|
||||||
|
|| modelVM.installProgress.phase == .finalizing {
|
||||||
|
ProgressView(value: modelVM.installProgress.fraction) {
|
||||||
|
Text(modelVM.progressLabel(for: modelVM.installProgress, language: lang))
|
||||||
|
.font(TypeStyle.caption)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if !modelVM.statusMessage.isEmpty {
|
||||||
|
Text(modelVM.statusMessage)
|
||||||
|
.font(TypeStyle.caption)
|
||||||
|
.foregroundStyle(palette.textSecondary)
|
||||||
|
}
|
||||||
|
|
||||||
|
Button(MacL10n.string("mac.localASR.openStorage", language: lang)) {
|
||||||
|
modelVM.revealStorageRoot()
|
||||||
|
}
|
||||||
|
} header: {
|
||||||
|
Text(MacL10n.string("mac.localASR.models", language: lang))
|
||||||
|
} footer: {
|
||||||
|
Text(MacL10n.string("mac.localASR.modelsDesc", language: lang))
|
||||||
|
.font(TypeStyle.caption)
|
||||||
|
.foregroundStyle(palette.textSecondary)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private func runtimeSection(catalog: LocalASRCatalogDocument) -> some View {
|
||||||
|
Group {
|
||||||
|
if let runtime = modelVM.currentRuntime(in: catalog) {
|
||||||
|
Section {
|
||||||
|
LabeledContent(runtime.displayName) {
|
||||||
|
Text(
|
||||||
|
modelVM.isRuntimeInstalled(runtime)
|
||||||
|
? MacL10n.string("mac.localASR.installed", language: lang)
|
||||||
|
: MacL10n.string("mac.localASR.notInstalled", language: lang)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
Text(MacL10n.string("mac.localASR.runtimeDesc", language: lang))
|
||||||
|
.font(TypeStyle.caption)
|
||||||
|
.foregroundStyle(palette.textSecondary)
|
||||||
|
} header: {
|
||||||
|
Text(MacL10n.string("mac.localASR.runtime", language: lang))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
@ViewBuilder
|
||||||
|
private func modelRow(_ model: LocalASRModelDefinition) -> some View {
|
||||||
|
let installed = modelVM.isInstalled(model)
|
||||||
|
let selected = modelVM.selectedModelId == model.id
|
||||||
|
let installing = modelVM.isInstallingModel(model)
|
||||||
|
|
||||||
|
VStack(alignment: .leading, spacing: Spacing.xs) {
|
||||||
|
HStack(alignment: .top) {
|
||||||
|
Button {
|
||||||
|
modelVM.selectModel(model.id)
|
||||||
|
} label: {
|
||||||
|
HStack(spacing: Spacing.sm) {
|
||||||
|
Image(systemName: selected ? "largecircle.fill.circle" : "circle")
|
||||||
|
.foregroundStyle(selected ? palette.accent : palette.textTertiary)
|
||||||
|
VStack(alignment: .leading, spacing: 2) {
|
||||||
|
Text(model.displayName)
|
||||||
|
.foregroundStyle(palette.textPrimary)
|
||||||
|
Text(modelSubtitle(model, installed: installed))
|
||||||
|
.font(TypeStyle.caption)
|
||||||
|
.foregroundStyle(palette.textSecondary)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
.buttonStyle(.plain)
|
||||||
|
.disabled(modelVM.isInstalling)
|
||||||
|
|
||||||
|
Spacer()
|
||||||
|
|
||||||
|
modelRowActions(model: model, installed: installed, installing: installing)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
.padding(.vertical, 2)
|
||||||
|
}
|
||||||
|
|
||||||
|
@ViewBuilder
|
||||||
|
private func modelRowActions(
|
||||||
|
model: LocalASRModelDefinition,
|
||||||
|
installed: Bool,
|
||||||
|
installing: Bool
|
||||||
|
) -> some View {
|
||||||
|
if installing {
|
||||||
|
HStack(spacing: Spacing.sm) {
|
||||||
|
circularInstallProgress(for: model)
|
||||||
|
if modelVM.installProgress.phase == .downloading
|
||||||
|
|| modelVM.installProgress.phase == .paused {
|
||||||
|
Button {
|
||||||
|
if modelVM.isDownloadPaused {
|
||||||
|
modelVM.resumeDownload()
|
||||||
|
} else {
|
||||||
|
modelVM.pauseDownload()
|
||||||
|
}
|
||||||
|
} label: {
|
||||||
|
Image(systemName: modelVM.isDownloadPaused ? "play.fill" : "pause.fill")
|
||||||
|
.font(.system(size: 12, weight: .semibold))
|
||||||
|
.frame(width: 28, height: 28)
|
||||||
|
}
|
||||||
|
.buttonStyle(.bordered)
|
||||||
|
.controlSize(.small)
|
||||||
|
.help(
|
||||||
|
modelVM.isDownloadPaused
|
||||||
|
? MacL10n.string("mac.localASR.resume", language: lang)
|
||||||
|
: MacL10n.string("mac.localASR.pause", language: lang)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else if model.installKind == .manual {
|
||||||
|
Button(MacL10n.string("mac.localASR.openFolder", language: lang)) {
|
||||||
|
modelVM.revealModelFolder(model)
|
||||||
|
}
|
||||||
|
.buttonStyle(.bordered)
|
||||||
|
.controlSize(.small)
|
||||||
|
} else if installed {
|
||||||
|
Button(MacL10n.string("mac.localASR.delete", language: lang), role: .destructive) {
|
||||||
|
modelVM.deleteModel(model)
|
||||||
|
}
|
||||||
|
.buttonStyle(.bordered)
|
||||||
|
.controlSize(.small)
|
||||||
|
} else {
|
||||||
|
Button(MacL10n.string("mac.localASR.download", language: lang)) {
|
||||||
|
modelVM.installModel(model)
|
||||||
|
}
|
||||||
|
.buttonStyle(.borderedProminent)
|
||||||
|
.controlSize(.small)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private func circularInstallProgress(for model: LocalASRModelDefinition) -> some View {
|
||||||
|
let fraction: Double = {
|
||||||
|
if modelVM.installProgress.phase == .downloading || modelVM.installProgress.phase == .paused,
|
||||||
|
let received = modelVM.installProgress.bytesReceived,
|
||||||
|
let total = modelVM.installProgress.bytesTotal,
|
||||||
|
total > 0 {
|
||||||
|
return min(1, max(0, Double(received) / Double(total)))
|
||||||
|
}
|
||||||
|
return modelVM.installProgress.fraction
|
||||||
|
}()
|
||||||
|
return ZStack {
|
||||||
|
Circle()
|
||||||
|
.stroke(palette.textTertiary.opacity(0.25), lineWidth: 3)
|
||||||
|
Circle()
|
||||||
|
.trim(from: 0, to: fraction)
|
||||||
|
.stroke(palette.accent, style: StrokeStyle(lineWidth: 3, lineCap: .round))
|
||||||
|
.rotationEffect(.degrees(-90))
|
||||||
|
.animation(.linear(duration: 0.15), value: fraction)
|
||||||
|
if modelVM.installProgress.phase == .paused {
|
||||||
|
Image(systemName: "pause.fill")
|
||||||
|
.font(.system(size: 10, weight: .bold))
|
||||||
|
.foregroundStyle(palette.textSecondary)
|
||||||
|
} else {
|
||||||
|
Text("\(Int(fraction * 100))%")
|
||||||
|
.font(.system(size: 9, weight: .medium, design: .rounded))
|
||||||
|
.foregroundStyle(palette.textSecondary)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
.frame(width: 36, height: 36)
|
||||||
|
.accessibilityLabel(modelVM.progressLabel(for: modelVM.installProgress, language: lang))
|
||||||
|
}
|
||||||
|
|
||||||
|
private func modelSubtitle(_ model: LocalASRModelDefinition, installed: Bool) -> String {
|
||||||
|
let size = modelVM.formattedSize(model.sizeBytes)
|
||||||
|
let hotword = model.supportsHotwords
|
||||||
|
? MacL10n.string("mac.localASR.hotwordsYes", language: lang)
|
||||||
|
: MacL10n.string("mac.localASR.hotwordsNo", language: lang)
|
||||||
|
let state = installed
|
||||||
|
? MacL10n.string("mac.localASR.installed", language: lang)
|
||||||
|
: MacL10n.string("mac.localASR.notInstalled", language: lang)
|
||||||
|
if let usage = modelVM.installedDiskUsage(model) {
|
||||||
|
return "\(size) · \(hotword) · \(state) · \(usage)"
|
||||||
|
}
|
||||||
|
return "\(size) · \(hotword) · \(state)"
|
||||||
|
}
|
||||||
|
|
||||||
|
private var diagnosticsSection: some View {
|
||||||
|
Section {
|
||||||
|
if let snapshot = modelVM.diagnosticsSnapshot {
|
||||||
|
LabeledContent(MacL10n.string("mac.localASR.diagBackend", language: lang)) {
|
||||||
|
Text(snapshot.backendLabel ?? "—")
|
||||||
|
}
|
||||||
|
LabeledContent(MacL10n.string("mac.localASR.diagUserTerms", language: lang)) {
|
||||||
|
Text("\(snapshot.diagnostics.userTermCount)")
|
||||||
|
}
|
||||||
|
LabeledContent(MacL10n.string("mac.localASR.diagBuiltinTerms", language: lang)) {
|
||||||
|
Text("\(snapshot.diagnostics.builtinTermCount)")
|
||||||
|
}
|
||||||
|
LabeledContent(MacL10n.string("mac.localASR.diagHotwords", language: lang)) {
|
||||||
|
Text("\(snapshot.hotwordCount)")
|
||||||
|
}
|
||||||
|
LabeledContent(MacL10n.string("mac.localASR.diagPrompt", language: lang)) {
|
||||||
|
Text("\(snapshot.promptBiasLength)")
|
||||||
|
}
|
||||||
|
if snapshot.diagnostics.truncated {
|
||||||
|
Label(
|
||||||
|
snapshot.diagnostics.truncationReason ?? MacL10n.string("mac.localASR.diagTruncated", language: lang),
|
||||||
|
systemImage: "exclamationmark.triangle"
|
||||||
|
)
|
||||||
|
.font(TypeStyle.caption)
|
||||||
|
.foregroundStyle(palette.warning)
|
||||||
|
}
|
||||||
|
Text(snapshot.diagnostics.selectedSources.joined(separator: ", "))
|
||||||
|
.font(TypeStyle.caption)
|
||||||
|
.foregroundStyle(palette.textSecondary)
|
||||||
|
} else {
|
||||||
|
Text(MacL10n.string("mac.localASR.diagEmpty", language: lang))
|
||||||
|
.foregroundStyle(palette.textSecondary)
|
||||||
|
}
|
||||||
|
} header: {
|
||||||
|
Text(MacL10n.string("mac.localASR.diagnostics", language: lang))
|
||||||
|
} footer: {
|
||||||
|
Text(MacL10n.string("mac.localASR.diagnosticsDesc", language: lang))
|
||||||
|
.font(TypeStyle.caption)
|
||||||
|
.foregroundStyle(palette.textSecondary)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,13 +1,15 @@
|
|||||||
// MacLocalASRService.swift
|
// MacLocalASRService.swift
|
||||||
// OSGKeyboard · Mac
|
// OSGKeyboard · Mac
|
||||||
//
|
//
|
||||||
// On-device ASR for macOS. Primary: Qwen3-ASR-1.7B (MLX via mlx-swift-asr).
|
// On-device ASR for macOS. Routes through the bundled local ASR catalog:
|
||||||
// Falls back to Apple Speech when Qwen3 weights are absent or backend is Apple Speech.
|
// Qwen3 MLX (default), Sherpa Qwen3 hotwords POC, SenseVoice, Apple Speech fallback.
|
||||||
|
|
||||||
import Foundation
|
import Foundation
|
||||||
|
|
||||||
enum MacLocalASRBackend: String, Sendable, CaseIterable {
|
enum MacLocalASRBackend: String, Sendable, CaseIterable {
|
||||||
case qwen3MLX
|
case qwen3MLX
|
||||||
|
case sherpaQwen3
|
||||||
|
case sherpaSenseVoice
|
||||||
case appleSpeech
|
case appleSpeech
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -39,28 +41,33 @@ enum MacLocalASRError: Error, LocalizedError {
|
|||||||
|
|
||||||
enum MacLocalASRPreferences {
|
enum MacLocalASRPreferences {
|
||||||
static let backendKey = "mac.localASR.backend"
|
static let backendKey = "mac.localASR.backend"
|
||||||
static let qwen3ModelPathKey = "mac.localASR.qwen3ModelPath"
|
static let selectedModelIdKey = LocalASRPreferenceKeys.selectedModelId
|
||||||
|
/// Shared managed subfolder for the manually-provided MLX weights.
|
||||||
|
static let qwen3ModelRelativePath = "models/qwen3-asr-1.7b-mlx"
|
||||||
|
|
||||||
static var backend: MacLocalASRBackend {
|
static var selectedModelId: String {
|
||||||
get {
|
get {
|
||||||
guard let raw = UserDefaults.standard.string(forKey: backendKey),
|
if let raw = UserDefaults.standard.string(forKey: selectedModelIdKey), !raw.isEmpty {
|
||||||
let value = MacLocalASRBackend(rawValue: raw) else {
|
return raw
|
||||||
return .qwen3MLX
|
|
||||||
}
|
}
|
||||||
return value
|
return legacyBackend == .appleSpeech ? "apple-speech-fallback" : "qwen3-mlx-1.7b"
|
||||||
}
|
}
|
||||||
set { UserDefaults.standard.set(newValue.rawValue, forKey: backendKey) }
|
set { UserDefaults.standard.set(newValue, forKey: selectedModelIdKey) }
|
||||||
}
|
}
|
||||||
|
|
||||||
|
static var legacyBackend: MacLocalASRBackend {
|
||||||
|
guard let raw = UserDefaults.standard.string(forKey: backendKey),
|
||||||
|
let value = MacLocalASRBackend(rawValue: raw) else {
|
||||||
|
return .qwen3MLX
|
||||||
|
}
|
||||||
|
return value
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Fixed location inside the shared managed model storage root. All three
|
||||||
|
/// catalog models live under the same directory, so MLX no longer needs a
|
||||||
|
/// per-model folder picker — the user drops converted weights here.
|
||||||
static var qwen3ModelPath: String {
|
static var qwen3ModelPath: String {
|
||||||
get { UserDefaults.standard.string(forKey: qwen3ModelPathKey) ?? defaultQwen3ModelPath }
|
LocalASRModelInstallState.installDirectory(for: qwen3ModelRelativePath).path
|
||||||
set { UserDefaults.standard.set(newValue, forKey: qwen3ModelPathKey) }
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Default install location for MLX-converted Qwen3-ASR weights.
|
|
||||||
static var defaultQwen3ModelPath: String {
|
|
||||||
let appSupport = FileManager.default.urls(for: .applicationSupportDirectory, in: .userDomainMask).first!
|
|
||||||
return appSupport.appendingPathComponent("OSGKeyboard/models/qwen3-asr-1.7b-mlx", isDirectory: true).path
|
|
||||||
}
|
}
|
||||||
|
|
||||||
static func qwen3ModelIsInstalled(at path: String = qwen3ModelPath) -> Bool {
|
static func qwen3ModelIsInstalled(at path: String = qwen3ModelPath) -> Bool {
|
||||||
@@ -80,24 +87,93 @@ enum MacLocalASRPreferences {
|
|||||||
}
|
}
|
||||||
|
|
||||||
enum MacLocalASRService {
|
enum MacLocalASRService {
|
||||||
/// Transcribe using the user's preferred local backend with automatic
|
|
||||||
/// fallback to Apple Speech when Qwen3 weights are not present.
|
static func loadCatalog() -> LocalASRCatalogDocument? {
|
||||||
static func transcribe(samples: [Float], locale: Locale) async throws -> String {
|
try? LocalASRModelCatalog.loadBundled()
|
||||||
let preferQwen3 = MacLocalASRPreferences.backend == .qwen3MLX
|
}
|
||||||
if preferQwen3, MacLocalASRPreferences.qwen3ModelIsInstalled() {
|
|
||||||
|
static func selectedModelDefinition() -> LocalASRModelDefinition? {
|
||||||
|
guard let catalog = loadCatalog() else { return nil }
|
||||||
|
let manifest = LocalASRInstalledManifestIO.load(defaultModelId: catalog.defaultModelId)
|
||||||
|
let selectedId = manifest.selectedModelId.isEmpty
|
||||||
|
? MacLocalASRPreferences.selectedModelId
|
||||||
|
: manifest.selectedModelId
|
||||||
|
if selectedId == "apple-speech-fallback" { return nil }
|
||||||
|
return LocalASRModelCatalog.model(selectedId, in: catalog)
|
||||||
|
?? LocalASRModelCatalog.model(catalog.defaultModelId, in: catalog)
|
||||||
|
}
|
||||||
|
|
||||||
|
static func currentCapabilities() -> LocalASRCapabilities {
|
||||||
|
guard let model = selectedModelDefinition() else { return .appleSpeech }
|
||||||
|
return LocalASRModelCatalog.capabilities(for: model)
|
||||||
|
}
|
||||||
|
|
||||||
|
static func currentBackendLabel() -> String {
|
||||||
|
guard let model = selectedModelDefinition() else { return "Apple Speech" }
|
||||||
|
return model.displayName
|
||||||
|
}
|
||||||
|
|
||||||
|
static func isModelInstalled(_ model: LocalASRModelDefinition) -> Bool {
|
||||||
|
LocalASRModelInstallState.isInstalled(
|
||||||
|
model,
|
||||||
|
manualMLXPath: MacLocalASRPreferences.qwen3ModelPath
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Transcribe using the selected catalog model, with MLX → Apple Speech fallback.
|
||||||
|
static func transcribe(
|
||||||
|
samples: [Float],
|
||||||
|
locale: Locale,
|
||||||
|
bias: LocalASRBiasPayload? = nil
|
||||||
|
) async throws -> String {
|
||||||
|
if let model = selectedModelDefinition(), isModelInstalled(model) {
|
||||||
do {
|
do {
|
||||||
return try await MacQwen3LocalASR.transcribe(
|
return try await transcribeWithModel(model, samples: samples, locale: locale, bias: bias)
|
||||||
samples: samples,
|
|
||||||
sampleRate: 16_000,
|
|
||||||
locale: locale,
|
|
||||||
modelPath: MacLocalASRPreferences.qwen3ModelPath
|
|
||||||
)
|
|
||||||
} catch MacLocalASRError.qwen3ModelMissing {
|
|
||||||
// Fall through to Apple Speech when weights are absent.
|
|
||||||
} catch {
|
} catch {
|
||||||
throw error
|
if model.backend != .mlx {
|
||||||
|
throw error
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if MacLocalASRPreferences.qwen3ModelIsInstalled() {
|
||||||
|
return try await MacQwen3LocalASR.transcribe(
|
||||||
|
samples: samples,
|
||||||
|
sampleRate: 16_000,
|
||||||
|
locale: locale,
|
||||||
|
modelPath: MacLocalASRPreferences.qwen3ModelPath,
|
||||||
|
bias: bias
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
return try await MacSpeechLocalASR.transcribe(samples: samples, locale: locale)
|
return try await MacSpeechLocalASR.transcribe(samples: samples, locale: locale)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private static func transcribeWithModel(
|
||||||
|
_ model: LocalASRModelDefinition,
|
||||||
|
samples: [Float],
|
||||||
|
locale: Locale,
|
||||||
|
bias: LocalASRBiasPayload?
|
||||||
|
) async throws -> String {
|
||||||
|
switch model.backend {
|
||||||
|
case .mlx:
|
||||||
|
return try await MacQwen3LocalASR.transcribe(
|
||||||
|
samples: samples,
|
||||||
|
sampleRate: 16_000,
|
||||||
|
locale: locale,
|
||||||
|
modelPath: MacLocalASRPreferences.qwen3ModelPath,
|
||||||
|
bias: bias
|
||||||
|
)
|
||||||
|
case .sherpaQwen3, .sherpaSenseVoice:
|
||||||
|
return try await MacSherpaLocalASR.transcribe(
|
||||||
|
samples: samples,
|
||||||
|
sampleRate: 16_000,
|
||||||
|
locale: locale,
|
||||||
|
model: model,
|
||||||
|
bias: bias
|
||||||
|
)
|
||||||
|
case .appleSpeech:
|
||||||
|
return try await MacSpeechLocalASR.transcribe(samples: samples, locale: locale)
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -50,14 +50,15 @@ actor MacQwen3ASREngine {
|
|||||||
func transcribe(
|
func transcribe(
|
||||||
samples: [Float],
|
samples: [Float],
|
||||||
language: String?,
|
language: String?,
|
||||||
modelPath: String
|
modelPath: String,
|
||||||
|
context: String? = nil
|
||||||
) async throws -> String {
|
) async throws -> String {
|
||||||
try await prepareIfNeeded(modelPath: modelPath)
|
try await prepareIfNeeded(modelPath: modelPath)
|
||||||
guard let stt else {
|
guard let stt else {
|
||||||
throw MacLocalASRError.qwen3LoadFailed("Engine not initialized")
|
throw MacLocalASRError.qwen3LoadFailed("Engine not initialized")
|
||||||
}
|
}
|
||||||
|
|
||||||
let result = try await stt.transcribe(audio: samples, language: language)
|
let result = try await stt.transcribe(audio: samples, language: language, context: context)
|
||||||
let text = result.text.trimmingCharacters(in: .whitespacesAndNewlines)
|
let text = result.text.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||||
guard !text.isEmpty else {
|
guard !text.isEmpty else {
|
||||||
throw MacLocalASRError.emptyTranscript
|
throw MacLocalASRError.emptyTranscript
|
||||||
|
|||||||
@@ -12,7 +12,8 @@ enum MacQwen3LocalASR {
|
|||||||
samples: [Float],
|
samples: [Float],
|
||||||
sampleRate: Int,
|
sampleRate: Int,
|
||||||
locale: Locale,
|
locale: Locale,
|
||||||
modelPath: String
|
modelPath: String,
|
||||||
|
bias: LocalASRBiasPayload? = nil
|
||||||
) async throws -> String {
|
) async throws -> String {
|
||||||
guard MacLocalASRPreferences.qwen3ModelIsInstalled(at: modelPath) else {
|
guard MacLocalASRPreferences.qwen3ModelIsInstalled(at: modelPath) else {
|
||||||
throw MacLocalASRError.qwen3ModelMissing
|
throw MacLocalASRError.qwen3ModelMissing
|
||||||
@@ -24,11 +25,14 @@ enum MacQwen3LocalASR {
|
|||||||
}
|
}
|
||||||
|
|
||||||
let language = MacQwen3LanguageHint.from(locale: locale)
|
let language = MacQwen3LanguageHint.from(locale: locale)
|
||||||
|
let context = bias?.promptBias?.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||||
|
let promptContext = (context?.isEmpty == false) ? context : nil
|
||||||
do {
|
do {
|
||||||
return try await MacQwen3ASREngine.shared.transcribe(
|
return try await MacQwen3ASREngine.shared.transcribe(
|
||||||
samples: samples,
|
samples: samples,
|
||||||
language: language,
|
language: language,
|
||||||
modelPath: modelPath
|
modelPath: modelPath,
|
||||||
|
context: promptContext
|
||||||
)
|
)
|
||||||
} catch let error as MacLocalASRError {
|
} catch let error as MacLocalASRError {
|
||||||
throw error
|
throw error
|
||||||
|
|||||||
@@ -56,7 +56,6 @@ struct MacRootView: View {
|
|||||||
Spacer()
|
Spacer()
|
||||||
devicesFooter
|
devicesFooter
|
||||||
}
|
}
|
||||||
.background(palette.surfaceMuted)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
private func sidebarRow(_ section: MacSection) -> some View {
|
private func sidebarRow(_ section: MacSection) -> some View {
|
||||||
|
|||||||
@@ -37,7 +37,7 @@ struct MacSettingsView: View {
|
|||||||
providerSection
|
providerSection
|
||||||
}
|
}
|
||||||
if viewModel.config.engineMode == "local" {
|
if viewModel.config.engineMode == "local" {
|
||||||
qwen3Section
|
MacLocalASRModelSettingsView(viewModel: viewModel)
|
||||||
}
|
}
|
||||||
inputSection
|
inputSection
|
||||||
syncSection
|
syncSection
|
||||||
@@ -147,12 +147,6 @@ struct MacSettingsView: View {
|
|||||||
systemImage: "cpu",
|
systemImage: "cpu",
|
||||||
selected: viewModel.config.engineMode == "local"
|
selected: viewModel.config.engineMode == "local"
|
||||||
) { viewModel.setEngineMode("local") }
|
) { viewModel.setEngineMode("local") }
|
||||||
|
|
||||||
if viewModel.config.engineMode == "local", !viewModel.qwen3ModelInstalled {
|
|
||||||
Label(MacL10n.string("mac.settings.qwen3Missing", language: lang), systemImage: "exclamationmark.triangle")
|
|
||||||
.font(TypeStyle.caption)
|
|
||||||
.foregroundStyle(palette.warning)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -196,25 +190,7 @@ struct MacSettingsView: View {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// MARK: - Qwen3 model path
|
// MARK: - Qwen3 model path (legacy — see MacLocalASRModelSettingsView)
|
||||||
|
|
||||||
private var qwen3Section: some View {
|
|
||||||
Section {
|
|
||||||
HStack(spacing: Spacing.sm) {
|
|
||||||
TextField("", text: qwen3PathBinding, prompt: Text(verbatim: "~/Models/Qwen3-ASR"))
|
|
||||||
.macFieldStyle()
|
|
||||||
Button(MacL10n.string("mac.settings.qwen3Browse", language: lang)) {
|
|
||||||
pickQwen3Folder()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
} header: {
|
|
||||||
Text(MacL10n.string("mac.settings.qwen3Model", language: lang))
|
|
||||||
} footer: {
|
|
||||||
Text(MacL10n.string("mac.settings.qwen3ModelDesc", language: lang))
|
|
||||||
.font(TypeStyle.caption)
|
|
||||||
.foregroundStyle(palette.textSecondary)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// MARK: - Row helpers
|
// MARK: - Row helpers
|
||||||
|
|
||||||
@@ -366,17 +342,6 @@ struct MacSettingsView: View {
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
private var qwen3PathBinding: Binding<String> {
|
|
||||||
Binding(
|
|
||||||
get: { MacLocalASRPreferences.qwen3ModelPath },
|
|
||||||
set: { newPath in
|
|
||||||
MacLocalASRPreferences.qwen3ModelPath = newPath
|
|
||||||
Task { await MacQwen3ASREngine.shared.unload() }
|
|
||||||
viewModel.warmUpQwen3IfNeeded()
|
|
||||||
}
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
// MARK: - AppKit actions (macOS only)
|
// MARK: - AppKit actions (macOS only)
|
||||||
|
|
||||||
private func openAccessibilitySettings() {
|
private func openAccessibilitySettings() {
|
||||||
@@ -413,19 +378,4 @@ struct MacSettingsView: View {
|
|||||||
private var accessibilityStatusNeeded: String {
|
private var accessibilityStatusNeeded: String {
|
||||||
lang.resolvedLanguageCode().hasPrefix("zh") ? "未授权" : "Needed"
|
lang.resolvedLanguageCode().hasPrefix("zh") ? "未授权" : "Needed"
|
||||||
}
|
}
|
||||||
|
|
||||||
private func pickQwen3Folder() {
|
|
||||||
#if os(macOS)
|
|
||||||
let panel = NSOpenPanel()
|
|
||||||
panel.canChooseDirectories = true
|
|
||||||
panel.canChooseFiles = false
|
|
||||||
panel.allowsMultipleSelection = false
|
|
||||||
panel.begin { response in
|
|
||||||
guard response == .OK, let url = panel.url else { return }
|
|
||||||
MacLocalASRPreferences.qwen3ModelPath = url.path
|
|
||||||
Task { await MacQwen3ASREngine.shared.unload() }
|
|
||||||
viewModel.warmUpQwen3IfNeeded()
|
|
||||||
}
|
|
||||||
#endif
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,56 @@
|
|||||||
|
// MacSherpaLocalASR.swift
|
||||||
|
// OSGKeyboard · Mac
|
||||||
|
//
|
||||||
|
// Sherpa-onnx backed local ASR (Qwen3 hotwords POC + SenseVoice baseline).
|
||||||
|
|
||||||
|
import Foundation
|
||||||
|
|
||||||
|
enum MacSherpaLocalASR {
|
||||||
|
|
||||||
|
static func transcribe(
|
||||||
|
samples: [Float],
|
||||||
|
sampleRate: Int,
|
||||||
|
locale: Locale,
|
||||||
|
model: LocalASRModelDefinition,
|
||||||
|
bias: LocalASRBiasPayload?
|
||||||
|
) async throws -> String {
|
||||||
|
let catalog = try LocalASRModelCatalog.loadBundled()
|
||||||
|
let manager = LocalASRModelManager.shared
|
||||||
|
guard let layout = model.layout,
|
||||||
|
let modelRoot = LocalASRModelInstallState.modelRootURL(model) else {
|
||||||
|
throw MacLocalASRError.qwen3ModelMissing
|
||||||
|
}
|
||||||
|
|
||||||
|
try await manager.ensureRuntimeInstalled(catalog: catalog)
|
||||||
|
guard let runtime = LocalASRModelCatalog.runtime(
|
||||||
|
for: LocalASRModelCatalog.currentRuntimePlatform(),
|
||||||
|
in: catalog
|
||||||
|
),
|
||||||
|
let binary = LocalASRModelInstallState.resolveRuntimeBinary(runtime: runtime) else {
|
||||||
|
throw MacLocalASRError.qwen3LoadFailed("Sherpa runtime binary missing")
|
||||||
|
}
|
||||||
|
|
||||||
|
switch model.backend {
|
||||||
|
case .sherpaQwen3:
|
||||||
|
return try await MacSherpaONNXRunner.transcribeQwen3(
|
||||||
|
samples: samples,
|
||||||
|
sampleRate: sampleRate,
|
||||||
|
locale: locale,
|
||||||
|
modelRoot: modelRoot,
|
||||||
|
layout: layout,
|
||||||
|
runtimeBinary: binary,
|
||||||
|
bias: bias
|
||||||
|
)
|
||||||
|
case .sherpaSenseVoice:
|
||||||
|
return try await MacSherpaONNXRunner.transcribeSenseVoice(
|
||||||
|
samples: samples,
|
||||||
|
sampleRate: sampleRate,
|
||||||
|
modelRoot: modelRoot,
|
||||||
|
layout: layout,
|
||||||
|
runtimeBinary: binary
|
||||||
|
)
|
||||||
|
default:
|
||||||
|
throw MacLocalASRError.qwen3InferenceFailed("Unsupported Sherpa backend")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,153 @@
|
|||||||
|
// MacSherpaONNXRunner.swift
|
||||||
|
// OSGKeyboard · Mac
|
||||||
|
//
|
||||||
|
// Invokes the downloaded `sherpa-onnx-offline` binary for Sherpa-backed POC models.
|
||||||
|
|
||||||
|
import Foundation
|
||||||
|
|
||||||
|
enum MacSherpaONNXRunner {
|
||||||
|
|
||||||
|
static func transcribeQwen3(
|
||||||
|
samples: [Float],
|
||||||
|
sampleRate: Int,
|
||||||
|
locale: Locale,
|
||||||
|
modelRoot: URL,
|
||||||
|
layout: LocalASRModelLayout,
|
||||||
|
runtimeBinary: URL,
|
||||||
|
bias: LocalASRBiasPayload?
|
||||||
|
) async throws -> String {
|
||||||
|
guard sampleRate == 16_000 else {
|
||||||
|
throw MacLocalASRError.qwen3InferenceFailed("Sherpa expects 16 kHz audio")
|
||||||
|
}
|
||||||
|
guard let conv = layout.convFrontend,
|
||||||
|
let encoder = layout.encoder,
|
||||||
|
let decoder = layout.decoder,
|
||||||
|
let tokenizer = layout.tokenizer else {
|
||||||
|
throw MacLocalASRError.qwen3InferenceFailed("Incomplete Sherpa Qwen3 layout")
|
||||||
|
}
|
||||||
|
|
||||||
|
let wavURL = try writeTemporaryWAV(samples: samples, sampleRate: sampleRate)
|
||||||
|
defer { try? FileManager.default.removeItem(at: wavURL) }
|
||||||
|
|
||||||
|
var arguments = [
|
||||||
|
"--qwen3-asr-conv-frontend=\(modelRoot.appendingPathComponent(conv).path)",
|
||||||
|
"--qwen3-asr-encoder=\(modelRoot.appendingPathComponent(encoder).path)",
|
||||||
|
"--qwen3-asr-decoder=\(modelRoot.appendingPathComponent(decoder).path)",
|
||||||
|
"--qwen3-asr-tokenizer=\(modelRoot.appendingPathComponent(tokenizer).path)",
|
||||||
|
"--qwen3-asr-max-new-tokens=512",
|
||||||
|
"--num-threads=2",
|
||||||
|
]
|
||||||
|
|
||||||
|
if let language = MacQwen3LanguageHint.from(locale: locale) {
|
||||||
|
arguments.append("--qwen3-asr-language=\(language)")
|
||||||
|
}
|
||||||
|
|
||||||
|
if let hotwords = bias?.hardHotwords, !hotwords.isEmpty {
|
||||||
|
arguments.append("--qwen3-asr-hotwords=\(hotwords.joined(separator: ","))")
|
||||||
|
}
|
||||||
|
|
||||||
|
arguments.append(wavURL.path)
|
||||||
|
return try await run(binary: runtimeBinary, arguments: arguments)
|
||||||
|
}
|
||||||
|
|
||||||
|
static func transcribeSenseVoice(
|
||||||
|
samples: [Float],
|
||||||
|
sampleRate: Int,
|
||||||
|
modelRoot: URL,
|
||||||
|
layout: LocalASRModelLayout,
|
||||||
|
runtimeBinary: URL
|
||||||
|
) async throws -> String {
|
||||||
|
guard sampleRate == 16_000 else {
|
||||||
|
throw MacLocalASRError.qwen3InferenceFailed("Sherpa expects 16 kHz audio")
|
||||||
|
}
|
||||||
|
guard let model = layout.senseVoiceModel,
|
||||||
|
let tokens = layout.tokens else {
|
||||||
|
throw MacLocalASRError.qwen3InferenceFailed("Incomplete SenseVoice layout")
|
||||||
|
}
|
||||||
|
|
||||||
|
let wavURL = try writeTemporaryWAV(samples: samples, sampleRate: sampleRate)
|
||||||
|
defer { try? FileManager.default.removeItem(at: wavURL) }
|
||||||
|
|
||||||
|
let arguments = [
|
||||||
|
"--tokens=\(modelRoot.appendingPathComponent(tokens).path)",
|
||||||
|
"--sense-voice-model=\(modelRoot.appendingPathComponent(model).path)",
|
||||||
|
"--num-threads=2",
|
||||||
|
wavURL.path,
|
||||||
|
]
|
||||||
|
return try await run(binary: runtimeBinary, arguments: arguments)
|
||||||
|
}
|
||||||
|
|
||||||
|
// MARK: - Private
|
||||||
|
|
||||||
|
private static func writeTemporaryWAV(samples: [Float], sampleRate: Int) throws -> URL {
|
||||||
|
let data = PCMSampleWavEncoder.encode(samples: samples, sampleRate: sampleRate)
|
||||||
|
let url = FileManager.default.temporaryDirectory
|
||||||
|
.appendingPathComponent("osg-sherpa-\(UUID().uuidString).wav")
|
||||||
|
try data.write(to: url, options: .atomic)
|
||||||
|
return url
|
||||||
|
}
|
||||||
|
|
||||||
|
private static func run(binary: URL, arguments: [String]) async throws -> String {
|
||||||
|
try await withCheckedThrowingContinuation { continuation in
|
||||||
|
let process = Process()
|
||||||
|
process.executableURL = binary
|
||||||
|
process.arguments = arguments
|
||||||
|
process.currentDirectoryURL = binary.deletingLastPathComponent()
|
||||||
|
|
||||||
|
let outputPipe = Pipe()
|
||||||
|
let errorPipe = Pipe()
|
||||||
|
process.standardOutput = outputPipe
|
||||||
|
process.standardError = errorPipe
|
||||||
|
|
||||||
|
process.terminationHandler = { proc in
|
||||||
|
let outputData = outputPipe.fileHandleForReading.readDataToEndOfFile()
|
||||||
|
let errorData = errorPipe.fileHandleForReading.readDataToEndOfFile()
|
||||||
|
let stdout = String(data: outputData, encoding: .utf8) ?? ""
|
||||||
|
let stderr = String(data: errorData, encoding: .utf8) ?? ""
|
||||||
|
|
||||||
|
guard proc.terminationStatus == 0 else {
|
||||||
|
let detail = stderr.isEmpty ? stdout : stderr
|
||||||
|
continuation.resume(
|
||||||
|
throwing: MacLocalASRError.qwen3InferenceFailed(
|
||||||
|
detail.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
let text = parseTranscript(stdout: stdout)
|
||||||
|
if text.isEmpty {
|
||||||
|
continuation.resume(throwing: MacLocalASRError.emptyTranscript)
|
||||||
|
} else {
|
||||||
|
continuation.resume(returning: text)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
do {
|
||||||
|
try process.run()
|
||||||
|
} catch {
|
||||||
|
continuation.resume(throwing: MacLocalASRError.qwen3InferenceFailed(error.localizedDescription))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private static func parseTranscript(stdout: String) -> String {
|
||||||
|
let lines = stdout
|
||||||
|
.split(whereSeparator: \.isNewline)
|
||||||
|
.map { String($0).trimmingCharacters(in: .whitespacesAndNewlines) }
|
||||||
|
.filter { !$0.isEmpty }
|
||||||
|
|
||||||
|
for line in lines.reversed() {
|
||||||
|
if line.hasPrefix("{"), let data = line.data(using: .utf8),
|
||||||
|
let object = try? JSONSerialization.jsonObject(with: data) as? [String: Any],
|
||||||
|
let text = object["text"] as? String {
|
||||||
|
let trimmed = text.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||||
|
if !trimmed.isEmpty { return trimmed }
|
||||||
|
}
|
||||||
|
if !line.hasPrefix("/"), !line.hasPrefix("--"), line.count > 1 {
|
||||||
|
return line
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -3,7 +3,7 @@
|
|||||||
<plist version="1.0">
|
<plist version="1.0">
|
||||||
<dict>
|
<dict>
|
||||||
<key>com.apple.security.app-sandbox</key>
|
<key>com.apple.security.app-sandbox</key>
|
||||||
<true/>
|
<false/>
|
||||||
<key>com.apple.security.device.audio-input</key>
|
<key>com.apple.security.device.audio-input</key>
|
||||||
<true/>
|
<true/>
|
||||||
<key>com.apple.security.network.client</key>
|
<key>com.apple.security.network.client</key>
|
||||||
|
|||||||
@@ -86,6 +86,20 @@ final class MacAppDelegate: NSObject, NSApplicationDelegate {
|
|||||||
MacAppearancePreference.applyToApp(.current)
|
MacAppearancePreference.applyToApp(.current)
|
||||||
configurePopover()
|
configurePopover()
|
||||||
configureStatusItem()
|
configureStatusItem()
|
||||||
|
|
||||||
|
// The menu bar always follows the *system* appearance, so the status
|
||||||
|
// item must ignore the app's forced light/dark override. Re-pin the
|
||||||
|
// button appearance whenever the system theme flips.
|
||||||
|
DistributedNotificationCenter.default.addObserver(
|
||||||
|
self,
|
||||||
|
selector: #selector(systemAppearanceDidChange),
|
||||||
|
name: NSNotification.Name("AppleInterfaceThemeChangedNotification"),
|
||||||
|
object: nil
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
deinit {
|
||||||
|
DistributedNotificationCenter.default.removeObserver(self)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Keep the app alive after the last window closes — it lives in the menu bar.
|
/// Keep the app alive after the last window closes — it lives in the menu bar.
|
||||||
@@ -96,18 +110,47 @@ final class MacAppDelegate: NSObject, NSApplicationDelegate {
|
|||||||
private func configureStatusItem() {
|
private func configureStatusItem() {
|
||||||
let item = NSStatusBar.system.statusItem(withLength: NSStatusItem.variableLength)
|
let item = NSStatusBar.system.statusItem(withLength: NSStatusItem.variableLength)
|
||||||
if let button = item.button {
|
if let button = item.button {
|
||||||
// Prefer the brand mark; fall back to an SF Symbol so the item is
|
button.image = Self.makeStatusBarImage()
|
||||||
// never invisible even if the asset fails to resolve.
|
|
||||||
let image = NSImage(named: "OSGBrandMark")
|
|
||||||
?? NSImage(systemSymbolName: "mic.circle.fill", accessibilityDescription: "OSGKeyboard")
|
|
||||||
image?.isTemplate = true
|
|
||||||
image?.size = NSSize(width: 18, height: 18)
|
|
||||||
button.image = image
|
|
||||||
button.image?.accessibilityDescription = "OSGKeyboard"
|
button.image?.accessibilityDescription = "OSGKeyboard"
|
||||||
button.action = #selector(togglePopover(_:))
|
button.action = #selector(togglePopover(_:))
|
||||||
button.target = self
|
button.target = self
|
||||||
}
|
}
|
||||||
statusItem = item
|
statusItem = item
|
||||||
|
applyStatusItemAppearance()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Builds the menu-bar glyph from the dedicated horizontal status mark.
|
||||||
|
/// Height is pinned to the tallest practical menu-bar slot so the logo reads
|
||||||
|
/// clearly; width follows the asset's aspect ratio.
|
||||||
|
private static func makeStatusBarImage() -> NSImage? {
|
||||||
|
guard let image = NSImage(named: "OSGStatusMark")
|
||||||
|
?? NSImage(named: "OSGBrandMark")
|
||||||
|
?? NSImage(systemSymbolName: "mic.circle.fill", accessibilityDescription: "OSGKeyboard") else {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
let height: CGFloat = 9
|
||||||
|
let aspect = max(image.size.width / max(image.size.height, 1), 1)
|
||||||
|
image.size = NSSize(width: height * aspect, height: height)
|
||||||
|
image.isTemplate = true
|
||||||
|
return image
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Pins the status-bar button to the current *system* appearance so its
|
||||||
|
/// template image tint matches the real menu-bar background — regardless of
|
||||||
|
/// the in-app light/dark preference forced on `NSApp.appearance`.
|
||||||
|
private func applyStatusItemAppearance() {
|
||||||
|
guard let button = statusItem?.button else { return }
|
||||||
|
let isDark = UserDefaults.standard.string(forKey: "AppleInterfaceStyle")?
|
||||||
|
.lowercased().contains("dark") ?? false
|
||||||
|
button.appearance = NSAppearance(named: isDark ? .darkAqua : .aqua)
|
||||||
|
}
|
||||||
|
|
||||||
|
@objc private func systemAppearanceDidChange() {
|
||||||
|
// The global-domain default lags the notification by a hair; hop to the
|
||||||
|
// next runloop tick so `AppleInterfaceStyle` reflects the new value.
|
||||||
|
DispatchQueue.main.async { [weak self] in
|
||||||
|
self?.applyStatusItemAppearance()
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
private func configurePopover() {
|
private func configurePopover() {
|
||||||
|
|||||||
@@ -0,0 +1,99 @@
|
|||||||
|
// LocalASRBiasPayload.swift
|
||||||
|
// OSGKeyboard · Shared
|
||||||
|
//
|
||||||
|
// Output of `LocalASRBiasAdapter` — vocabulary signals for each pipeline layer.
|
||||||
|
|
||||||
|
import Foundation
|
||||||
|
|
||||||
|
public struct LocalASRCorrectionPair: Sendable, Equatable {
|
||||||
|
public let alias: String
|
||||||
|
public let term: String
|
||||||
|
|
||||||
|
public init(alias: String, term: String) {
|
||||||
|
self.alias = alias
|
||||||
|
self.term = term
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public struct LocalASRBiasDiagnostics: Sendable, Equatable, Codable {
|
||||||
|
public var userTermCount: Int
|
||||||
|
public var builtinTermCount: Int
|
||||||
|
public var truncated: Bool
|
||||||
|
public var truncationReason: String?
|
||||||
|
public var selectedSources: [String]
|
||||||
|
|
||||||
|
public init(
|
||||||
|
userTermCount: Int = 0,
|
||||||
|
builtinTermCount: Int = 0,
|
||||||
|
truncated: Bool = false,
|
||||||
|
truncationReason: String? = nil,
|
||||||
|
selectedSources: [String] = []
|
||||||
|
) {
|
||||||
|
self.userTermCount = userTermCount
|
||||||
|
self.builtinTermCount = builtinTermCount
|
||||||
|
self.truncated = truncated
|
||||||
|
self.truncationReason = truncationReason
|
||||||
|
self.selectedSources = selectedSources
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public struct LocalASRBiasPayload: Sendable, Equatable {
|
||||||
|
public var hardHotwords: [String]
|
||||||
|
public var promptBias: String?
|
||||||
|
public var corpusContext: String?
|
||||||
|
public var polishFragment: String
|
||||||
|
public var correctionPairs: [LocalASRCorrectionPair]
|
||||||
|
public var diagnostics: LocalASRBiasDiagnostics
|
||||||
|
|
||||||
|
public static let empty = LocalASRBiasPayload(
|
||||||
|
hardHotwords: [],
|
||||||
|
promptBias: nil,
|
||||||
|
corpusContext: nil,
|
||||||
|
polishFragment: "",
|
||||||
|
correctionPairs: [],
|
||||||
|
diagnostics: LocalASRBiasDiagnostics()
|
||||||
|
)
|
||||||
|
|
||||||
|
public init(
|
||||||
|
hardHotwords: [String],
|
||||||
|
promptBias: String?,
|
||||||
|
corpusContext: String?,
|
||||||
|
polishFragment: String,
|
||||||
|
correctionPairs: [LocalASRCorrectionPair],
|
||||||
|
diagnostics: LocalASRBiasDiagnostics
|
||||||
|
) {
|
||||||
|
self.hardHotwords = hardHotwords
|
||||||
|
self.promptBias = promptBias
|
||||||
|
self.corpusContext = corpusContext
|
||||||
|
self.polishFragment = polishFragment
|
||||||
|
self.correctionPairs = correctionPairs
|
||||||
|
self.diagnostics = diagnostics
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public struct LocalASRBiasRequest: Sendable {
|
||||||
|
public var dictionary: PersonalDictionary
|
||||||
|
public var locale: Locale
|
||||||
|
public var frontAppBundleId: String?
|
||||||
|
public var capabilities: LocalASRCapabilities
|
||||||
|
/// Max builtin `phrases.tsv` terms considered for ASR bias (not polish-only).
|
||||||
|
public var builtinASRLimit: Int
|
||||||
|
/// Max builtin terms referenced in the polish supplement block.
|
||||||
|
public var builtinPolishLimit: Int
|
||||||
|
|
||||||
|
public init(
|
||||||
|
dictionary: PersonalDictionary,
|
||||||
|
locale: Locale,
|
||||||
|
frontAppBundleId: String? = nil,
|
||||||
|
capabilities: LocalASRCapabilities,
|
||||||
|
builtinASRLimit: Int = 300,
|
||||||
|
builtinPolishLimit: Int = 40
|
||||||
|
) {
|
||||||
|
self.dictionary = dictionary
|
||||||
|
self.locale = locale
|
||||||
|
self.frontAppBundleId = frontAppBundleId
|
||||||
|
self.capabilities = capabilities
|
||||||
|
self.builtinASRLimit = builtinASRLimit
|
||||||
|
self.builtinPolishLimit = builtinPolishLimit
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,82 @@
|
|||||||
|
// LocalASRCapabilities.swift
|
||||||
|
// OSGKeyboard · Shared
|
||||||
|
//
|
||||||
|
// Declares what each on-device ASR backend can accept for vocabulary bias.
|
||||||
|
// Callers must consult capabilities before building a `LocalASRBiasPayload`.
|
||||||
|
|
||||||
|
import Foundation
|
||||||
|
|
||||||
|
/// How a backend accepts vocabulary hints (honest matrix — not every model
|
||||||
|
/// supports hard hotwords).
|
||||||
|
public enum LocalASRHotwordMode: String, Sendable, Codable, Equatable {
|
||||||
|
case none
|
||||||
|
case promptOnly
|
||||||
|
case perRequest
|
||||||
|
case recognizerScoped
|
||||||
|
case cloudVocabulary
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Cost of refreshing hotwords on a backend (e.g. Sherpa Qwen3 reloads recognizer).
|
||||||
|
public enum LocalASRHotwordReloadCost: String, Sendable, Codable, Equatable {
|
||||||
|
case none
|
||||||
|
case recognizerReload
|
||||||
|
case modelReload
|
||||||
|
}
|
||||||
|
|
||||||
|
public struct LocalASRCapabilities: Sendable, Equatable {
|
||||||
|
public let hotwordMode: LocalASRHotwordMode
|
||||||
|
public let maxHotwordCount: Int
|
||||||
|
public let maxPromptCharacters: Int
|
||||||
|
public let supportsStreaming: Bool
|
||||||
|
public let hotwordReloadCost: LocalASRHotwordReloadCost
|
||||||
|
|
||||||
|
public init(
|
||||||
|
hotwordMode: LocalASRHotwordMode,
|
||||||
|
maxHotwordCount: Int,
|
||||||
|
maxPromptCharacters: Int,
|
||||||
|
supportsStreaming: Bool,
|
||||||
|
hotwordReloadCost: LocalASRHotwordReloadCost
|
||||||
|
) {
|
||||||
|
self.hotwordMode = hotwordMode
|
||||||
|
self.maxHotwordCount = maxHotwordCount
|
||||||
|
self.maxPromptCharacters = maxPromptCharacters
|
||||||
|
self.supportsStreaming = supportsStreaming
|
||||||
|
self.hotwordReloadCost = hotwordReloadCost
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Qwen3 MLX via mlx-swift-asr — `context` soft prompt on `transcribe`.
|
||||||
|
public static let qwen3MLX = LocalASRCapabilities(
|
||||||
|
hotwordMode: .promptOnly,
|
||||||
|
maxHotwordCount: 0,
|
||||||
|
maxPromptCharacters: 800,
|
||||||
|
supportsStreaming: false,
|
||||||
|
hotwordReloadCost: .none
|
||||||
|
)
|
||||||
|
|
||||||
|
/// Apple Speech on macOS — no project-controlled hotword API today.
|
||||||
|
public static let appleSpeech = LocalASRCapabilities(
|
||||||
|
hotwordMode: .none,
|
||||||
|
maxHotwordCount: 0,
|
||||||
|
maxPromptCharacters: 0,
|
||||||
|
supportsStreaming: false,
|
||||||
|
hotwordReloadCost: .none
|
||||||
|
)
|
||||||
|
|
||||||
|
/// Sherpa Qwen3 — hard hotwords via `--qwen3-asr-hotwords`.
|
||||||
|
public static let sherpaQwen3 = LocalASRCapabilities(
|
||||||
|
hotwordMode: .recognizerScoped,
|
||||||
|
maxHotwordCount: 100,
|
||||||
|
maxPromptCharacters: 0,
|
||||||
|
supportsStreaming: false,
|
||||||
|
hotwordReloadCost: .recognizerReload
|
||||||
|
)
|
||||||
|
|
||||||
|
/// Sherpa SenseVoice — fast Chinese baseline without hotwords.
|
||||||
|
public static let sherpaSenseVoice = LocalASRCapabilities(
|
||||||
|
hotwordMode: .none,
|
||||||
|
maxHotwordCount: 0,
|
||||||
|
maxPromptCharacters: 0,
|
||||||
|
supportsStreaming: false,
|
||||||
|
hotwordReloadCost: .none
|
||||||
|
)
|
||||||
|
}
|
||||||
@@ -0,0 +1,134 @@
|
|||||||
|
// LocalASRModelCatalog.swift
|
||||||
|
// OSGKeyboard · Shared
|
||||||
|
//
|
||||||
|
// Bundled catalog of downloadable / manual local ASR models and Sherpa runtimes.
|
||||||
|
|
||||||
|
import Foundation
|
||||||
|
|
||||||
|
public enum LocalASRModelBackend: String, Codable, Sendable, Equatable {
|
||||||
|
case mlx
|
||||||
|
case sherpaQwen3
|
||||||
|
case sherpaSenseVoice
|
||||||
|
case appleSpeech
|
||||||
|
}
|
||||||
|
|
||||||
|
public enum LocalASRInstallKind: String, Codable, Sendable, Equatable {
|
||||||
|
case manual
|
||||||
|
case archive
|
||||||
|
case runtime
|
||||||
|
}
|
||||||
|
|
||||||
|
public struct LocalASRDownloadSource: Codable, Sendable, Equatable {
|
||||||
|
public let type: String
|
||||||
|
public let priority: Int
|
||||||
|
public let url: String
|
||||||
|
}
|
||||||
|
|
||||||
|
public struct LocalASRModelLayout: Codable, Sendable, Equatable {
|
||||||
|
public var convFrontend: String?
|
||||||
|
public var encoder: String?
|
||||||
|
public var decoder: String?
|
||||||
|
public var tokenizer: String?
|
||||||
|
public var senseVoiceModel: String?
|
||||||
|
public var tokens: String?
|
||||||
|
}
|
||||||
|
|
||||||
|
public struct LocalASRRuntimeDefinition: Codable, Sendable, Equatable, Identifiable {
|
||||||
|
public let id: String
|
||||||
|
public let displayName: String
|
||||||
|
public let installRelativePath: String
|
||||||
|
public let binaryCandidates: [String]
|
||||||
|
public let archiveFileName: String
|
||||||
|
public let sizeBytes: Int
|
||||||
|
public let platform: String
|
||||||
|
public let sources: [LocalASRDownloadSource]
|
||||||
|
}
|
||||||
|
|
||||||
|
public struct LocalASRModelDefinition: Codable, Sendable, Equatable, Identifiable {
|
||||||
|
public let id: String
|
||||||
|
public let displayName: String
|
||||||
|
public let backend: LocalASRModelBackend
|
||||||
|
public let sizeBytes: Int
|
||||||
|
public let recommendedLocales: [String]
|
||||||
|
public let supportsHotwords: Bool
|
||||||
|
public let hotwordMode: LocalASRHotwordMode
|
||||||
|
public let installKind: LocalASRInstallKind
|
||||||
|
public let installRelativePath: String?
|
||||||
|
public let archiveBaseName: String?
|
||||||
|
public let layout: LocalASRModelLayout?
|
||||||
|
public let requiredRelativeFiles: [String]?
|
||||||
|
public let runtimePlatform: String?
|
||||||
|
public let sources: [LocalASRDownloadSource]?
|
||||||
|
}
|
||||||
|
|
||||||
|
public struct LocalASRCatalogDocument: Codable, Sendable, Equatable {
|
||||||
|
public let schemaVersion: Int
|
||||||
|
public let defaultModelId: String
|
||||||
|
public let runtimes: [LocalASRRuntimeDefinition]
|
||||||
|
public let models: [LocalASRModelDefinition]
|
||||||
|
}
|
||||||
|
|
||||||
|
public enum LocalASRModelCatalog {
|
||||||
|
|
||||||
|
public static func loadBundled() throws -> LocalASRCatalogDocument {
|
||||||
|
let bundle = Bundle(for: LocalASRCatalogBundleToken.self)
|
||||||
|
guard let url = bundle.url(forResource: "local-asr-catalog", withExtension: "json") else {
|
||||||
|
throw LocalASRModelCatalogError.missingBundledCatalog
|
||||||
|
}
|
||||||
|
let data = try Data(contentsOf: url)
|
||||||
|
return try JSONDecoder().decode(LocalASRCatalogDocument.self, from: data)
|
||||||
|
}
|
||||||
|
|
||||||
|
public static func model(_ id: String, in catalog: LocalASRCatalogDocument) -> LocalASRModelDefinition? {
|
||||||
|
catalog.models.first { $0.id == id }
|
||||||
|
}
|
||||||
|
|
||||||
|
public static func capabilities(for model: LocalASRModelDefinition) -> LocalASRCapabilities {
|
||||||
|
switch model.backend {
|
||||||
|
case .mlx:
|
||||||
|
return .qwen3MLX
|
||||||
|
case .sherpaQwen3:
|
||||||
|
return .sherpaQwen3
|
||||||
|
case .sherpaSenseVoice:
|
||||||
|
return .sherpaSenseVoice
|
||||||
|
case .appleSpeech:
|
||||||
|
return .appleSpeech
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#if os(macOS)
|
||||||
|
public static func runtime(for platform: String, in catalog: LocalASRCatalogDocument) -> LocalASRRuntimeDefinition? {
|
||||||
|
if platform == "macos-arm64" {
|
||||||
|
return catalog.runtimes.first { $0.platform == "macos-arm64" }
|
||||||
|
}
|
||||||
|
if platform == "macos-x64" {
|
||||||
|
return catalog.runtimes.first { $0.platform == "macos-x64" }
|
||||||
|
}
|
||||||
|
return catalog.runtimes.first
|
||||||
|
}
|
||||||
|
|
||||||
|
public static func currentRuntimePlatform() -> String {
|
||||||
|
#if arch(arm64)
|
||||||
|
return "macos-arm64"
|
||||||
|
#else
|
||||||
|
return "macos-x64"
|
||||||
|
#endif
|
||||||
|
}
|
||||||
|
#endif
|
||||||
|
}
|
||||||
|
|
||||||
|
public enum LocalASRModelCatalogError: Error, LocalizedError {
|
||||||
|
case missingBundledCatalog
|
||||||
|
case modelNotFound(String)
|
||||||
|
|
||||||
|
public var errorDescription: String? {
|
||||||
|
switch self {
|
||||||
|
case .missingBundledCatalog:
|
||||||
|
return "Missing bundled local ASR catalog."
|
||||||
|
case .modelNotFound(let id):
|
||||||
|
return "Local ASR model not found: \(id)"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private final class LocalASRCatalogBundleToken {}
|
||||||
@@ -111,4 +111,29 @@ extension PersonalDictionary {
|
|||||||
if hasNonASCII { return "zh" }
|
if hasNonASCII { return "zh" }
|
||||||
return "en"
|
return "en"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Alias → canonical term pairs for deterministic post-ASR correction.
|
||||||
|
/// Sorted longest-alias-first by the caller (`LocalASRTranscriptCorrector`).
|
||||||
|
public func localCorrectionPairs() -> [LocalASRCorrectionPair] {
|
||||||
|
var seen = Set<String>()
|
||||||
|
var pairs: [LocalASRCorrectionPair] = []
|
||||||
|
for entry in effectiveEntries {
|
||||||
|
let term = entry.term.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||||
|
guard !term.isEmpty else { continue }
|
||||||
|
for alias in entry.aliases {
|
||||||
|
let trimmed = alias.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||||
|
guard !trimmed.isEmpty else { continue }
|
||||||
|
guard trimmed.caseInsensitiveCompare(term) != .orderedSame else { continue }
|
||||||
|
let key = "\(trimmed.lowercased())|\(term.lowercased())"
|
||||||
|
guard seen.insert(key).inserted else { continue }
|
||||||
|
pairs.append(LocalASRCorrectionPair(alias: trimmed, term: term))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return pairs.sorted { lhs, rhs in
|
||||||
|
if lhs.alias.count != rhs.alias.count {
|
||||||
|
return lhs.alias.count > rhs.alias.count
|
||||||
|
}
|
||||||
|
return lhs.alias.localizedCaseInsensitiveCompare(rhs.alias) == .orderedAscending
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -24,6 +24,10 @@ public struct PolishContext: Sendable {
|
|||||||
/// bias terminology choices.
|
/// bias terminology choices.
|
||||||
public let precedingText: String?
|
public let precedingText: String?
|
||||||
|
|
||||||
|
/// Extra dictionary block appended after `PersonalDictionary.promptFragment()`
|
||||||
|
/// (e.g. builtin `phrases.tsv` terms on macOS local ASR).
|
||||||
|
public let dictionarySupplement: String?
|
||||||
|
|
||||||
/// Cap on how many characters of `precedingText` we actually
|
/// Cap on how many characters of `precedingText` we actually
|
||||||
/// include in the prompt. The full preceding text is often
|
/// include in the prompt. The full preceding text is often
|
||||||
/// hundreds of KB in a long note — we only need the tail.
|
/// hundreds of KB in a long note — we only need the tail.
|
||||||
@@ -33,11 +37,13 @@ public struct PolishContext: Sendable {
|
|||||||
appContext: AppContext = .unknown,
|
appContext: AppContext = .unknown,
|
||||||
intensity: PolishIntensity = .default,
|
intensity: PolishIntensity = .default,
|
||||||
precedingText: String? = nil,
|
precedingText: String? = nil,
|
||||||
|
dictionarySupplement: String? = nil,
|
||||||
maxPrecedingChars: Int = 500
|
maxPrecedingChars: Int = 500
|
||||||
) {
|
) {
|
||||||
self.appContext = appContext
|
self.appContext = appContext
|
||||||
self.intensity = intensity
|
self.intensity = intensity
|
||||||
self.precedingText = precedingText
|
self.precedingText = precedingText
|
||||||
|
self.dictionarySupplement = dictionarySupplement
|
||||||
self.maxPrecedingChars = maxPrecedingChars
|
self.maxPrecedingChars = maxPrecedingChars
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -74,7 +74,7 @@ public final class ProviderConfig: ObservableObject, @unchecked Sendable {
|
|||||||
guard !isApplyingConfiguration, engineMode != configuration.engineMode else { return }
|
guard !isApplyingConfiguration, engineMode != configuration.engineMode else { return }
|
||||||
configuration.engineMode = engineMode
|
configuration.engineMode = engineMode
|
||||||
applyEngineModeSideEffects()
|
applyEngineModeSideEffects()
|
||||||
persistConfiguration()
|
persistConfiguration(postConfigChanged: true)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@Published public var hasCompletedOnboarding: Bool {
|
@Published public var hasCompletedOnboarding: Bool {
|
||||||
|
|||||||
@@ -0,0 +1,102 @@
|
|||||||
|
{
|
||||||
|
"schemaVersion": 1,
|
||||||
|
"defaultModelId": "qwen3-mlx-1.7b",
|
||||||
|
"runtimes": [
|
||||||
|
{
|
||||||
|
"id": "sherpa-onnx-1.13.4-macos-arm64",
|
||||||
|
"displayName": "sherpa-onnx 1.13.4 (Apple Silicon)",
|
||||||
|
"installRelativePath": "runtimes/sherpa-onnx-1.13.4-macos-arm64",
|
||||||
|
"binaryCandidates": ["bin/sherpa-onnx-offline", "sherpa-onnx-offline"],
|
||||||
|
"archiveFileName": "sherpa-onnx-v1.13.4-osx-arm64-static-no-tts.tar.bz2",
|
||||||
|
"sizeBytes": 120000000,
|
||||||
|
"platform": "macos-arm64",
|
||||||
|
"sources": [
|
||||||
|
{
|
||||||
|
"type": "github",
|
||||||
|
"priority": 1,
|
||||||
|
"url": "https://github.com/k2-fsa/sherpa-onnx/releases/download/v1.13.4/sherpa-onnx-v1.13.4-osx-arm64-static-no-tts.tar.bz2"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "sherpa-onnx-1.13.4-macos-x64",
|
||||||
|
"displayName": "sherpa-onnx 1.13.4 (Intel)",
|
||||||
|
"installRelativePath": "runtimes/sherpa-onnx-1.13.4-macos-x64",
|
||||||
|
"binaryCandidates": ["bin/sherpa-onnx-offline", "sherpa-onnx-offline"],
|
||||||
|
"archiveFileName": "sherpa-onnx-v1.13.4-osx-x64-static-no-tts.tar.bz2",
|
||||||
|
"sizeBytes": 130000000,
|
||||||
|
"platform": "macos-x64",
|
||||||
|
"sources": [
|
||||||
|
{
|
||||||
|
"type": "github",
|
||||||
|
"priority": 1,
|
||||||
|
"url": "https://github.com/k2-fsa/sherpa-onnx/releases/download/v1.13.4/sherpa-onnx-v1.13.4-osx-x64-static-no-tts.tar.bz2"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"models": [
|
||||||
|
{
|
||||||
|
"id": "qwen3-mlx-1.7b",
|
||||||
|
"displayName": "Qwen3-ASR 1.7B (MLX)",
|
||||||
|
"backend": "mlx",
|
||||||
|
"sizeBytes": 1400000000,
|
||||||
|
"recommendedLocales": ["zh-CN", "en-US"],
|
||||||
|
"supportsHotwords": true,
|
||||||
|
"hotwordMode": "promptOnly",
|
||||||
|
"installKind": "manual",
|
||||||
|
"installRelativePath": "models/qwen3-asr-1.7b-mlx",
|
||||||
|
"requiredRelativeFiles": ["config.json", "model.safetensors", "vocab.json", "merges.txt"]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "sherpa-qwen3-0.6b-int8",
|
||||||
|
"displayName": "Qwen3-ASR 0.6B (Sherpa · hotwords)",
|
||||||
|
"backend": "sherpaQwen3",
|
||||||
|
"runtimePlatform": "macos",
|
||||||
|
"sizeBytes": 650000000,
|
||||||
|
"recommendedLocales": ["zh-CN", "en-US"],
|
||||||
|
"supportsHotwords": true,
|
||||||
|
"hotwordMode": "recognizerScoped",
|
||||||
|
"installKind": "archive",
|
||||||
|
"installRelativePath": "models/sherpa-qwen3-0.6b-int8",
|
||||||
|
"archiveBaseName": "sherpa-onnx-qwen3-asr-0.6B-int8-2026-03-25",
|
||||||
|
"layout": {
|
||||||
|
"convFrontend": "conv_frontend.onnx",
|
||||||
|
"encoder": "encoder.int8.onnx",
|
||||||
|
"decoder": "decoder.int8.onnx",
|
||||||
|
"tokenizer": "tokenizer"
|
||||||
|
},
|
||||||
|
"sources": [
|
||||||
|
{
|
||||||
|
"type": "github",
|
||||||
|
"priority": 1,
|
||||||
|
"url": "https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-qwen3-asr-0.6B-int8-2026-03-25.tar.bz2"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "sherpa-sensevoice-small-int8",
|
||||||
|
"displayName": "SenseVoice Small (Sherpa)",
|
||||||
|
"backend": "sherpaSenseVoice",
|
||||||
|
"runtimePlatform": "macos",
|
||||||
|
"sizeBytes": 250000000,
|
||||||
|
"recommendedLocales": ["zh-CN", "en-US", "ja-JP", "ko-KR"],
|
||||||
|
"supportsHotwords": false,
|
||||||
|
"hotwordMode": "none",
|
||||||
|
"installKind": "archive",
|
||||||
|
"installRelativePath": "models/sherpa-sensevoice-small-int8",
|
||||||
|
"archiveBaseName": "sherpa-onnx-sense-voice-zh-en-ja-ko-yue-int8-2024-07-17",
|
||||||
|
"layout": {
|
||||||
|
"senseVoiceModel": "model.int8.onnx",
|
||||||
|
"tokens": "tokens.txt"
|
||||||
|
},
|
||||||
|
"sources": [
|
||||||
|
{
|
||||||
|
"type": "github",
|
||||||
|
"priority": 1,
|
||||||
|
"url": "https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-sense-voice-zh-en-ja-ko-yue-int8-2024-07-17.tar.bz2"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
@@ -92,6 +92,7 @@ public struct AppGroupStore: @unchecked Sendable {
|
|||||||
config.model = openAI.defaultModel
|
config.model = openAI.defaultModel
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
AppGroupConfigDarwin.postConfigChanged()
|
||||||
}
|
}
|
||||||
|
|
||||||
public func setUILanguage(_ language: AppUILanguage) {
|
public func setUILanguage(_ language: AppUILanguage) {
|
||||||
|
|||||||
@@ -0,0 +1,152 @@
|
|||||||
|
// BuiltinLexiconIndex.swift
|
||||||
|
// OSGKeyboard · Shared
|
||||||
|
//
|
||||||
|
// In-memory index over bundled `phrases.tsv` (~10k computer terms).
|
||||||
|
// macOS local ASR consumes a Top-N subset; the full index also backs
|
||||||
|
// polish supplements and future retrieval.
|
||||||
|
|
||||||
|
import Foundation
|
||||||
|
|
||||||
|
public final class BuiltinLexiconIndex: @unchecked Sendable {
|
||||||
|
|
||||||
|
public struct Term: Sendable, Equatable {
|
||||||
|
public let word: String
|
||||||
|
public let pinyin: String
|
||||||
|
public let source: String
|
||||||
|
public let weight: Int
|
||||||
|
}
|
||||||
|
|
||||||
|
public static let shared = BuiltinLexiconIndex()
|
||||||
|
|
||||||
|
private let lock = NSLock()
|
||||||
|
private var cachedTerms: [Term]?
|
||||||
|
private let injectedURL: URL?
|
||||||
|
|
||||||
|
/// Production singleton loads from the app bundle.
|
||||||
|
private init() {
|
||||||
|
injectedURL = nil
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Test / preview hook with an explicit TSV file or inline fixture.
|
||||||
|
init(fixtureURL: URL) {
|
||||||
|
injectedURL = fixtureURL
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Parse TSV content without touching the bundle (unit tests).
|
||||||
|
public static func parseTSV(_ content: String) -> [Term] {
|
||||||
|
var terms: [Term] = []
|
||||||
|
terms.reserveCapacity(256)
|
||||||
|
|
||||||
|
for (lineIndex, line) in content.split(whereSeparator: \.isNewline).enumerated() {
|
||||||
|
if lineIndex == 0, line.hasPrefix("word\t") { continue }
|
||||||
|
let columns = line.split(separator: "\t", omittingEmptySubsequences: false)
|
||||||
|
guard columns.count >= 4 else { continue }
|
||||||
|
let word = String(columns[0]).trimmingCharacters(in: .whitespacesAndNewlines)
|
||||||
|
guard !word.isEmpty else { continue }
|
||||||
|
let pinyin = String(columns[1])
|
||||||
|
let source = String(columns[2])
|
||||||
|
let weight = Int(columns[3]) ?? 1
|
||||||
|
terms.append(Term(word: word, pinyin: pinyin, source: source, weight: weight))
|
||||||
|
}
|
||||||
|
return terms
|
||||||
|
}
|
||||||
|
|
||||||
|
public func termCount() -> Int {
|
||||||
|
lock.lock()
|
||||||
|
defer { lock.unlock() }
|
||||||
|
return loadTermsLocked().count
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns canonical words ranked for ASR bias injection.
|
||||||
|
public func topTerms(
|
||||||
|
limit: Int,
|
||||||
|
minimumWeight: Int = 4,
|
||||||
|
preferredSources: Set<String>? = nil
|
||||||
|
) -> [String] {
|
||||||
|
guard limit > 0 else { return [] }
|
||||||
|
|
||||||
|
lock.lock()
|
||||||
|
let all = loadTermsLocked()
|
||||||
|
lock.unlock()
|
||||||
|
|
||||||
|
let filtered = all.filter { term in
|
||||||
|
guard term.weight >= minimumWeight else { return false }
|
||||||
|
if let preferredSources, !preferredSources.isEmpty {
|
||||||
|
return preferredSources.contains(term.source)
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
let ranked = filtered.sorted { lhs, rhs in
|
||||||
|
let leftScore = Self.rankingScore(lhs)
|
||||||
|
let rightScore = Self.rankingScore(rhs)
|
||||||
|
if leftScore != rightScore { return leftScore > rightScore }
|
||||||
|
return lhs.word.localizedCaseInsensitiveCompare(rhs.word) == .orderedAscending
|
||||||
|
}
|
||||||
|
|
||||||
|
var seen = Set<String>()
|
||||||
|
var words: [String] = []
|
||||||
|
words.reserveCapacity(min(limit, ranked.count))
|
||||||
|
for term in ranked {
|
||||||
|
let key = term.word.lowercased()
|
||||||
|
guard seen.insert(key).inserted else { continue }
|
||||||
|
words.append(term.word)
|
||||||
|
if words.count >= limit { break }
|
||||||
|
}
|
||||||
|
return words
|
||||||
|
}
|
||||||
|
|
||||||
|
// MARK: - Private
|
||||||
|
|
||||||
|
private func loadTermsLocked() -> [Term] {
|
||||||
|
if let cachedTerms { return cachedTerms }
|
||||||
|
let loaded: [Term]
|
||||||
|
if let injectedURL {
|
||||||
|
loaded = Self.load(from: injectedURL)
|
||||||
|
} else if let url = Self.locateBundledPhrasesURL() {
|
||||||
|
loaded = Self.load(from: url)
|
||||||
|
} else {
|
||||||
|
loaded = []
|
||||||
|
}
|
||||||
|
cachedTerms = loaded
|
||||||
|
return loaded
|
||||||
|
}
|
||||||
|
|
||||||
|
private static func load(from url: URL) -> [Term] {
|
||||||
|
guard let data = try? Data(contentsOf: url),
|
||||||
|
let content = String(data: data, encoding: .utf8) else {
|
||||||
|
return []
|
||||||
|
}
|
||||||
|
return parseTSV(content)
|
||||||
|
}
|
||||||
|
|
||||||
|
private static func locateBundledPhrasesURL() -> URL? {
|
||||||
|
let candidates: [Bundle] = [Bundle.main, Bundle(for: BuiltinLexiconIndex.self)]
|
||||||
|
for bundle in candidates {
|
||||||
|
if let url = bundle.url(
|
||||||
|
forResource: "phrases",
|
||||||
|
withExtension: "tsv",
|
||||||
|
subdirectory: "CustomLanguageModel/v1"
|
||||||
|
) {
|
||||||
|
return url
|
||||||
|
}
|
||||||
|
if let url = bundle.url(forResource: "phrases", withExtension: "tsv") {
|
||||||
|
return url
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
private static func rankingScore(_ term: Term) -> Int {
|
||||||
|
var score = term.weight * 100
|
||||||
|
if containsLatinLetters(term.word) { score += 50 }
|
||||||
|
if term.word.count <= 8 { score += 10 }
|
||||||
|
return score
|
||||||
|
}
|
||||||
|
|
||||||
|
private static func containsLatinLetters(_ text: String) -> Bool {
|
||||||
|
text.unicodeScalars.contains { scalar in
|
||||||
|
scalar.isASCII && CharacterSet.letters.contains(scalar)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -384,19 +384,26 @@ public final class FlowContinuousCapture {
|
|||||||
|
|
||||||
/// Re-activate capture after returning from background without
|
/// Re-activate capture after returning from background without
|
||||||
/// reinstalling the tap (iOS may deactivate the audio session).
|
/// reinstalling the tap (iOS may deactivate the audio session).
|
||||||
public func reassertIfRunning() {
|
@discardableResult
|
||||||
guard isRunning else { return }
|
public func reassertIfRunning() -> Bool {
|
||||||
|
guard isRunning else { return false }
|
||||||
let session = AVAudioSession.sharedInstance()
|
let session = AVAudioSession.sharedInstance()
|
||||||
try? session.setCategory(
|
do {
|
||||||
.playAndRecord,
|
try session.setCategory(
|
||||||
mode: .measurement,
|
.playAndRecord,
|
||||||
options: [.defaultToSpeaker, .allowBluetoothHFP, .mixWithOthers]
|
mode: .measurement,
|
||||||
)
|
options: [.defaultToSpeaker, .allowBluetoothHFP, .mixWithOthers]
|
||||||
try? session.setActive(true, options: .notifyOthersOnDeactivation)
|
)
|
||||||
if !audioEngine.isRunning {
|
try session.setActive(true, options: .notifyOthersOnDeactivation)
|
||||||
try? audioEngine.start()
|
if !audioEngine.isRunning {
|
||||||
|
try audioEngine.start()
|
||||||
|
}
|
||||||
|
notifyEngineLiveChanged()
|
||||||
|
return engineIsLive
|
||||||
|
} catch {
|
||||||
|
notifyEngineLiveChanged()
|
||||||
|
return false
|
||||||
}
|
}
|
||||||
notifyEngineLiveChanged()
|
|
||||||
}
|
}
|
||||||
|
|
||||||
public func awaitAudioFlowing(
|
public func awaitAudioFlowing(
|
||||||
|
|||||||
@@ -6,6 +6,158 @@
|
|||||||
|
|
||||||
import Foundation
|
import Foundation
|
||||||
|
|
||||||
|
public struct FlowCommand: Codable, Equatable, Sendable {
|
||||||
|
public enum Action: String, Codable, Sendable {
|
||||||
|
case startRecording
|
||||||
|
case stopRecording
|
||||||
|
case abort
|
||||||
|
}
|
||||||
|
|
||||||
|
public let protocolVersion: Int
|
||||||
|
public let sessionId: UUID
|
||||||
|
public let utteranceId: UUID
|
||||||
|
public let commandSeq: Int64
|
||||||
|
public let action: Action
|
||||||
|
public let localeId: String
|
||||||
|
public let createdAt: TimeInterval
|
||||||
|
|
||||||
|
public init(
|
||||||
|
protocolVersion: Int = 1,
|
||||||
|
sessionId: UUID,
|
||||||
|
utteranceId: UUID,
|
||||||
|
commandSeq: Int64,
|
||||||
|
action: Action,
|
||||||
|
localeId: String,
|
||||||
|
createdAt: TimeInterval = Date().timeIntervalSince1970
|
||||||
|
) {
|
||||||
|
self.protocolVersion = protocolVersion
|
||||||
|
self.sessionId = sessionId
|
||||||
|
self.utteranceId = utteranceId
|
||||||
|
self.commandSeq = commandSeq
|
||||||
|
self.action = action
|
||||||
|
self.localeId = localeId
|
||||||
|
self.createdAt = createdAt
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public struct FlowResult: Codable, Equatable, Sendable {
|
||||||
|
public enum Status: String, Codable, Sendable {
|
||||||
|
case partial
|
||||||
|
case final
|
||||||
|
case error
|
||||||
|
case aborted
|
||||||
|
case timeout
|
||||||
|
}
|
||||||
|
|
||||||
|
public let protocolVersion: Int
|
||||||
|
public let sessionId: UUID
|
||||||
|
public let utteranceId: UUID
|
||||||
|
public let commandSeq: Int64
|
||||||
|
public let status: Status
|
||||||
|
public let text: String?
|
||||||
|
public let warning: String?
|
||||||
|
public let errorKind: FlowSessionKeys.TranscriptionErrorKind?
|
||||||
|
public let createdAt: TimeInterval
|
||||||
|
|
||||||
|
public init(
|
||||||
|
protocolVersion: Int = 1,
|
||||||
|
sessionId: UUID,
|
||||||
|
utteranceId: UUID,
|
||||||
|
commandSeq: Int64,
|
||||||
|
status: Status,
|
||||||
|
text: String? = nil,
|
||||||
|
warning: String? = nil,
|
||||||
|
errorKind: FlowSessionKeys.TranscriptionErrorKind? = nil,
|
||||||
|
createdAt: TimeInterval = Date().timeIntervalSince1970
|
||||||
|
) {
|
||||||
|
self.protocolVersion = protocolVersion
|
||||||
|
self.sessionId = sessionId
|
||||||
|
self.utteranceId = utteranceId
|
||||||
|
self.commandSeq = commandSeq
|
||||||
|
self.status = status
|
||||||
|
self.text = text
|
||||||
|
self.warning = warning
|
||||||
|
self.errorKind = errorKind
|
||||||
|
self.createdAt = createdAt
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public struct FlowAck: Codable, Equatable, Sendable {
|
||||||
|
public let protocolVersion: Int
|
||||||
|
public let sessionId: UUID
|
||||||
|
public let utteranceId: UUID
|
||||||
|
public let commandSeq: Int64
|
||||||
|
public let consumedAt: TimeInterval
|
||||||
|
|
||||||
|
public init(
|
||||||
|
protocolVersion: Int = 1,
|
||||||
|
sessionId: UUID,
|
||||||
|
utteranceId: UUID,
|
||||||
|
commandSeq: Int64,
|
||||||
|
consumedAt: TimeInterval = Date().timeIntervalSince1970
|
||||||
|
) {
|
||||||
|
self.protocolVersion = protocolVersion
|
||||||
|
self.sessionId = sessionId
|
||||||
|
self.utteranceId = utteranceId
|
||||||
|
self.commandSeq = commandSeq
|
||||||
|
self.consumedAt = consumedAt
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public struct FlowReadySnapshot: Codable, Equatable, Sendable {
|
||||||
|
public enum Reason: String, Codable, Sendable {
|
||||||
|
case ready
|
||||||
|
case noSession
|
||||||
|
case starting
|
||||||
|
case audioEngineNotLive
|
||||||
|
case waitingForAudioProof
|
||||||
|
case recording
|
||||||
|
case processing
|
||||||
|
case permissionMissing
|
||||||
|
case appGroupUnavailable
|
||||||
|
case hostLost
|
||||||
|
case error
|
||||||
|
}
|
||||||
|
|
||||||
|
public let protocolVersion: Int
|
||||||
|
public let sessionId: UUID?
|
||||||
|
public let ready: Bool
|
||||||
|
public let reason: Reason
|
||||||
|
public let heartbeatAt: TimeInterval
|
||||||
|
public let readyAt: TimeInterval?
|
||||||
|
public let audioProofAt: TimeInterval?
|
||||||
|
public let engineMode: String
|
||||||
|
public let localeId: String
|
||||||
|
public let busyUtteranceId: UUID?
|
||||||
|
public let sessionExpiresAt: TimeInterval?
|
||||||
|
|
||||||
|
public init(
|
||||||
|
protocolVersion: Int = 1,
|
||||||
|
sessionId: UUID?,
|
||||||
|
ready: Bool,
|
||||||
|
reason: Reason,
|
||||||
|
heartbeatAt: TimeInterval = Date().timeIntervalSince1970,
|
||||||
|
readyAt: TimeInterval? = nil,
|
||||||
|
audioProofAt: TimeInterval? = nil,
|
||||||
|
engineMode: String,
|
||||||
|
localeId: String,
|
||||||
|
busyUtteranceId: UUID? = nil,
|
||||||
|
sessionExpiresAt: TimeInterval? = nil
|
||||||
|
) {
|
||||||
|
self.protocolVersion = protocolVersion
|
||||||
|
self.sessionId = sessionId
|
||||||
|
self.ready = ready
|
||||||
|
self.reason = reason
|
||||||
|
self.heartbeatAt = heartbeatAt
|
||||||
|
self.readyAt = readyAt
|
||||||
|
self.audioProofAt = audioProofAt
|
||||||
|
self.engineMode = engineMode
|
||||||
|
self.localeId = localeId
|
||||||
|
self.busyUtteranceId = busyUtteranceId
|
||||||
|
self.sessionExpiresAt = sessionExpiresAt
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
public struct FlowTranscriptionError: Equatable, Sendable {
|
public struct FlowTranscriptionError: Equatable, Sendable {
|
||||||
public let message: String
|
public let message: String
|
||||||
public let kind: FlowSessionKeys.TranscriptionErrorKind
|
public let kind: FlowSessionKeys.TranscriptionErrorKind
|
||||||
@@ -36,6 +188,15 @@ public enum FlowSessionBridge {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private static func encode<T: Encodable>(_ value: T) -> Data? {
|
||||||
|
try? JSONEncoder().encode(value)
|
||||||
|
}
|
||||||
|
|
||||||
|
private static func decode<T: Decodable>(_ type: T.Type, from data: Data?) -> T? {
|
||||||
|
guard let data else { return nil }
|
||||||
|
return try? JSONDecoder().decode(type, from: data)
|
||||||
|
}
|
||||||
|
|
||||||
/// Keyboard/read side: refresh App Group defaults after the extension was
|
/// Keyboard/read side: refresh App Group defaults after the extension was
|
||||||
/// suspended so decisions are not based on stale in-process caches.
|
/// suspended so decisions are not based on stale in-process caches.
|
||||||
public static func reloadFromDisk(defaults: UserDefaults? = nil) {
|
public static func reloadFromDisk(defaults: UserDefaults? = nil) {
|
||||||
@@ -45,10 +206,87 @@ public enum FlowSessionBridge {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// MARK: - Typed Flow protocol
|
||||||
|
|
||||||
|
public static func writeCommand(_ command: FlowCommand, defaults: UserDefaults? = nil) {
|
||||||
|
let store = resolvedDefaults(defaults)
|
||||||
|
if let data = encode(command) {
|
||||||
|
store.set(data, forKey: FlowSessionKeys.flowCommandPayload)
|
||||||
|
}
|
||||||
|
flush(store)
|
||||||
|
FlowSessionDarwin.postCommandChanged()
|
||||||
|
}
|
||||||
|
|
||||||
|
public static func latestCommand(defaults: UserDefaults? = nil) -> FlowCommand? {
|
||||||
|
let store = resolvedDefaults(defaults)
|
||||||
|
return decode(FlowCommand.self, from: store.data(forKey: FlowSessionKeys.flowCommandPayload))
|
||||||
|
}
|
||||||
|
|
||||||
|
public static func writeResult(_ result: FlowResult, defaults: UserDefaults? = nil) {
|
||||||
|
let store = resolvedDefaults(defaults)
|
||||||
|
if let data = encode(result) {
|
||||||
|
store.set(data, forKey: FlowSessionKeys.flowResultPayload)
|
||||||
|
}
|
||||||
|
flush(store)
|
||||||
|
FlowSessionDarwin.postTranscriptionChanged()
|
||||||
|
}
|
||||||
|
|
||||||
|
public static func latestResult(defaults: UserDefaults? = nil) -> FlowResult? {
|
||||||
|
let store = resolvedDefaults(defaults)
|
||||||
|
return decode(FlowResult.self, from: store.data(forKey: FlowSessionKeys.flowResultPayload))
|
||||||
|
}
|
||||||
|
|
||||||
|
public static func clearResult(defaults: UserDefaults? = nil) {
|
||||||
|
let store = resolvedDefaults(defaults)
|
||||||
|
store.removeObject(forKey: FlowSessionKeys.flowResultPayload)
|
||||||
|
flush(store)
|
||||||
|
}
|
||||||
|
|
||||||
|
public static func writeAck(_ ack: FlowAck, defaults: UserDefaults? = nil) {
|
||||||
|
let store = resolvedDefaults(defaults)
|
||||||
|
if let data = encode(ack) {
|
||||||
|
store.set(data, forKey: FlowSessionKeys.flowAckPayload)
|
||||||
|
}
|
||||||
|
flush(store)
|
||||||
|
}
|
||||||
|
|
||||||
|
public static func latestAck(defaults: UserDefaults? = nil) -> FlowAck? {
|
||||||
|
let store = resolvedDefaults(defaults)
|
||||||
|
return decode(FlowAck.self, from: store.data(forKey: FlowSessionKeys.flowAckPayload))
|
||||||
|
}
|
||||||
|
|
||||||
|
public static func writeReadySnapshot(_ snapshot: FlowReadySnapshot, defaults: UserDefaults? = nil) {
|
||||||
|
let store = resolvedDefaults(defaults)
|
||||||
|
if let data = encode(snapshot) {
|
||||||
|
store.set(data, forKey: FlowSessionKeys.flowReadyPayload)
|
||||||
|
}
|
||||||
|
if snapshot.ready {
|
||||||
|
store.set(true, forKey: FlowSessionKeys.flowHostReady)
|
||||||
|
if let readyAt = snapshot.readyAt {
|
||||||
|
store.set(readyAt, forKey: FlowSessionKeys.flowHostReadyAt)
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
clearHostReady(defaults: store, notify: false)
|
||||||
|
store.removeObject(forKey: FlowSessionKeys.flowReadyPayload)
|
||||||
|
}
|
||||||
|
if let expires = snapshot.sessionExpiresAt {
|
||||||
|
store.set(expires, forKey: FlowSessionKeys.flowSessionExpires)
|
||||||
|
}
|
||||||
|
store.set(snapshot.heartbeatAt, forKey: FlowSessionKeys.flowHeartbeat)
|
||||||
|
flush(store)
|
||||||
|
FlowSessionDarwin.postHostReadyChanged()
|
||||||
|
}
|
||||||
|
|
||||||
|
public static func readySnapshot(defaults: UserDefaults? = nil) -> FlowReadySnapshot? {
|
||||||
|
let store = resolvedDefaults(defaults)
|
||||||
|
return decode(FlowReadySnapshot.self, from: store.data(forKey: FlowSessionKeys.flowReadyPayload))
|
||||||
|
}
|
||||||
|
|
||||||
// MARK: - Session lifecycle (host app)
|
// MARK: - Session lifecycle (host app)
|
||||||
|
|
||||||
public static func markSessionActive(
|
public static func markSessionActive(
|
||||||
duration: TimeInterval? = nil,
|
duration: TimeInterval? = nil,
|
||||||
|
sessionId: UUID? = nil,
|
||||||
defaults: UserDefaults? = nil
|
defaults: UserDefaults? = nil
|
||||||
) {
|
) {
|
||||||
let store = resolvedDefaults(defaults)
|
let store = resolvedDefaults(defaults)
|
||||||
@@ -59,8 +297,26 @@ public enum FlowSessionBridge {
|
|||||||
store.set(expires, forKey: FlowSessionKeys.flowSessionExpires)
|
store.set(expires, forKey: FlowSessionKeys.flowSessionExpires)
|
||||||
store.set(now, forKey: FlowSessionKeys.lastActivityAt)
|
store.set(now, forKey: FlowSessionKeys.lastActivityAt)
|
||||||
writeHeartbeat(defaults: store)
|
writeHeartbeat(defaults: store)
|
||||||
setRecordingState(.idle, defaults: store)
|
|
||||||
clearTranscription(defaults: store)
|
clearTranscription(defaults: store)
|
||||||
|
store.removeObject(forKey: FlowSessionKeys.flowCommandPayload)
|
||||||
|
store.removeObject(forKey: FlowSessionKeys.flowResultPayload)
|
||||||
|
store.removeObject(forKey: FlowSessionKeys.flowAckPayload)
|
||||||
|
if let sessionId {
|
||||||
|
let snapshot = FlowReadySnapshot(
|
||||||
|
sessionId: sessionId,
|
||||||
|
ready: false,
|
||||||
|
reason: .starting,
|
||||||
|
heartbeatAt: now,
|
||||||
|
engineMode: AppGroupConfiguration.load(fromAvailable: store).engineMode,
|
||||||
|
localeId: AppGroupConfiguration.load(fromAvailable: store).localeId,
|
||||||
|
sessionExpiresAt: expires
|
||||||
|
)
|
||||||
|
if let data = encode(snapshot) {
|
||||||
|
store.set(data, forKey: FlowSessionKeys.flowReadyPayload)
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
store.removeObject(forKey: FlowSessionKeys.flowReadyPayload)
|
||||||
|
}
|
||||||
flush(store)
|
flush(store)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -69,8 +325,11 @@ public enum FlowSessionBridge {
|
|||||||
store.set(false, forKey: FlowSessionKeys.flowSessionActive)
|
store.set(false, forKey: FlowSessionKeys.flowSessionActive)
|
||||||
store.removeObject(forKey: FlowSessionKeys.flowSessionExpires)
|
store.removeObject(forKey: FlowSessionKeys.flowSessionExpires)
|
||||||
store.removeObject(forKey: FlowSessionKeys.flowHeartbeat)
|
store.removeObject(forKey: FlowSessionKeys.flowHeartbeat)
|
||||||
setRecordingState(.idle, defaults: store)
|
|
||||||
clearTranscription(defaults: store)
|
clearTranscription(defaults: store)
|
||||||
|
store.removeObject(forKey: FlowSessionKeys.flowCommandPayload)
|
||||||
|
store.removeObject(forKey: FlowSessionKeys.flowResultPayload)
|
||||||
|
store.removeObject(forKey: FlowSessionKeys.flowAckPayload)
|
||||||
|
store.removeObject(forKey: FlowSessionKeys.flowReadyPayload)
|
||||||
clearHostReady(defaults: store, notify: false)
|
clearHostReady(defaults: store, notify: false)
|
||||||
flush(store)
|
flush(store)
|
||||||
}
|
}
|
||||||
@@ -185,6 +444,15 @@ public enum FlowSessionBridge {
|
|||||||
/// True when the host has published a fresh ready contract (stricter than heartbeat alone).
|
/// True when the host has published a fresh ready contract (stricter than heartbeat alone).
|
||||||
public static func isHostReady(defaults: UserDefaults? = nil) -> Bool {
|
public static func isHostReady(defaults: UserDefaults? = nil) -> Bool {
|
||||||
let store = resolvedDefaults(defaults)
|
let store = resolvedDefaults(defaults)
|
||||||
|
if let snapshot = readySnapshot(defaults: store) {
|
||||||
|
guard snapshot.ready else { return false }
|
||||||
|
guard isHostReachable(defaults: store) else { return false }
|
||||||
|
if let readyAt = snapshot.readyAt {
|
||||||
|
let skew = abs(snapshot.heartbeatAt - readyAt)
|
||||||
|
guard skew <= FlowSessionKeys.hostReadyMaxHeartbeatSkew else { return false }
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
}
|
||||||
guard isHostReachable(defaults: store) else { return false }
|
guard isHostReachable(defaults: store) else { return false }
|
||||||
return store.bool(forKey: FlowSessionKeys.flowHostReady)
|
return store.bool(forKey: FlowSessionKeys.flowHostReady)
|
||||||
}
|
}
|
||||||
@@ -393,6 +661,10 @@ public enum FlowSessionBridge {
|
|||||||
store.removeObject(forKey: FlowSessionKeys.flowHeartbeat)
|
store.removeObject(forKey: FlowSessionKeys.flowHeartbeat)
|
||||||
store.removeObject(forKey: FlowSessionKeys.keyboardRecordingState)
|
store.removeObject(forKey: FlowSessionKeys.keyboardRecordingState)
|
||||||
store.removeObject(forKey: FlowSessionKeys.transcriptionLanguage)
|
store.removeObject(forKey: FlowSessionKeys.transcriptionLanguage)
|
||||||
|
store.removeObject(forKey: FlowSessionKeys.flowCommandPayload)
|
||||||
|
store.removeObject(forKey: FlowSessionKeys.flowResultPayload)
|
||||||
|
store.removeObject(forKey: FlowSessionKeys.flowAckPayload)
|
||||||
|
store.removeObject(forKey: FlowSessionKeys.flowReadyPayload)
|
||||||
clearTranscription(defaults: store)
|
clearTranscription(defaults: store)
|
||||||
store.removeObject(forKey: FlowSessionKeys.audioLevels)
|
store.removeObject(forKey: FlowSessionKeys.audioLevels)
|
||||||
store.removeObject(forKey: FlowSessionKeys.pendingHostBundleId)
|
store.removeObject(forKey: FlowSessionKeys.pendingHostBundleId)
|
||||||
|
|||||||
@@ -8,6 +8,8 @@ import Foundation
|
|||||||
|
|
||||||
public enum FlowSessionDarwin {
|
public enum FlowSessionDarwin {
|
||||||
public static let notificationName = "com.osgkeyboard.flow.session.changed"
|
public static let notificationName = "com.osgkeyboard.flow.session.changed"
|
||||||
|
/// Posted when the keyboard writes a command for the host app.
|
||||||
|
public static let commandNotificationName = "com.osgkeyboard.flow.command.changed"
|
||||||
/// Posted when the host app writes a transcription result or error.
|
/// Posted when the host app writes a transcription result or error.
|
||||||
public static let transcriptionNotificationName = "com.osgkeyboard.flow.transcription.changed"
|
public static let transcriptionNotificationName = "com.osgkeyboard.flow.transcription.changed"
|
||||||
/// Posted when the host app publishes or clears the ready contract.
|
/// Posted when the host app publishes or clears the ready contract.
|
||||||
@@ -23,6 +25,16 @@ public enum FlowSessionDarwin {
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
public static func postCommandChanged() {
|
||||||
|
CFNotificationCenterPostNotification(
|
||||||
|
CFNotificationCenterGetDarwinNotifyCenter(),
|
||||||
|
CFNotificationName(commandNotificationName as CFString),
|
||||||
|
nil,
|
||||||
|
nil,
|
||||||
|
true
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
public static func postTranscriptionChanged() {
|
public static func postTranscriptionChanged() {
|
||||||
CFNotificationCenterPostNotification(
|
CFNotificationCenterPostNotification(
|
||||||
CFNotificationCenterGetDarwinNotifyCenter(),
|
CFNotificationCenterGetDarwinNotifyCenter(),
|
||||||
|
|||||||
@@ -7,6 +7,10 @@
|
|||||||
import Foundation
|
import Foundation
|
||||||
|
|
||||||
public enum FlowSessionKeys {
|
public enum FlowSessionKeys {
|
||||||
|
public static let flowCommandPayload = "flow.commandPayload.v1"
|
||||||
|
public static let flowResultPayload = "flow.resultPayload.v1"
|
||||||
|
public static let flowAckPayload = "flow.ackPayload.v1"
|
||||||
|
public static let flowReadyPayload = "flow.readyPayload.v1"
|
||||||
public static let flowSessionActive = "flow.flowSessionActive"
|
public static let flowSessionActive = "flow.flowSessionActive"
|
||||||
public static let flowSessionExpires = "flow.flowSessionExpires"
|
public static let flowSessionExpires = "flow.flowSessionExpires"
|
||||||
public static let flowHeartbeat = "flow.flowHeartbeat"
|
public static let flowHeartbeat = "flow.flowHeartbeat"
|
||||||
@@ -74,7 +78,7 @@ public enum FlowSessionKeys {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// Structured host → keyboard transcription failure kind.
|
/// Structured host → keyboard transcription failure kind.
|
||||||
public enum TranscriptionErrorKind: String, Sendable, Equatable {
|
public enum TranscriptionErrorKind: String, Sendable, Equatable, Codable {
|
||||||
case noSpeech
|
case noSpeech
|
||||||
case recognitionInterrupted
|
case recognitionInterrupted
|
||||||
case audioUnavailable
|
case audioUnavailable
|
||||||
|
|||||||
@@ -0,0 +1,159 @@
|
|||||||
|
// LocalASRBiasAdapter.swift
|
||||||
|
// OSGKeyboard · Shared
|
||||||
|
//
|
||||||
|
// Maps `PersonalDictionary` + builtin lexicon + runtime context into the
|
||||||
|
// layered bias outputs consumed by local ASR, correction, and polish.
|
||||||
|
|
||||||
|
import Foundation
|
||||||
|
|
||||||
|
public enum LocalASRBiasAdapter {
|
||||||
|
|
||||||
|
/// Bundle IDs where computer-science vocabulary is especially likely.
|
||||||
|
private static let codeEditorBundleIDs: Set<String> = [
|
||||||
|
"com.apple.dt.Xcode",
|
||||||
|
"com.microsoft.VSCode",
|
||||||
|
"com.google.android.studio",
|
||||||
|
"com.jetbrains.intellij",
|
||||||
|
"com.jetbrains.AppCode",
|
||||||
|
"com.sublimetext.4",
|
||||||
|
"com.apple.Terminal",
|
||||||
|
"com.googlecode.iterm2",
|
||||||
|
"dev.warp.Warp-Stable",
|
||||||
|
]
|
||||||
|
|
||||||
|
public static func adapt(
|
||||||
|
_ request: LocalASRBiasRequest,
|
||||||
|
lexicon: BuiltinLexiconIndex = .shared
|
||||||
|
) -> LocalASRBiasPayload {
|
||||||
|
let capabilities = request.capabilities
|
||||||
|
let dictionary = request.dictionary
|
||||||
|
|
||||||
|
var selectedSources = ["user"]
|
||||||
|
let preferredSources = Self.preferredLexiconSources(for: request.frontAppBundleId)
|
||||||
|
if preferredSources != nil {
|
||||||
|
selectedSources.append("builtin-computer")
|
||||||
|
} else {
|
||||||
|
selectedSources.append("builtin-top")
|
||||||
|
}
|
||||||
|
|
||||||
|
let userSorted = dictionary.effectiveEntries.sorted { $0.usageCount > $1.usageCount }
|
||||||
|
var mergedTerms: [String] = []
|
||||||
|
var seen = Set<String>()
|
||||||
|
func appendTerm(_ term: String) {
|
||||||
|
let trimmed = term.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||||
|
guard !trimmed.isEmpty else { return }
|
||||||
|
let key = trimmed.lowercased()
|
||||||
|
guard seen.insert(key).inserted else { return }
|
||||||
|
mergedTerms.append(trimmed)
|
||||||
|
}
|
||||||
|
|
||||||
|
for entry in userSorted {
|
||||||
|
appendTerm(entry.term)
|
||||||
|
}
|
||||||
|
let userTermCount = mergedTerms.count
|
||||||
|
|
||||||
|
let builtinWords = lexicon.topTerms(
|
||||||
|
limit: request.builtinASRLimit,
|
||||||
|
minimumWeight: 4,
|
||||||
|
preferredSources: preferredSources
|
||||||
|
)
|
||||||
|
let beforeBuiltin = mergedTerms.count
|
||||||
|
for word in builtinWords {
|
||||||
|
appendTerm(word)
|
||||||
|
}
|
||||||
|
let builtinTermCount = mergedTerms.count - beforeBuiltin
|
||||||
|
|
||||||
|
var hardHotwords: [String] = []
|
||||||
|
switch capabilities.hotwordMode {
|
||||||
|
case .perRequest, .recognizerScoped:
|
||||||
|
let cap = max(capabilities.maxHotwordCount, 1)
|
||||||
|
hardHotwords = Self.hardHotwordList(from: mergedTerms, maxCount: cap)
|
||||||
|
case .cloudVocabulary:
|
||||||
|
hardHotwords = dictionary.asrHotwords(maxCount: max(capabilities.maxHotwordCount, 1))
|
||||||
|
case .none, .promptOnly:
|
||||||
|
break
|
||||||
|
}
|
||||||
|
|
||||||
|
var promptBias: String?
|
||||||
|
var truncated = false
|
||||||
|
var truncationReason: String?
|
||||||
|
|
||||||
|
if capabilities.hotwordMode == .promptOnly, capabilities.maxPromptCharacters > 0 {
|
||||||
|
let built = Self.buildPromptBias(
|
||||||
|
dictionary: dictionary,
|
||||||
|
builtinTerms: builtinWords,
|
||||||
|
maxCharacters: capabilities.maxPromptCharacters
|
||||||
|
)
|
||||||
|
if built.count > capabilities.maxPromptCharacters {
|
||||||
|
truncated = true
|
||||||
|
truncationReason = "promptBias exceeded \(capabilities.maxPromptCharacters) characters"
|
||||||
|
}
|
||||||
|
promptBias = built.isEmpty ? nil : built
|
||||||
|
}
|
||||||
|
|
||||||
|
let polishFragment = Self.buildPolishFragment(
|
||||||
|
dictionary: dictionary,
|
||||||
|
builtinTerms: builtinWords,
|
||||||
|
maxTerms: request.builtinPolishLimit
|
||||||
|
)
|
||||||
|
|
||||||
|
let correctionPairs = dictionary.localCorrectionPairs()
|
||||||
|
|
||||||
|
return LocalASRBiasPayload(
|
||||||
|
hardHotwords: hardHotwords,
|
||||||
|
promptBias: promptBias,
|
||||||
|
corpusContext: promptBias,
|
||||||
|
polishFragment: polishFragment,
|
||||||
|
correctionPairs: correctionPairs,
|
||||||
|
diagnostics: LocalASRBiasDiagnostics(
|
||||||
|
userTermCount: userTermCount,
|
||||||
|
builtinTermCount: builtinTermCount,
|
||||||
|
truncated: truncated,
|
||||||
|
truncationReason: truncationReason,
|
||||||
|
selectedSources: selectedSources
|
||||||
|
)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// MARK: - Private
|
||||||
|
|
||||||
|
private static func preferredLexiconSources(for bundleId: String?) -> Set<String>? {
|
||||||
|
guard let bundleId, codeEditorBundleIDs.contains(bundleId) else { return nil }
|
||||||
|
return ["computer_terms"]
|
||||||
|
}
|
||||||
|
|
||||||
|
private static func hardHotwordList(from terms: [String], maxCount: Int) -> [String] {
|
||||||
|
Array(terms.prefix(maxCount))
|
||||||
|
}
|
||||||
|
|
||||||
|
private static func buildPromptBias(
|
||||||
|
dictionary: PersonalDictionary,
|
||||||
|
builtinTerms: [String],
|
||||||
|
maxCharacters: Int
|
||||||
|
) -> String {
|
||||||
|
let userBias = dictionary.asrPromptBias(maxCharacters: maxCharacters)
|
||||||
|
let userTermsLower = Set(dictionary.effectiveEntries.map { $0.term.lowercased() })
|
||||||
|
let extras = builtinTerms.filter { !userTermsLower.contains($0.lowercased()) }
|
||||||
|
guard !extras.isEmpty else { return userBias }
|
||||||
|
|
||||||
|
let extraBlock = "常见技术词汇:\(extras.prefix(80).joined(separator: "、"))"
|
||||||
|
if userBias.isEmpty {
|
||||||
|
return String(extraBlock.prefix(maxCharacters))
|
||||||
|
}
|
||||||
|
let combined = userBias + ";" + extraBlock
|
||||||
|
return String(combined.prefix(maxCharacters))
|
||||||
|
}
|
||||||
|
|
||||||
|
private static func buildPolishFragment(
|
||||||
|
dictionary: PersonalDictionary,
|
||||||
|
builtinTerms: [String],
|
||||||
|
maxTerms: Int
|
||||||
|
) -> String {
|
||||||
|
let userTermsLower = Set(dictionary.effectiveEntries.map { $0.term.lowercased() })
|
||||||
|
let extras = builtinTerms
|
||||||
|
.filter { !userTermsLower.contains($0.lowercased()) }
|
||||||
|
.prefix(maxTerms)
|
||||||
|
guard !extras.isEmpty else { return "" }
|
||||||
|
return "内置技术词汇参考(需原样保留):\(extras.joined(separator: "、"))"
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,56 @@
|
|||||||
|
// LocalASRBiasDiagnosticsStore.swift
|
||||||
|
// OSGKeyboard · Shared
|
||||||
|
//
|
||||||
|
// Persists the most recent local ASR bias diagnostics for settings / debug UI.
|
||||||
|
|
||||||
|
import Foundation
|
||||||
|
|
||||||
|
public struct LocalASRBiasDiagnosticsSnapshot: Codable, Sendable, Equatable {
|
||||||
|
public var capturedAt: Date
|
||||||
|
public var modelId: String?
|
||||||
|
public var backendLabel: String?
|
||||||
|
public var diagnostics: LocalASRBiasDiagnostics
|
||||||
|
public var hotwordCount: Int
|
||||||
|
public var promptBiasLength: Int
|
||||||
|
|
||||||
|
public init(
|
||||||
|
capturedAt: Date = Date(),
|
||||||
|
modelId: String? = nil,
|
||||||
|
backendLabel: String? = nil,
|
||||||
|
diagnostics: LocalASRBiasDiagnostics,
|
||||||
|
hotwordCount: Int = 0,
|
||||||
|
promptBiasLength: Int = 0
|
||||||
|
) {
|
||||||
|
self.capturedAt = capturedAt
|
||||||
|
self.modelId = modelId
|
||||||
|
self.backendLabel = backendLabel
|
||||||
|
self.diagnostics = diagnostics
|
||||||
|
self.hotwordCount = hotwordCount
|
||||||
|
self.promptBiasLength = promptBiasLength
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public enum LocalASRBiasDiagnosticsStore {
|
||||||
|
private static let defaultsKey = "mac.localASR.lastBiasDiagnostics"
|
||||||
|
|
||||||
|
public static func save(payload: LocalASRBiasPayload, modelId: String?, backendLabel: String?) {
|
||||||
|
let snapshot = LocalASRBiasDiagnosticsSnapshot(
|
||||||
|
modelId: modelId,
|
||||||
|
backendLabel: backendLabel,
|
||||||
|
diagnostics: payload.diagnostics,
|
||||||
|
hotwordCount: payload.hardHotwords.count,
|
||||||
|
promptBiasLength: payload.promptBias?.count ?? 0
|
||||||
|
)
|
||||||
|
guard let data = try? JSONEncoder().encode(snapshot) else { return }
|
||||||
|
UserDefaults.standard.set(data, forKey: defaultsKey)
|
||||||
|
}
|
||||||
|
|
||||||
|
public static func load() -> LocalASRBiasDiagnosticsSnapshot? {
|
||||||
|
guard let data = UserDefaults.standard.data(forKey: defaultsKey) else { return nil }
|
||||||
|
return try? JSONDecoder().decode(LocalASRBiasDiagnosticsSnapshot.self, from: data)
|
||||||
|
}
|
||||||
|
|
||||||
|
public static func clear() {
|
||||||
|
UserDefaults.standard.removeObject(forKey: defaultsKey)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
// LocalASRInstalledManifestIO.swift
|
||||||
|
// OSGKeyboard · Shared
|
||||||
|
|
||||||
|
import Foundation
|
||||||
|
|
||||||
|
public enum LocalASRInstalledManifestIO {
|
||||||
|
|
||||||
|
public static func manifestURL(fileManager: FileManager = .default) -> URL {
|
||||||
|
let appSupport = fileManager.urls(for: .applicationSupportDirectory, in: .userDomainMask).first!
|
||||||
|
return appSupport
|
||||||
|
.appendingPathComponent("OSGKeyboard/LocalASRModels/installed-manifest.json")
|
||||||
|
}
|
||||||
|
|
||||||
|
public static func load(defaultModelId: String, fileManager: FileManager = .default) -> LocalASRInstalledManifest {
|
||||||
|
let url = manifestURL(fileManager: fileManager)
|
||||||
|
guard let data = try? Data(contentsOf: url),
|
||||||
|
let manifest = try? JSONDecoder().decode(LocalASRInstalledManifest.self, from: data) else {
|
||||||
|
return LocalASRInstalledManifest(selectedModelId: defaultModelId)
|
||||||
|
}
|
||||||
|
return manifest
|
||||||
|
}
|
||||||
|
|
||||||
|
public static func save(_ manifest: LocalASRInstalledManifest, fileManager: FileManager = .default) throws {
|
||||||
|
let url = manifestURL(fileManager: fileManager)
|
||||||
|
try fileManager.createDirectory(at: url.deletingLastPathComponent(), withIntermediateDirectories: true)
|
||||||
|
let encoder = JSONEncoder()
|
||||||
|
encoder.outputFormatting = [.prettyPrinted, .sortedKeys]
|
||||||
|
encoder.dateEncodingStrategy = .iso8601
|
||||||
|
let data = try encoder.encode(manifest)
|
||||||
|
try data.write(to: url, options: .atomic)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,157 @@
|
|||||||
|
// LocalASRModelDownloadClient.swift
|
||||||
|
// OSGKeyboard · Shared
|
||||||
|
//
|
||||||
|
// URLSession download with byte-level progress and pause/resume (macOS local model installs).
|
||||||
|
|
||||||
|
import Foundation
|
||||||
|
|
||||||
|
#if os(macOS)
|
||||||
|
|
||||||
|
public struct LocalASRDownloadProgressUpdate: Sendable {
|
||||||
|
public let bytesReceived: Int64
|
||||||
|
public let bytesTotal: Int64
|
||||||
|
|
||||||
|
public var fraction: Double {
|
||||||
|
guard bytesTotal > 0 else { return 0 }
|
||||||
|
return min(1, max(0, Double(bytesReceived) / Double(bytesTotal)))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Controls an in-flight URLSession download; supports pause via resume data.
|
||||||
|
public final class LocalASRModelDownloadController: NSObject, URLSessionDownloadDelegate, @unchecked Sendable {
|
||||||
|
private let destinationURL: URL
|
||||||
|
private let onProgress: @Sendable (LocalASRDownloadProgressUpdate) -> Void
|
||||||
|
private lazy var delegateSession: URLSession = {
|
||||||
|
URLSession(configuration: .default, delegate: self, delegateQueue: nil)
|
||||||
|
}()
|
||||||
|
|
||||||
|
private var remoteURL: URL?
|
||||||
|
private var task: URLSessionDownloadTask?
|
||||||
|
private var completionContinuation: CheckedContinuation<Void, Error>?
|
||||||
|
private var isPausing = false
|
||||||
|
private var finished = false
|
||||||
|
|
||||||
|
init(
|
||||||
|
destinationURL: URL,
|
||||||
|
onProgress: @escaping @Sendable (LocalASRDownloadProgressUpdate) -> Void
|
||||||
|
) {
|
||||||
|
self.destinationURL = destinationURL
|
||||||
|
self.onProgress = onProgress
|
||||||
|
super.init()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Runs until the archive is fully written to `destinationURL` (survives pause/resume).
|
||||||
|
public func download(from remoteURL: URL) async throws {
|
||||||
|
self.remoteURL = remoteURL
|
||||||
|
try await withCheckedThrowingContinuation { (continuation: CheckedContinuation<Void, Error>) in
|
||||||
|
completionContinuation = continuation
|
||||||
|
startTask(resumeData: nil)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public func pause() async throws -> Data {
|
||||||
|
guard task != nil, !finished else {
|
||||||
|
throw LocalASRModelManagerError.downloadFailed("No active download to pause.")
|
||||||
|
}
|
||||||
|
return try await withCheckedThrowingContinuation { (continuation: CheckedContinuation<Data, Error>) in
|
||||||
|
isPausing = true
|
||||||
|
task?.cancel(byProducingResumeData: { [weak self] data in
|
||||||
|
guard let self else { return }
|
||||||
|
self.isPausing = false
|
||||||
|
if let data {
|
||||||
|
continuation.resume(returning: data)
|
||||||
|
} else {
|
||||||
|
continuation.resume(throwing: LocalASRModelManagerError.downloadFailed("Pause failed."))
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Continues a paused download; `download(from:)` must still be awaiting.
|
||||||
|
public func resumeFromPause(_ resumeData: Data) {
|
||||||
|
finished = false
|
||||||
|
startTask(resumeData: resumeData)
|
||||||
|
}
|
||||||
|
|
||||||
|
public func cancel() {
|
||||||
|
finished = true
|
||||||
|
task?.cancel()
|
||||||
|
completionContinuation?.resume(throwing: CancellationError())
|
||||||
|
completionContinuation = nil
|
||||||
|
delegateSession.invalidateAndCancel()
|
||||||
|
}
|
||||||
|
|
||||||
|
private func startTask(resumeData: Data?) {
|
||||||
|
if let resumeData {
|
||||||
|
task = delegateSession.downloadTask(withResumeData: resumeData)
|
||||||
|
} else if let remoteURL {
|
||||||
|
task = delegateSession.downloadTask(with: remoteURL)
|
||||||
|
}
|
||||||
|
task?.resume()
|
||||||
|
}
|
||||||
|
|
||||||
|
// MARK: - URLSessionDownloadDelegate
|
||||||
|
|
||||||
|
public func urlSession(
|
||||||
|
_ session: URLSession,
|
||||||
|
downloadTask: URLSessionDownloadTask,
|
||||||
|
didWriteData bytesWritten: Int64,
|
||||||
|
totalBytesWritten: Int64,
|
||||||
|
totalBytesExpectedToWrite: Int64
|
||||||
|
) {
|
||||||
|
onProgress(
|
||||||
|
LocalASRDownloadProgressUpdate(
|
||||||
|
bytesReceived: totalBytesWritten,
|
||||||
|
bytesTotal: max(totalBytesExpectedToWrite, 1)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
public func urlSession(
|
||||||
|
_ session: URLSession,
|
||||||
|
downloadTask: URLSessionDownloadTask,
|
||||||
|
didFinishDownloadingTo location: URL
|
||||||
|
) {
|
||||||
|
guard !finished else { return }
|
||||||
|
finished = true
|
||||||
|
do {
|
||||||
|
let fm = FileManager.default
|
||||||
|
if fm.fileExists(atPath: destinationURL.path) {
|
||||||
|
try fm.removeItem(at: destinationURL)
|
||||||
|
}
|
||||||
|
try fm.moveItem(at: location, to: destinationURL)
|
||||||
|
completionContinuation?.resume()
|
||||||
|
} catch {
|
||||||
|
completionContinuation?.resume(
|
||||||
|
throwing: LocalASRModelManagerError.downloadFailed(error.localizedDescription)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
completionContinuation = nil
|
||||||
|
session.finishTasksAndInvalidate()
|
||||||
|
}
|
||||||
|
|
||||||
|
public func urlSession(_ session: URLSession, task: URLSessionTask, didCompleteWithError error: Error?) {
|
||||||
|
guard !finished else { return }
|
||||||
|
if isPausing { return }
|
||||||
|
if let error {
|
||||||
|
finished = true
|
||||||
|
completionContinuation?.resume(
|
||||||
|
throwing: LocalASRModelManagerError.downloadFailed(error.localizedDescription)
|
||||||
|
)
|
||||||
|
completionContinuation = nil
|
||||||
|
session.finishTasksAndInvalidate()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public enum LocalASRModelDownloadClient {
|
||||||
|
|
||||||
|
public static func makeController(
|
||||||
|
destinationURL: URL,
|
||||||
|
onProgress: @escaping @Sendable (LocalASRDownloadProgressUpdate) -> Void
|
||||||
|
) -> LocalASRModelDownloadController {
|
||||||
|
LocalASRModelDownloadController(destinationURL: destinationURL, onProgress: onProgress)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#endif
|
||||||
@@ -0,0 +1,138 @@
|
|||||||
|
// LocalASRModelInstallState.swift
|
||||||
|
// OSGKeyboard · Shared
|
||||||
|
|
||||||
|
import Foundation
|
||||||
|
|
||||||
|
public enum LocalASRModelInstallState {
|
||||||
|
|
||||||
|
public static func rootDirectory(fileManager: FileManager = .default) -> URL {
|
||||||
|
let appSupport = fileManager.urls(for: .applicationSupportDirectory, in: .userDomainMask).first!
|
||||||
|
return appSupport.appendingPathComponent("OSGKeyboard/LocalASRModels", isDirectory: true)
|
||||||
|
}
|
||||||
|
|
||||||
|
public static func installDirectory(for relativePath: String, fileManager: FileManager = .default) -> URL {
|
||||||
|
rootDirectory(fileManager: fileManager).appendingPathComponent(relativePath, isDirectory: true)
|
||||||
|
}
|
||||||
|
|
||||||
|
public static func isInstalled(
|
||||||
|
_ model: LocalASRModelDefinition,
|
||||||
|
manualMLXPath: String?,
|
||||||
|
fileManager: FileManager = .default
|
||||||
|
) -> Bool {
|
||||||
|
switch model.installKind {
|
||||||
|
case .manual:
|
||||||
|
guard let required = model.requiredRelativeFiles, !required.isEmpty else { return false }
|
||||||
|
let base = URL(fileURLWithPath: manualMLXPath ?? "", isDirectory: true)
|
||||||
|
guard fileManager.fileExists(atPath: base.path) else { return false }
|
||||||
|
return required.allSatisfy { fileManager.fileExists(atPath: base.appendingPathComponent($0).path) }
|
||||||
|
case .archive:
|
||||||
|
guard let relative = model.installRelativePath,
|
||||||
|
let layout = model.layout,
|
||||||
|
let baseName = model.archiveBaseName else { return false }
|
||||||
|
let root = installDirectory(for: relative, fileManager: fileManager)
|
||||||
|
.appendingPathComponent(baseName, isDirectory: true)
|
||||||
|
return validateArchiveModel(at: root, model: model, layout: layout, fileManager: fileManager)
|
||||||
|
case .runtime:
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public static func modelRootURL(
|
||||||
|
_ model: LocalASRModelDefinition,
|
||||||
|
fileManager: FileManager = .default
|
||||||
|
) -> URL? {
|
||||||
|
guard model.installKind == .archive,
|
||||||
|
let relative = model.installRelativePath,
|
||||||
|
let baseName = model.archiveBaseName else { return nil }
|
||||||
|
return installDirectory(for: relative, fileManager: fileManager)
|
||||||
|
.appendingPathComponent(baseName, isDirectory: true)
|
||||||
|
}
|
||||||
|
|
||||||
|
public static func resolveRuntimeBinary(
|
||||||
|
runtime: LocalASRRuntimeDefinition,
|
||||||
|
fileManager: FileManager = .default
|
||||||
|
) -> URL? {
|
||||||
|
let root = installDirectory(for: runtime.installRelativePath, fileManager: fileManager)
|
||||||
|
for candidate in runtime.binaryCandidates {
|
||||||
|
let direct = root.appendingPathComponent(candidate)
|
||||||
|
if fileManager.isExecutableFile(atPath: direct.path) {
|
||||||
|
return direct
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for candidate in runtime.binaryCandidates {
|
||||||
|
let name = (candidate as NSString).lastPathComponent
|
||||||
|
if let found = findExecutable(named: name, under: root, fileManager: fileManager) {
|
||||||
|
return found
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
public static func isRuntimeInstalled(
|
||||||
|
_ runtime: LocalASRRuntimeDefinition,
|
||||||
|
fileManager: FileManager = .default
|
||||||
|
) -> Bool {
|
||||||
|
resolveRuntimeBinary(runtime: runtime, fileManager: fileManager) != nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// MARK: - Private
|
||||||
|
|
||||||
|
private static func validateArchiveModel(
|
||||||
|
at root: URL,
|
||||||
|
model: LocalASRModelDefinition,
|
||||||
|
layout: LocalASRModelLayout,
|
||||||
|
fileManager: FileManager
|
||||||
|
) -> Bool {
|
||||||
|
switch model.backend {
|
||||||
|
case .sherpaQwen3:
|
||||||
|
guard let conv = layout.convFrontend,
|
||||||
|
let encoder = layout.encoder,
|
||||||
|
let decoder = layout.decoder,
|
||||||
|
let tokenizer = layout.tokenizer else { return false }
|
||||||
|
return fileManager.fileExists(atPath: root.appendingPathComponent(conv).path)
|
||||||
|
&& fileManager.fileExists(atPath: root.appendingPathComponent(encoder).path)
|
||||||
|
&& fileManager.fileExists(atPath: root.appendingPathComponent(decoder).path)
|
||||||
|
&& fileManager.fileExists(atPath: root.appendingPathComponent(tokenizer, isDirectory: true).path)
|
||||||
|
case .sherpaSenseVoice:
|
||||||
|
guard let onnx = layout.senseVoiceModel,
|
||||||
|
let tokens = layout.tokens else { return false }
|
||||||
|
return fileManager.fileExists(atPath: root.appendingPathComponent(onnx).path)
|
||||||
|
&& fileManager.fileExists(atPath: root.appendingPathComponent(tokens).path)
|
||||||
|
default:
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private static func findExecutable(
|
||||||
|
named name: String,
|
||||||
|
under root: URL,
|
||||||
|
fileManager: FileManager
|
||||||
|
) -> URL? {
|
||||||
|
guard let enumerator = fileManager.enumerator(
|
||||||
|
at: root,
|
||||||
|
includingPropertiesForKeys: [.isExecutableKey],
|
||||||
|
options: [.skipsHiddenFiles]
|
||||||
|
) else { return nil }
|
||||||
|
for case let url as URL in enumerator {
|
||||||
|
guard url.lastPathComponent == name else { continue }
|
||||||
|
if fileManager.isExecutableFile(atPath: url.path) {
|
||||||
|
return url
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
public static func directoryByteCount(at url: URL, fileManager: FileManager = .default) -> Int64 {
|
||||||
|
guard let enumerator = fileManager.enumerator(
|
||||||
|
at: url,
|
||||||
|
includingPropertiesForKeys: [.fileSizeKey],
|
||||||
|
options: [.skipsHiddenFiles]
|
||||||
|
) else { return 0 }
|
||||||
|
var total: Int64 = 0
|
||||||
|
for case let fileURL as URL in enumerator {
|
||||||
|
let size = (try? fileURL.resourceValues(forKeys: [.fileSizeKey]).fileSize) ?? 0
|
||||||
|
total += Int64(size)
|
||||||
|
}
|
||||||
|
return total
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,492 @@
|
|||||||
|
// LocalASRModelManager.swift
|
||||||
|
// OSGKeyboard · Shared
|
||||||
|
//
|
||||||
|
// Installs local ASR model archives and Sherpa runtimes under Application Support.
|
||||||
|
// Catalog is bundled; installed state is persisted in `installed-manifest.json`.
|
||||||
|
|
||||||
|
import Foundation
|
||||||
|
|
||||||
|
public struct LocalASRInstalledManifest: Codable, Sendable, Equatable {
|
||||||
|
public var schemaVersion: Int
|
||||||
|
public var selectedModelId: String
|
||||||
|
public var installedModelIDs: [String]
|
||||||
|
public var installedRuntimeIDs: [String]
|
||||||
|
public var updatedAt: Date
|
||||||
|
|
||||||
|
public init(
|
||||||
|
schemaVersion: Int = 1,
|
||||||
|
selectedModelId: String,
|
||||||
|
installedModelIDs: [String] = [],
|
||||||
|
installedRuntimeIDs: [String] = [],
|
||||||
|
updatedAt: Date = Date()
|
||||||
|
) {
|
||||||
|
self.schemaVersion = schemaVersion
|
||||||
|
self.selectedModelId = selectedModelId
|
||||||
|
self.installedModelIDs = installedModelIDs
|
||||||
|
self.installedRuntimeIDs = installedRuntimeIDs
|
||||||
|
self.updatedAt = updatedAt
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public enum LocalASRModelInstallPhase: String, Sendable, Equatable {
|
||||||
|
case idle
|
||||||
|
case downloading
|
||||||
|
case paused
|
||||||
|
case extracting
|
||||||
|
case validating
|
||||||
|
case finalizing
|
||||||
|
case failed
|
||||||
|
case completed
|
||||||
|
}
|
||||||
|
|
||||||
|
public struct LocalASRModelInstallProgress: Sendable, Equatable {
|
||||||
|
public var phase: LocalASRModelInstallPhase
|
||||||
|
public var fraction: Double
|
||||||
|
public var message: String
|
||||||
|
public var bytesReceived: Int64?
|
||||||
|
public var bytesTotal: Int64?
|
||||||
|
public var activeItemId: String?
|
||||||
|
|
||||||
|
public init(
|
||||||
|
phase: LocalASRModelInstallPhase,
|
||||||
|
fraction: Double,
|
||||||
|
message: String,
|
||||||
|
bytesReceived: Int64? = nil,
|
||||||
|
bytesTotal: Int64? = nil,
|
||||||
|
activeItemId: String? = nil
|
||||||
|
) {
|
||||||
|
self.phase = phase
|
||||||
|
self.fraction = fraction
|
||||||
|
self.message = message
|
||||||
|
self.bytesReceived = bytesReceived
|
||||||
|
self.bytesTotal = bytesTotal
|
||||||
|
self.activeItemId = activeItemId
|
||||||
|
}
|
||||||
|
|
||||||
|
public static let idle = LocalASRModelInstallProgress(phase: .idle, fraction: 0, message: "")
|
||||||
|
}
|
||||||
|
|
||||||
|
public enum LocalASRModelManagerError: Error, LocalizedError {
|
||||||
|
case downloadFailed(String)
|
||||||
|
case extractFailed(String)
|
||||||
|
case validationFailed(String)
|
||||||
|
case runtimeMissing
|
||||||
|
case binaryMissing
|
||||||
|
|
||||||
|
public var errorDescription: String? {
|
||||||
|
switch self {
|
||||||
|
case .downloadFailed(let detail): return "Download failed: \(detail)"
|
||||||
|
case .extractFailed(let detail): return "Extract failed: \(detail)"
|
||||||
|
case .validationFailed(let detail): return "Validation failed: \(detail)"
|
||||||
|
case .runtimeMissing: return "Sherpa runtime is not installed."
|
||||||
|
case .binaryMissing: return "Sherpa binary not found in runtime bundle."
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public actor LocalASRModelManager {
|
||||||
|
|
||||||
|
public static let shared = LocalASRModelManager()
|
||||||
|
|
||||||
|
private let fileManager = FileManager.default
|
||||||
|
private var progress = LocalASRModelInstallProgress.idle
|
||||||
|
#if os(macOS)
|
||||||
|
private var activeDownloadController: LocalASRModelDownloadController?
|
||||||
|
private var pausedResumeData: Data?
|
||||||
|
#endif
|
||||||
|
|
||||||
|
private init() {}
|
||||||
|
|
||||||
|
public func currentProgress() -> LocalASRModelInstallProgress {
|
||||||
|
progress
|
||||||
|
}
|
||||||
|
|
||||||
|
#if os(macOS)
|
||||||
|
public func pauseDownload() async throws {
|
||||||
|
guard progress.phase == .downloading, let controller = activeDownloadController else { return }
|
||||||
|
let resumeData = try await controller.pause()
|
||||||
|
pausedResumeData = resumeData
|
||||||
|
progress = LocalASRModelInstallProgress(
|
||||||
|
phase: .paused,
|
||||||
|
fraction: progress.fraction,
|
||||||
|
message: progress.message,
|
||||||
|
bytesReceived: progress.bytesReceived,
|
||||||
|
bytesTotal: progress.bytesTotal,
|
||||||
|
activeItemId: progress.activeItemId
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
public func resumeDownload() async throws {
|
||||||
|
guard progress.phase == .paused,
|
||||||
|
let resumeData = pausedResumeData,
|
||||||
|
let controller = activeDownloadController else {
|
||||||
|
throw LocalASRModelManagerError.downloadFailed("No paused download to resume.")
|
||||||
|
}
|
||||||
|
pausedResumeData = nil
|
||||||
|
progress = LocalASRModelInstallProgress(
|
||||||
|
phase: .downloading,
|
||||||
|
fraction: progress.fraction,
|
||||||
|
message: progress.message,
|
||||||
|
bytesReceived: progress.bytesReceived,
|
||||||
|
bytesTotal: progress.bytesTotal,
|
||||||
|
activeItemId: progress.activeItemId
|
||||||
|
)
|
||||||
|
controller.resumeFromPause(resumeData)
|
||||||
|
}
|
||||||
|
|
||||||
|
public func isDownloadPaused() -> Bool {
|
||||||
|
progress.phase == .paused
|
||||||
|
}
|
||||||
|
#endif
|
||||||
|
|
||||||
|
public func rootDirectory() -> URL {
|
||||||
|
let appSupport = fileManager.urls(for: .applicationSupportDirectory, in: .userDomainMask).first!
|
||||||
|
return appSupport.appendingPathComponent("OSGKeyboard/LocalASRModels", isDirectory: true)
|
||||||
|
}
|
||||||
|
|
||||||
|
public func manifestURL() -> URL {
|
||||||
|
LocalASRInstalledManifestIO.manifestURL(fileManager: fileManager)
|
||||||
|
}
|
||||||
|
|
||||||
|
public func loadManifest(defaultModelId: String) -> LocalASRInstalledManifest {
|
||||||
|
LocalASRInstalledManifestIO.load(defaultModelId: defaultModelId, fileManager: fileManager)
|
||||||
|
}
|
||||||
|
|
||||||
|
public func saveManifest(_ manifest: LocalASRInstalledManifest) throws {
|
||||||
|
try LocalASRInstalledManifestIO.save(manifest, fileManager: fileManager)
|
||||||
|
}
|
||||||
|
|
||||||
|
public func setSelectedModelId(_ modelId: String, catalog: LocalASRCatalogDocument) throws {
|
||||||
|
var manifest = loadManifest(defaultModelId: catalog.defaultModelId)
|
||||||
|
manifest.selectedModelId = modelId
|
||||||
|
manifest.updatedAt = Date()
|
||||||
|
try saveManifest(manifest)
|
||||||
|
}
|
||||||
|
|
||||||
|
public func installDirectory(for relativePath: String) -> URL {
|
||||||
|
rootDirectory().appendingPathComponent(relativePath, isDirectory: true)
|
||||||
|
}
|
||||||
|
|
||||||
|
public func isModelInstalled(_ model: LocalASRModelDefinition, manualMLXPath: String?) -> Bool {
|
||||||
|
LocalASRModelInstallState.isInstalled(model, manualMLXPath: manualMLXPath, fileManager: fileManager)
|
||||||
|
}
|
||||||
|
|
||||||
|
public func isRuntimeInstalled(_ runtime: LocalASRRuntimeDefinition) -> Bool {
|
||||||
|
LocalASRModelInstallState.isRuntimeInstalled(runtime, fileManager: fileManager)
|
||||||
|
}
|
||||||
|
|
||||||
|
#if os(macOS)
|
||||||
|
public func resolveRuntimeBinary(runtime: LocalASRRuntimeDefinition) -> URL? {
|
||||||
|
LocalASRModelInstallState.resolveRuntimeBinary(runtime: runtime, fileManager: fileManager)
|
||||||
|
}
|
||||||
|
|
||||||
|
public func installModel(
|
||||||
|
_ model: LocalASRModelDefinition,
|
||||||
|
catalog: LocalASRCatalogDocument
|
||||||
|
) async throws {
|
||||||
|
guard model.installKind == .archive,
|
||||||
|
let relative = model.installRelativePath,
|
||||||
|
let baseName = model.archiveBaseName,
|
||||||
|
let sources = model.sources,
|
||||||
|
!sources.isEmpty else {
|
||||||
|
throw LocalASRModelManagerError.validationFailed("Model is not downloadable.")
|
||||||
|
}
|
||||||
|
|
||||||
|
progress = LocalASRModelInstallProgress(
|
||||||
|
phase: .downloading,
|
||||||
|
fraction: 0.05,
|
||||||
|
message: model.displayName,
|
||||||
|
activeItemId: model.id
|
||||||
|
)
|
||||||
|
if model.backend == .sherpaQwen3 || model.backend == .sherpaSenseVoice {
|
||||||
|
try await ensureRuntimeInstalled(catalog: catalog)
|
||||||
|
}
|
||||||
|
let sortedSources = sources.sorted { $0.priority < $1.priority }
|
||||||
|
var lastError: Error?
|
||||||
|
for source in sortedSources {
|
||||||
|
do {
|
||||||
|
try await installArchive(
|
||||||
|
from: source.url,
|
||||||
|
installRelativePath: relative,
|
||||||
|
archiveBaseName: baseName,
|
||||||
|
layoutModel: model,
|
||||||
|
itemId: model.id,
|
||||||
|
displayName: model.displayName
|
||||||
|
)
|
||||||
|
var manifest = loadManifest(defaultModelId: catalog.defaultModelId)
|
||||||
|
if !manifest.installedModelIDs.contains(model.id) {
|
||||||
|
manifest.installedModelIDs.append(model.id)
|
||||||
|
}
|
||||||
|
manifest.updatedAt = Date()
|
||||||
|
try saveManifest(manifest)
|
||||||
|
progress = LocalASRModelInstallProgress(
|
||||||
|
phase: .completed,
|
||||||
|
fraction: 1,
|
||||||
|
message: model.displayName,
|
||||||
|
activeItemId: model.id
|
||||||
|
)
|
||||||
|
return
|
||||||
|
} catch {
|
||||||
|
lastError = error
|
||||||
|
}
|
||||||
|
}
|
||||||
|
progress = LocalASRModelInstallProgress(
|
||||||
|
phase: .failed,
|
||||||
|
fraction: 0,
|
||||||
|
message: lastError?.localizedDescription ?? "Download failed"
|
||||||
|
)
|
||||||
|
throw lastError ?? LocalASRModelManagerError.downloadFailed("All mirrors failed")
|
||||||
|
}
|
||||||
|
|
||||||
|
public func installRuntime(
|
||||||
|
_ runtime: LocalASRRuntimeDefinition,
|
||||||
|
catalog: LocalASRCatalogDocument
|
||||||
|
) async throws {
|
||||||
|
guard let source = runtime.sources.sorted(by: { $0.priority < $1.priority }).first else {
|
||||||
|
throw LocalASRModelManagerError.downloadFailed("No runtime source configured.")
|
||||||
|
}
|
||||||
|
progress = LocalASRModelInstallProgress(
|
||||||
|
phase: .downloading,
|
||||||
|
fraction: 0.05,
|
||||||
|
message: runtime.displayName,
|
||||||
|
activeItemId: runtime.id
|
||||||
|
)
|
||||||
|
try await installArchive(
|
||||||
|
from: source.url,
|
||||||
|
installRelativePath: runtime.installRelativePath,
|
||||||
|
archiveBaseName: runtime.installRelativePath.split(separator: "/").last.map(String.init) ?? runtime.id,
|
||||||
|
layoutModel: nil,
|
||||||
|
expectedBinaryCandidates: runtime.binaryCandidates,
|
||||||
|
itemId: runtime.id,
|
||||||
|
displayName: runtime.displayName
|
||||||
|
)
|
||||||
|
guard isRuntimeInstalled(runtime) else {
|
||||||
|
throw LocalASRModelManagerError.binaryMissing
|
||||||
|
}
|
||||||
|
var manifest = loadManifest(defaultModelId: catalog.defaultModelId)
|
||||||
|
if !manifest.installedRuntimeIDs.contains(runtime.id) {
|
||||||
|
manifest.installedRuntimeIDs.append(runtime.id)
|
||||||
|
}
|
||||||
|
manifest.updatedAt = Date()
|
||||||
|
try saveManifest(manifest)
|
||||||
|
progress = LocalASRModelInstallProgress(phase: .completed, fraction: 1, message: runtime.displayName)
|
||||||
|
}
|
||||||
|
|
||||||
|
public func modelRootURL(_ model: LocalASRModelDefinition) -> URL? {
|
||||||
|
LocalASRModelInstallState.modelRootURL(model, fileManager: fileManager)
|
||||||
|
}
|
||||||
|
|
||||||
|
public func installDirectoryURL(for model: LocalASRModelDefinition) -> URL? {
|
||||||
|
guard let relative = model.installRelativePath else { return nil }
|
||||||
|
return installDirectory(for: relative)
|
||||||
|
}
|
||||||
|
|
||||||
|
public func deleteModel(
|
||||||
|
_ model: LocalASRModelDefinition,
|
||||||
|
catalog: LocalASRCatalogDocument
|
||||||
|
) throws {
|
||||||
|
guard model.installKind == .archive, let relative = model.installRelativePath else { return }
|
||||||
|
let dir = installDirectory(for: relative)
|
||||||
|
if fileManager.fileExists(atPath: dir.path) {
|
||||||
|
try fileManager.removeItem(at: dir)
|
||||||
|
}
|
||||||
|
var manifest = loadManifest(defaultModelId: catalog.defaultModelId)
|
||||||
|
manifest.installedModelIDs.removeAll { $0 == model.id }
|
||||||
|
if manifest.selectedModelId == model.id {
|
||||||
|
manifest.selectedModelId = catalog.defaultModelId
|
||||||
|
UserDefaults.standard.set(catalog.defaultModelId, forKey: LocalASRPreferenceKeys.selectedModelId)
|
||||||
|
}
|
||||||
|
manifest.updatedAt = Date()
|
||||||
|
try saveManifest(manifest)
|
||||||
|
if progress.activeItemId == model.id {
|
||||||
|
progress = .idle
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
public func deleteRuntime(
|
||||||
|
_ runtime: LocalASRRuntimeDefinition,
|
||||||
|
catalog: LocalASRCatalogDocument
|
||||||
|
) throws {
|
||||||
|
let dir = installDirectory(for: runtime.installRelativePath)
|
||||||
|
if fileManager.fileExists(atPath: dir.path) {
|
||||||
|
try fileManager.removeItem(at: dir)
|
||||||
|
}
|
||||||
|
var manifest = loadManifest(defaultModelId: catalog.defaultModelId)
|
||||||
|
manifest.installedRuntimeIDs.removeAll { $0 == runtime.id }
|
||||||
|
manifest.updatedAt = Date()
|
||||||
|
try saveManifest(manifest)
|
||||||
|
}
|
||||||
|
|
||||||
|
private func setProgress(_ update: LocalASRModelInstallProgress) {
|
||||||
|
progress = update
|
||||||
|
}
|
||||||
|
|
||||||
|
func updateDownloadProgress(
|
||||||
|
itemId: String,
|
||||||
|
displayName: String,
|
||||||
|
update: LocalASRDownloadProgressUpdate
|
||||||
|
) {
|
||||||
|
reportDownloadProgress(itemId: itemId, displayName: displayName, update: update)
|
||||||
|
}
|
||||||
|
|
||||||
|
private func reportDownloadProgress(
|
||||||
|
itemId: String,
|
||||||
|
displayName: String,
|
||||||
|
update: LocalASRDownloadProgressUpdate
|
||||||
|
) {
|
||||||
|
// Download phase occupies 10%–55% of the overall install bar.
|
||||||
|
let mapped = 0.10 + update.fraction * 0.45
|
||||||
|
progress = LocalASRModelInstallProgress(
|
||||||
|
phase: .downloading,
|
||||||
|
fraction: mapped,
|
||||||
|
message: displayName,
|
||||||
|
bytesReceived: update.bytesReceived,
|
||||||
|
bytesTotal: update.bytesTotal,
|
||||||
|
activeItemId: itemId
|
||||||
|
)
|
||||||
|
}
|
||||||
|
#endif
|
||||||
|
|
||||||
|
// MARK: - Private
|
||||||
|
|
||||||
|
#if os(macOS)
|
||||||
|
private func installArchive(
|
||||||
|
from urlString: String,
|
||||||
|
installRelativePath: String,
|
||||||
|
archiveBaseName: String,
|
||||||
|
layoutModel: LocalASRModelDefinition?,
|
||||||
|
expectedBinaryCandidates: [String]? = nil,
|
||||||
|
itemId: String,
|
||||||
|
displayName: String
|
||||||
|
) async throws {
|
||||||
|
guard let remoteURL = URL(string: urlString) else {
|
||||||
|
throw LocalASRModelManagerError.downloadFailed("Invalid URL")
|
||||||
|
}
|
||||||
|
|
||||||
|
let stagingRoot = rootDirectory().appendingPathComponent("staging/\(UUID().uuidString)", isDirectory: true)
|
||||||
|
let destinationParent = installDirectory(for: installRelativePath)
|
||||||
|
try fileManager.createDirectory(at: stagingRoot, withIntermediateDirectories: true)
|
||||||
|
defer { try? fileManager.removeItem(at: stagingRoot) }
|
||||||
|
|
||||||
|
let archiveURL = stagingRoot.appendingPathComponent(remoteURL.lastPathComponent)
|
||||||
|
progress = LocalASRModelInstallProgress(
|
||||||
|
phase: .downloading,
|
||||||
|
fraction: 0.10,
|
||||||
|
message: displayName,
|
||||||
|
activeItemId: itemId
|
||||||
|
)
|
||||||
|
|
||||||
|
do {
|
||||||
|
let controller = LocalASRModelDownloadClient.makeController(destinationURL: archiveURL) { update in
|
||||||
|
Task {
|
||||||
|
await LocalASRModelManager.shared.updateDownloadProgress(
|
||||||
|
itemId: itemId,
|
||||||
|
displayName: displayName,
|
||||||
|
update: update
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
activeDownloadController = controller
|
||||||
|
try await controller.download(from: remoteURL)
|
||||||
|
activeDownloadController = nil
|
||||||
|
pausedResumeData = nil
|
||||||
|
} catch {
|
||||||
|
activeDownloadController = nil
|
||||||
|
pausedResumeData = nil
|
||||||
|
throw LocalASRModelManagerError.downloadFailed(error.localizedDescription)
|
||||||
|
}
|
||||||
|
|
||||||
|
progress = LocalASRModelInstallProgress(
|
||||||
|
phase: .extracting,
|
||||||
|
fraction: 0.58,
|
||||||
|
message: displayName,
|
||||||
|
activeItemId: itemId
|
||||||
|
)
|
||||||
|
try fileManager.createDirectory(at: destinationParent, withIntermediateDirectories: true)
|
||||||
|
let extractOK = try await extractTarBz2(archiveURL: archiveURL, destination: destinationParent)
|
||||||
|
guard extractOK else {
|
||||||
|
throw LocalASRModelManagerError.extractFailed("tar extraction failed")
|
||||||
|
}
|
||||||
|
|
||||||
|
progress = LocalASRModelInstallProgress(
|
||||||
|
phase: .validating,
|
||||||
|
fraction: 0.82,
|
||||||
|
message: displayName,
|
||||||
|
activeItemId: itemId
|
||||||
|
)
|
||||||
|
if let layoutModel {
|
||||||
|
guard LocalASRModelInstallState.isInstalled(
|
||||||
|
layoutModel,
|
||||||
|
manualMLXPath: nil,
|
||||||
|
fileManager: fileManager
|
||||||
|
) else {
|
||||||
|
throw LocalASRModelManagerError.validationFailed("Required model files missing after extract.")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if let expectedBinaryCandidates {
|
||||||
|
let runtimeRoot = destinationParent
|
||||||
|
let found = expectedBinaryCandidates.contains { candidate in
|
||||||
|
let direct = runtimeRoot.appendingPathComponent(candidate)
|
||||||
|
if fileManager.isExecutableFile(atPath: direct.path) { return true }
|
||||||
|
let name = (candidate as NSString).lastPathComponent
|
||||||
|
return findExecutable(named: name, under: runtimeRoot) != nil
|
||||||
|
}
|
||||||
|
guard found else {
|
||||||
|
throw LocalASRModelManagerError.binaryMissing
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
progress = LocalASRModelInstallProgress(
|
||||||
|
phase: .finalizing,
|
||||||
|
fraction: 0.95,
|
||||||
|
message: displayName,
|
||||||
|
activeItemId: itemId
|
||||||
|
)
|
||||||
|
try? fileManager.removeItem(at: archiveURL)
|
||||||
|
}
|
||||||
|
|
||||||
|
public func ensureRuntimeInstalled(catalog: LocalASRCatalogDocument) async throws {
|
||||||
|
guard let runtime = LocalASRModelCatalog.runtime(
|
||||||
|
for: LocalASRModelCatalog.currentRuntimePlatform(),
|
||||||
|
in: catalog
|
||||||
|
) else {
|
||||||
|
throw LocalASRModelManagerError.runtimeMissing
|
||||||
|
}
|
||||||
|
if isRuntimeInstalled(runtime) { return }
|
||||||
|
try await installRuntime(runtime, catalog: catalog)
|
||||||
|
}
|
||||||
|
|
||||||
|
private func findExecutable(named name: String, under root: URL) -> URL? {
|
||||||
|
guard let enumerator = fileManager.enumerator(
|
||||||
|
at: root,
|
||||||
|
includingPropertiesForKeys: [.isExecutableKey],
|
||||||
|
options: [.skipsHiddenFiles]
|
||||||
|
) else { return nil }
|
||||||
|
for case let url as URL in enumerator {
|
||||||
|
guard url.lastPathComponent == name else { continue }
|
||||||
|
if fileManager.isExecutableFile(atPath: url.path) {
|
||||||
|
return url
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
private func extractTarBz2(archiveURL: URL, destination: URL) async throws -> Bool {
|
||||||
|
try await withCheckedThrowingContinuation { continuation in
|
||||||
|
let process = Process()
|
||||||
|
process.executableURL = URL(fileURLWithPath: "/usr/bin/tar")
|
||||||
|
process.arguments = ["-xjf", archiveURL.path, "-C", destination.path]
|
||||||
|
process.standardOutput = FileHandle.nullDevice
|
||||||
|
process.standardError = FileHandle.nullDevice
|
||||||
|
process.terminationHandler = { proc in
|
||||||
|
continuation.resume(returning: proc.terminationStatus == 0)
|
||||||
|
}
|
||||||
|
do {
|
||||||
|
try process.run()
|
||||||
|
} catch {
|
||||||
|
continuation.resume(throwing: LocalASRModelManagerError.extractFailed(error.localizedDescription))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
#endif
|
||||||
|
}
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
// LocalASRPreferenceKeys.swift
|
||||||
|
// OSGKeyboard · Shared
|
||||||
|
|
||||||
|
import Foundation
|
||||||
|
|
||||||
|
enum LocalASRPreferenceKeys {
|
||||||
|
static let selectedModelId = "mac.localASR.selectedModelId"
|
||||||
|
}
|
||||||
@@ -0,0 +1,68 @@
|
|||||||
|
// LocalASRTranscriptCorrector.swift
|
||||||
|
// OSGKeyboard · Shared
|
||||||
|
//
|
||||||
|
// Deterministic alias → canonical term replacement between raw ASR output
|
||||||
|
// and the LLM polish step. Only applies whole-phrase matches.
|
||||||
|
|
||||||
|
import Foundation
|
||||||
|
|
||||||
|
public enum LocalASRTranscriptCorrector {
|
||||||
|
|
||||||
|
/// Applies high-confidence alias replacements (longest match first).
|
||||||
|
public static func apply(
|
||||||
|
_ text: String,
|
||||||
|
pairs: [LocalASRCorrectionPair]
|
||||||
|
) -> String {
|
||||||
|
guard !text.isEmpty, !pairs.isEmpty else { return text }
|
||||||
|
|
||||||
|
let sorted = pairs.sorted { lhs, rhs in
|
||||||
|
if lhs.alias.count != rhs.alias.count {
|
||||||
|
return lhs.alias.count > rhs.alias.count
|
||||||
|
}
|
||||||
|
return lhs.alias.localizedCaseInsensitiveCompare(rhs.alias) == .orderedAscending
|
||||||
|
}
|
||||||
|
|
||||||
|
var result = text
|
||||||
|
for pair in sorted {
|
||||||
|
result = replaceWholeMatches(
|
||||||
|
in: result,
|
||||||
|
alias: pair.alias,
|
||||||
|
term: pair.term
|
||||||
|
)
|
||||||
|
}
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
// MARK: - Private
|
||||||
|
|
||||||
|
private static func replaceWholeMatches(
|
||||||
|
in text: String,
|
||||||
|
alias: String,
|
||||||
|
term: String
|
||||||
|
) -> String {
|
||||||
|
guard !alias.isEmpty, alias != term else { return text }
|
||||||
|
|
||||||
|
if alias.unicodeScalars.allSatisfy({ $0.isASCII }) {
|
||||||
|
return replaceASCIIWord(in: text, alias: alias, term: term)
|
||||||
|
}
|
||||||
|
return text.replacingOccurrences(of: alias, with: term)
|
||||||
|
}
|
||||||
|
|
||||||
|
private static func replaceASCIIWord(
|
||||||
|
in text: String,
|
||||||
|
alias: String,
|
||||||
|
term: String
|
||||||
|
) -> String {
|
||||||
|
let escaped = NSRegularExpression.escapedPattern(for: alias)
|
||||||
|
let pattern = "(?i)(?<![A-Za-z0-9_])\(escaped)(?![A-Za-z0-9_])"
|
||||||
|
guard let regex = try? NSRegularExpression(pattern: pattern) else {
|
||||||
|
return text
|
||||||
|
}
|
||||||
|
let range = NSRange(text.startIndex..<text.endIndex, in: text)
|
||||||
|
return regex.stringByReplacingMatches(
|
||||||
|
in: text,
|
||||||
|
range: range,
|
||||||
|
withTemplate: term
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -252,7 +252,10 @@ public actor PolishingService {
|
|||||||
providerId: String
|
providerId: String
|
||||||
) -> String {
|
) -> String {
|
||||||
let dictionary = store.personalDictionary
|
let dictionary = store.personalDictionary
|
||||||
let dictionaryBlock = dictionary.promptFragment()
|
let dictionaryBlock = Self.mergedDictionaryBlock(
|
||||||
|
dictionary: dictionary,
|
||||||
|
supplement: context.dictionarySupplement
|
||||||
|
)
|
||||||
let contextGuideline = context.appContext.polishGuideline
|
let contextGuideline = context.appContext.polishGuideline
|
||||||
let intensityGuideline = context.intensity.promptGuideline
|
let intensityGuideline = context.intensity.promptGuideline
|
||||||
let contract = Self.globalOutputContract(useChinese: shouldUseChineseGuidance(providerId: providerId))
|
let contract = Self.globalOutputContract(useChinese: shouldUseChineseGuidance(providerId: providerId))
|
||||||
@@ -321,6 +324,17 @@ public actor PolishingService {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
internal static func mergedDictionaryBlock(
|
||||||
|
dictionary: PersonalDictionary,
|
||||||
|
supplement: String?
|
||||||
|
) -> String {
|
||||||
|
let base = dictionary.promptFragment()
|
||||||
|
let extra = supplement?.trimmingCharacters(in: .whitespacesAndNewlines) ?? ""
|
||||||
|
if base.isEmpty { return extra }
|
||||||
|
if extra.isEmpty { return base }
|
||||||
|
return base + "\n" + extra
|
||||||
|
}
|
||||||
|
|
||||||
private func shouldUseChineseGuidance(providerId: String) -> Bool {
|
private func shouldUseChineseGuidance(providerId: String) -> Bool {
|
||||||
switch providerId {
|
switch providerId {
|
||||||
case "zhipu", "moonshot", "qwen", "deepseek":
|
case "zhipu", "moonshot", "qwen", "deepseek":
|
||||||
|
|||||||
@@ -157,8 +157,8 @@
|
|||||||
"mac.settings.recognition" = "RECOGNITION METHOD";
|
"mac.settings.recognition" = "RECOGNITION METHOD";
|
||||||
"mac.settings.cloudEngine" = "Cloud Engine & AI Refinement";
|
"mac.settings.cloudEngine" = "Cloud Engine & AI Refinement";
|
||||||
"mac.settings.cloudEngineDesc" = "Premium transcription via your provider's API, plus AI grammar and style polishing.";
|
"mac.settings.cloudEngineDesc" = "Premium transcription via your provider's API, plus AI grammar and style polishing.";
|
||||||
"mac.settings.localEngine" = "Local Recognition (Qwen3-ASR)";
|
"mac.settings.localEngine" = "Local Recognition";
|
||||||
"mac.settings.localEngineDesc" = "On-device ASR with Qwen3-ASR 1.7B (MLX). High privacy, zero latency.";
|
"mac.settings.localEngineDesc" = "On-device transcription with a local model. High privacy, zero latency.";
|
||||||
"mac.settings.localSpeechFallback" = "Local Recognition (Apple Speech)";
|
"mac.settings.localSpeechFallback" = "Local Recognition (Apple Speech)";
|
||||||
"mac.settings.localSpeechFallbackDesc" = "On-device Apple Speech when the Qwen3 model is not installed.";
|
"mac.settings.localSpeechFallbackDesc" = "On-device Apple Speech when the Qwen3 model is not installed.";
|
||||||
"mac.settings.about" = "About";
|
"mac.settings.about" = "About";
|
||||||
@@ -174,6 +174,9 @@
|
|||||||
"mac.settings.qwen3ModelDesc" = "Folder with config.json, model.safetensors, vocab.json, and merges.txt.";
|
"mac.settings.qwen3ModelDesc" = "Folder with config.json, model.safetensors, vocab.json, and merges.txt.";
|
||||||
"mac.settings.qwen3Browse" = "Choose folder…";
|
"mac.settings.qwen3Browse" = "Choose folder…";
|
||||||
"mac.settings.qwen3Missing" = "Qwen3 model not found — using Apple Speech for now.";
|
"mac.settings.qwen3Missing" = "Qwen3 model not found — using Apple Speech for now.";
|
||||||
|
"mac.settings.mlxModelMissing" = "Select a Qwen3 MLX model folder below, or choose another installed model.";
|
||||||
|
"mac.settings.selectedModelMissing" = "%@ is not installed — using Apple Speech for now.";
|
||||||
|
"mac.settings.localModelFallbackApple" = "No local model is ready — using Apple Speech for now.";
|
||||||
"mac.settings.accessibility" = "Accessibility";
|
"mac.settings.accessibility" = "Accessibility";
|
||||||
"mac.settings.accessibilityDesc" = "Required for global shortcut and auto-paste.";
|
"mac.settings.accessibilityDesc" = "Required for global shortcut and auto-paste.";
|
||||||
"mac.settings.openAccessibility" = "Open System Settings";
|
"mac.settings.openAccessibility" = "Open System Settings";
|
||||||
@@ -188,6 +191,43 @@
|
|||||||
"mac.error.qwen3ModelMissing" = "Qwen3-ASR model not installed";
|
"mac.error.qwen3ModelMissing" = "Qwen3-ASR model not installed";
|
||||||
"mac.error.qwen3LoadFailed" = "Failed to load Qwen3 model: %@";
|
"mac.error.qwen3LoadFailed" = "Failed to load Qwen3 model: %@";
|
||||||
"mac.error.qwen3InferenceFailed" = "Qwen3 transcription failed: %@";
|
"mac.error.qwen3InferenceFailed" = "Qwen3 transcription failed: %@";
|
||||||
|
"mac.localASR.models" = "Local ASR Models";
|
||||||
|
"mac.localASR.modelsDesc" = "Sherpa models download directly. For MLX Qwen3, drop your converted weights into the folder opened by “Open folder”. All three share one storage directory.";
|
||||||
|
"mac.localASR.download" = "Download";
|
||||||
|
"mac.localASR.selectFolder" = "Choose folder";
|
||||||
|
"mac.localASR.openFolder" = "Open folder";
|
||||||
|
"mac.localASR.pause" = "Pause";
|
||||||
|
"mac.localASR.resume" = "Resume";
|
||||||
|
"mac.localASR.needsFolder" = "Folder required";
|
||||||
|
"mac.localASR.installDone" = "Install completed.";
|
||||||
|
"mac.localASR.installed" = "Installed";
|
||||||
|
"mac.localASR.notInstalled" = "Not installed";
|
||||||
|
"mac.localASR.hotwordsYes" = "Hotwords";
|
||||||
|
"mac.localASR.hotwordsNo" = "No hotwords";
|
||||||
|
"mac.localASR.catalogMissing" = "Local ASR catalog is missing from the app bundle.";
|
||||||
|
"mac.localASR.diagnostics" = "Last Bias Diagnostics";
|
||||||
|
"mac.localASR.diagnosticsDesc" = "Captured after your most recent local dictation.";
|
||||||
|
"mac.localASR.diagEmpty" = "No local dictation yet.";
|
||||||
|
"mac.localASR.diagBackend" = "Backend";
|
||||||
|
"mac.localASR.diagUserTerms" = "User terms";
|
||||||
|
"mac.localASR.diagBuiltinTerms" = "Builtin terms";
|
||||||
|
"mac.localASR.diagHotwords" = "Hotwords sent";
|
||||||
|
"mac.localASR.diagPrompt" = "Prompt chars";
|
||||||
|
"mac.localASR.diagTruncated" = "Prompt truncated";
|
||||||
|
"mac.localASR.delete" = "Delete";
|
||||||
|
"mac.localASR.deleteDone" = "Model deleted.";
|
||||||
|
"mac.localASR.redownload" = "Re-download";
|
||||||
|
"mac.localASR.revealInFinder" = "Reveal in Finder";
|
||||||
|
"mac.localASR.openStorage" = "Open model storage folder";
|
||||||
|
"mac.localASR.runtime" = "Sherpa Runtime";
|
||||||
|
"mac.localASR.runtimeDesc" = "Required for Sherpa Qwen3 and SenseVoice models. Installed automatically with those models.";
|
||||||
|
"mac.localASR.phase.downloading" = "Downloading";
|
||||||
|
"mac.localASR.phase.paused" = "Paused";
|
||||||
|
"mac.localASR.phase.extracting" = "Extracting";
|
||||||
|
"mac.localASR.phase.validating" = "Validating";
|
||||||
|
"mac.localASR.phase.finalizing" = "Finalizing";
|
||||||
|
"mac.localASR.phase.failed" = "Failed";
|
||||||
|
"mac.localASR.phase.completed" = "Completed";
|
||||||
"mac.error.accessibilityRequired" = "Enable Accessibility for OSGKeyboard in System Settings";
|
"mac.error.accessibilityRequired" = "Enable Accessibility for OSGKeyboard in System Settings";
|
||||||
"mac.foregroundApp" = "Front app: %@";
|
"mac.foregroundApp" = "Front app: %@";
|
||||||
"mac.sync.settingsTitle" = "iCloud Sync";
|
"mac.sync.settingsTitle" = "iCloud Sync";
|
||||||
|
|||||||
@@ -157,8 +157,8 @@
|
|||||||
"mac.settings.recognition" = "识别方式";
|
"mac.settings.recognition" = "识别方式";
|
||||||
"mac.settings.cloudEngine" = "云端引擎与 AI 润色";
|
"mac.settings.cloudEngine" = "云端引擎与 AI 润色";
|
||||||
"mac.settings.cloudEngineDesc" = "通过服务商 API 进行高质量转写,并自动润色语法与风格。";
|
"mac.settings.cloudEngineDesc" = "通过服务商 API 进行高质量转写,并自动润色语法与风格。";
|
||||||
"mac.settings.localEngine" = "本地识别(Qwen3-ASR)";
|
"mac.settings.localEngine" = "本地识别";
|
||||||
"mac.settings.localEngineDesc" = "使用 Qwen3-ASR 1.7B(MLX)本地转写,高隐私、低延迟。";
|
"mac.settings.localEngineDesc" = "在本机使用本地模型转写,高隐私、低延迟。";
|
||||||
"mac.settings.localSpeechFallback" = "本地识别(Apple Speech)";
|
"mac.settings.localSpeechFallback" = "本地识别(Apple Speech)";
|
||||||
"mac.settings.localSpeechFallbackDesc" = "未安装 Qwen3 模型时,使用 Apple 本地语音识别。";
|
"mac.settings.localSpeechFallbackDesc" = "未安装 Qwen3 模型时,使用 Apple 本地语音识别。";
|
||||||
"mac.settings.about" = "关于";
|
"mac.settings.about" = "关于";
|
||||||
@@ -174,6 +174,9 @@
|
|||||||
"mac.settings.qwen3ModelDesc" = "需包含 config.json、model.safetensors、vocab.json 与 merges.txt。";
|
"mac.settings.qwen3ModelDesc" = "需包含 config.json、model.safetensors、vocab.json 与 merges.txt。";
|
||||||
"mac.settings.qwen3Browse" = "选择文件夹…";
|
"mac.settings.qwen3Browse" = "选择文件夹…";
|
||||||
"mac.settings.qwen3Missing" = "未找到 Qwen3 模型,暂时使用 Apple Speech。";
|
"mac.settings.qwen3Missing" = "未找到 Qwen3 模型,暂时使用 Apple Speech。";
|
||||||
|
"mac.settings.mlxModelMissing" = "请在下方选择 Qwen3 MLX 模型目录,或改用其他已安装的模型。";
|
||||||
|
"mac.settings.selectedModelMissing" = "「%@」尚未安装,暂时使用 Apple Speech。";
|
||||||
|
"mac.settings.localModelFallbackApple" = "没有可用的本地模型,暂时使用 Apple Speech。";
|
||||||
"mac.settings.accessibility" = "辅助功能";
|
"mac.settings.accessibility" = "辅助功能";
|
||||||
"mac.settings.accessibilityDesc" = "全局快捷键与自动粘贴需要此权限。";
|
"mac.settings.accessibilityDesc" = "全局快捷键与自动粘贴需要此权限。";
|
||||||
"mac.settings.openAccessibility" = "打开系统设置";
|
"mac.settings.openAccessibility" = "打开系统设置";
|
||||||
@@ -188,6 +191,43 @@
|
|||||||
"mac.error.qwen3ModelMissing" = "未安装 Qwen3-ASR 模型";
|
"mac.error.qwen3ModelMissing" = "未安装 Qwen3-ASR 模型";
|
||||||
"mac.error.qwen3LoadFailed" = "Qwen3 模型加载失败:%@";
|
"mac.error.qwen3LoadFailed" = "Qwen3 模型加载失败:%@";
|
||||||
"mac.error.qwen3InferenceFailed" = "Qwen3 转写失败:%@";
|
"mac.error.qwen3InferenceFailed" = "Qwen3 转写失败:%@";
|
||||||
|
"mac.localASR.models" = "本地 ASR 模型";
|
||||||
|
"mac.localASR.modelsDesc" = "Sherpa 模型可直接下载;MLX Qwen3 请将转换好的权重放入「打开目录」指向的文件夹。三个模型共用同一存储目录。";
|
||||||
|
"mac.localASR.download" = "下载";
|
||||||
|
"mac.localASR.selectFolder" = "选择目录";
|
||||||
|
"mac.localASR.openFolder" = "打开目录";
|
||||||
|
"mac.localASR.pause" = "暂停";
|
||||||
|
"mac.localASR.resume" = "继续";
|
||||||
|
"mac.localASR.needsFolder" = "需选择目录";
|
||||||
|
"mac.localASR.installDone" = "安装完成。";
|
||||||
|
"mac.localASR.installed" = "已安装";
|
||||||
|
"mac.localASR.notInstalled" = "未安装";
|
||||||
|
"mac.localASR.hotwordsYes" = "支持热词";
|
||||||
|
"mac.localASR.hotwordsNo" = "无热词";
|
||||||
|
"mac.localASR.catalogMissing" = "应用包内缺少本地 ASR 模型目录。";
|
||||||
|
"mac.localASR.diagnostics" = "最近一次词库诊断";
|
||||||
|
"mac.localASR.diagnosticsDesc" = "在上一轮本地听写后记录。";
|
||||||
|
"mac.localASR.diagEmpty" = "尚无本地听写记录。";
|
||||||
|
"mac.localASR.diagBackend" = "后端";
|
||||||
|
"mac.localASR.diagUserTerms" = "用户词条";
|
||||||
|
"mac.localASR.diagBuiltinTerms" = "内置词条";
|
||||||
|
"mac.localASR.diagHotwords" = "热词数量";
|
||||||
|
"mac.localASR.diagPrompt" = "Prompt 字符";
|
||||||
|
"mac.localASR.diagTruncated" = "Prompt 已截断";
|
||||||
|
"mac.localASR.delete" = "删除";
|
||||||
|
"mac.localASR.deleteDone" = "模型已删除。";
|
||||||
|
"mac.localASR.redownload" = "重新下载";
|
||||||
|
"mac.localASR.revealInFinder" = "在 Finder 中显示";
|
||||||
|
"mac.localASR.openStorage" = "打开模型存储目录";
|
||||||
|
"mac.localASR.runtime" = "Sherpa 运行时";
|
||||||
|
"mac.localASR.runtimeDesc" = "Sherpa Qwen3 与 SenseVoice 模型需要此运行时;下载上述模型时会自动安装。";
|
||||||
|
"mac.localASR.phase.downloading" = "下载中";
|
||||||
|
"mac.localASR.phase.paused" = "已暂停";
|
||||||
|
"mac.localASR.phase.extracting" = "解压中";
|
||||||
|
"mac.localASR.phase.validating" = "校验中";
|
||||||
|
"mac.localASR.phase.finalizing" = "完成安装";
|
||||||
|
"mac.localASR.phase.failed" = "失败";
|
||||||
|
"mac.localASR.phase.completed" = "已完成";
|
||||||
"mac.error.accessibilityRequired" = "请在系统设置中为 OSGKeyboard 启用辅助功能";
|
"mac.error.accessibilityRequired" = "请在系统设置中为 OSGKeyboard 启用辅助功能";
|
||||||
"mac.foregroundApp" = "前台应用:%@";
|
"mac.foregroundApp" = "前台应用:%@";
|
||||||
"mac.sync.settingsTitle" = "iCloud 同步";
|
"mac.sync.settingsTitle" = "iCloud 同步";
|
||||||
|
|||||||
@@ -141,6 +141,7 @@ final class FlowSessionBridgeTests: XCTestCase {
|
|||||||
|
|
||||||
func testDarwinNotificationPostsWithoutCrashing() {
|
func testDarwinNotificationPostsWithoutCrashing() {
|
||||||
FlowSessionDarwin.postSessionChanged()
|
FlowSessionDarwin.postSessionChanged()
|
||||||
|
FlowSessionDarwin.postCommandChanged()
|
||||||
FlowSessionDarwin.postHostReadyChanged()
|
FlowSessionDarwin.postHostReadyChanged()
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -183,4 +184,112 @@ final class FlowSessionBridgeTests: XCTestCase {
|
|||||||
XCTAssertFalse(defaults.bool(forKey: FlowSessionKeys.flowHostReady))
|
XCTAssertFalse(defaults.bool(forKey: FlowSessionKeys.flowHostReady))
|
||||||
XCTAssertFalse(FlowSessionBridge.isHostReady(defaults: defaults))
|
XCTAssertFalse(FlowSessionBridge.isHostReady(defaults: defaults))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func testFlowCommandRoundTrip() {
|
||||||
|
let defaults = makeDefaults()
|
||||||
|
let sessionId = UUID()
|
||||||
|
let utteranceId = UUID()
|
||||||
|
let command = FlowCommand(
|
||||||
|
sessionId: sessionId,
|
||||||
|
utteranceId: utteranceId,
|
||||||
|
commandSeq: 42,
|
||||||
|
action: .startRecording,
|
||||||
|
localeId: "zh-Hans",
|
||||||
|
createdAt: 123
|
||||||
|
)
|
||||||
|
|
||||||
|
FlowSessionBridge.writeCommand(command, defaults: defaults)
|
||||||
|
|
||||||
|
XCTAssertEqual(FlowSessionBridge.latestCommand(defaults: defaults), command)
|
||||||
|
}
|
||||||
|
|
||||||
|
func testFlowResultRoundTripPreservesUtteranceIdentity() {
|
||||||
|
let defaults = makeDefaults()
|
||||||
|
let sessionId = UUID()
|
||||||
|
let utteranceId = UUID()
|
||||||
|
let result = FlowResult(
|
||||||
|
sessionId: sessionId,
|
||||||
|
utteranceId: utteranceId,
|
||||||
|
commandSeq: 43,
|
||||||
|
status: .final,
|
||||||
|
text: "hello",
|
||||||
|
warning: "raw fallback",
|
||||||
|
createdAt: 124
|
||||||
|
)
|
||||||
|
|
||||||
|
FlowSessionBridge.writeResult(result, defaults: defaults)
|
||||||
|
|
||||||
|
XCTAssertEqual(FlowSessionBridge.latestResult(defaults: defaults), result)
|
||||||
|
FlowSessionBridge.clearResult(defaults: defaults)
|
||||||
|
XCTAssertNil(FlowSessionBridge.latestResult(defaults: defaults))
|
||||||
|
}
|
||||||
|
|
||||||
|
func testFlowAckRoundTrip() {
|
||||||
|
let defaults = makeDefaults()
|
||||||
|
let ack = FlowAck(
|
||||||
|
sessionId: UUID(),
|
||||||
|
utteranceId: UUID(),
|
||||||
|
commandSeq: 44,
|
||||||
|
consumedAt: 125
|
||||||
|
)
|
||||||
|
|
||||||
|
FlowSessionBridge.writeAck(ack, defaults: defaults)
|
||||||
|
|
||||||
|
XCTAssertEqual(FlowSessionBridge.latestAck(defaults: defaults), ack)
|
||||||
|
}
|
||||||
|
|
||||||
|
func testReadySnapshotDrivesHostReady() {
|
||||||
|
let defaults = makeDefaults()
|
||||||
|
let sessionId = UUID()
|
||||||
|
let now = Date().timeIntervalSince1970
|
||||||
|
FlowSessionBridge.markSessionActive(duration: 60, sessionId: sessionId, defaults: defaults)
|
||||||
|
let snapshot = FlowReadySnapshot(
|
||||||
|
sessionId: sessionId,
|
||||||
|
ready: true,
|
||||||
|
reason: .ready,
|
||||||
|
heartbeatAt: now,
|
||||||
|
readyAt: now,
|
||||||
|
audioProofAt: now,
|
||||||
|
engineMode: "local",
|
||||||
|
localeId: "zh-Hans",
|
||||||
|
sessionExpiresAt: now + 60
|
||||||
|
)
|
||||||
|
|
||||||
|
FlowSessionBridge.writeReadySnapshot(snapshot, defaults: defaults)
|
||||||
|
|
||||||
|
XCTAssertEqual(FlowSessionBridge.readySnapshot(defaults: defaults), snapshot)
|
||||||
|
XCTAssertTrue(FlowSessionBridge.isHostReady(defaults: defaults))
|
||||||
|
}
|
||||||
|
|
||||||
|
func testClearFlowStateRemovesProtocolPayloads() {
|
||||||
|
let defaults = makeDefaults()
|
||||||
|
let sessionId = UUID()
|
||||||
|
let utteranceId = UUID()
|
||||||
|
FlowSessionBridge.writeCommand(
|
||||||
|
FlowCommand(
|
||||||
|
sessionId: sessionId,
|
||||||
|
utteranceId: utteranceId,
|
||||||
|
commandSeq: 1,
|
||||||
|
action: .startRecording,
|
||||||
|
localeId: "en-US"
|
||||||
|
),
|
||||||
|
defaults: defaults
|
||||||
|
)
|
||||||
|
FlowSessionBridge.writeResult(
|
||||||
|
FlowResult(
|
||||||
|
sessionId: sessionId,
|
||||||
|
utteranceId: utteranceId,
|
||||||
|
commandSeq: 1,
|
||||||
|
status: .partial,
|
||||||
|
text: "hello"
|
||||||
|
),
|
||||||
|
defaults: defaults
|
||||||
|
)
|
||||||
|
|
||||||
|
FlowSessionBridge.clearFlowState(defaults: defaults)
|
||||||
|
|
||||||
|
XCTAssertNil(FlowSessionBridge.latestCommand(defaults: defaults))
|
||||||
|
XCTAssertNil(FlowSessionBridge.latestResult(defaults: defaults))
|
||||||
|
XCTAssertNil(FlowSessionBridge.readySnapshot(defaults: defaults))
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,124 @@
|
|||||||
|
// LocalASRBiasAdapterTests.swift
|
||||||
|
// OSGKeyboardTests
|
||||||
|
|
||||||
|
import XCTest
|
||||||
|
@testable import OSGKeyboardShared
|
||||||
|
|
||||||
|
final class LocalASRBiasAdapterTests: XCTestCase {
|
||||||
|
|
||||||
|
private func makeFixtureLexicon() throws -> BuiltinLexiconIndex {
|
||||||
|
let dir = FileManager.default.temporaryDirectory
|
||||||
|
.appendingPathComponent("osg-phrases-\(UUID().uuidString)", isDirectory: true)
|
||||||
|
try FileManager.default.createDirectory(at: dir, withIntermediateDirectories: true)
|
||||||
|
let url = dir.appendingPathComponent("phrases.tsv")
|
||||||
|
let tsv = """
|
||||||
|
word\tpinyin\tsource\tweight
|
||||||
|
SwiftUI\tswift ui\tcomputer_terms\t5
|
||||||
|
Kubernetes\tku bo ne si\tcomputer_terms\t5
|
||||||
|
一致性\tyi zhi xing\tcomputer_terms\t5
|
||||||
|
"""
|
||||||
|
try tsv.write(to: url, atomically: true, encoding: .utf8)
|
||||||
|
addTeardownBlock {
|
||||||
|
try? FileManager.default.removeItem(at: dir)
|
||||||
|
}
|
||||||
|
return BuiltinLexiconIndex(fixtureURL: url)
|
||||||
|
}
|
||||||
|
|
||||||
|
func testAdaptBuildsPromptBiasForQwen3MLX() throws {
|
||||||
|
let lexicon = try makeFixtureLexicon()
|
||||||
|
var dict = PersonalDictionary.empty
|
||||||
|
_ = dict.upsertManual(term: "Cursor")
|
||||||
|
dict.updateAliases(for: dict.entries[0].id, aliases: ["cursor"])
|
||||||
|
|
||||||
|
let payload = LocalASRBiasAdapter.adapt(
|
||||||
|
LocalASRBiasRequest(
|
||||||
|
dictionary: dict,
|
||||||
|
locale: Locale(identifier: "zh-CN"),
|
||||||
|
capabilities: .qwen3MLX
|
||||||
|
),
|
||||||
|
lexicon: lexicon
|
||||||
|
)
|
||||||
|
|
||||||
|
XCTAssertNotNil(payload.promptBias)
|
||||||
|
XCTAssertTrue(payload.promptBias?.contains("Cursor") == true)
|
||||||
|
XCTAssertTrue(payload.promptBias?.contains("SwiftUI") == true)
|
||||||
|
XCTAssertEqual(payload.diagnostics.userTermCount, 2) // OSGKeyboard system + Cursor
|
||||||
|
XCTAssertGreaterThan(payload.diagnostics.builtinTermCount, 0)
|
||||||
|
}
|
||||||
|
|
||||||
|
func testAdaptProducesPolishFragmentWithoutUserDuplicates() throws {
|
||||||
|
let lexicon = try makeFixtureLexicon()
|
||||||
|
var dict = PersonalDictionary.empty
|
||||||
|
_ = dict.upsertManual(term: "SwiftUI")
|
||||||
|
|
||||||
|
let payload = LocalASRBiasAdapter.adapt(
|
||||||
|
LocalASRBiasRequest(
|
||||||
|
dictionary: dict,
|
||||||
|
locale: Locale(identifier: "zh-CN"),
|
||||||
|
capabilities: .qwen3MLX
|
||||||
|
),
|
||||||
|
lexicon: lexicon
|
||||||
|
)
|
||||||
|
|
||||||
|
XCTAssertFalse(payload.polishFragment.contains("SwiftUI"))
|
||||||
|
XCTAssertTrue(payload.polishFragment.contains("Kubernetes"))
|
||||||
|
}
|
||||||
|
|
||||||
|
func testCorrectionPairsFromAliases() {
|
||||||
|
var dict = PersonalDictionary.empty
|
||||||
|
_ = dict.upsertManual(term: "Kubernetes")
|
||||||
|
dict.updateAliases(for: dict.entries[0].id, aliases: ["k8s"])
|
||||||
|
|
||||||
|
let payload = LocalASRBiasAdapter.adapt(
|
||||||
|
LocalASRBiasRequest(
|
||||||
|
dictionary: dict,
|
||||||
|
locale: Locale(identifier: "zh-CN"),
|
||||||
|
capabilities: .qwen3MLX
|
||||||
|
),
|
||||||
|
lexicon: BuiltinLexiconIndex.shared
|
||||||
|
)
|
||||||
|
|
||||||
|
XCTAssertEqual(payload.correctionPairs.count, 1)
|
||||||
|
XCTAssertEqual(payload.correctionPairs[0].alias, "k8s")
|
||||||
|
XCTAssertEqual(payload.correctionPairs[0].term, "Kubernetes")
|
||||||
|
}
|
||||||
|
|
||||||
|
func testTranscriptCorrectorReplacesASCIIAlias() {
|
||||||
|
let pairs = [LocalASRCorrectionPair(alias: "k8s", term: "Kubernetes")]
|
||||||
|
let result = LocalASRTranscriptCorrector.apply(
|
||||||
|
"部署 k8s 集群",
|
||||||
|
pairs: pairs
|
||||||
|
)
|
||||||
|
XCTAssertEqual(result, "部署 Kubernetes 集群")
|
||||||
|
}
|
||||||
|
|
||||||
|
func testTranscriptCorrectorSkipsPartialASCIIMatch() {
|
||||||
|
let pairs = [LocalASRCorrectionPair(alias: "k8s", term: "Kubernetes")]
|
||||||
|
let result = LocalASRTranscriptCorrector.apply(
|
||||||
|
"xk8s集群",
|
||||||
|
pairs: pairs
|
||||||
|
)
|
||||||
|
XCTAssertEqual(result, "xk8s集群")
|
||||||
|
}
|
||||||
|
|
||||||
|
func testBuiltinLexiconParsesTSV() {
|
||||||
|
let terms = BuiltinLexiconIndex.parseTSV(
|
||||||
|
"word\tpinyin\tsource\tweight\nFoo\tfoo\tcomputer_terms\t5\n"
|
||||||
|
)
|
||||||
|
XCTAssertEqual(terms.count, 1)
|
||||||
|
XCTAssertEqual(terms[0].word, "Foo")
|
||||||
|
XCTAssertEqual(terms[0].weight, 5)
|
||||||
|
}
|
||||||
|
|
||||||
|
func testPolishingServiceMergesDictionarySupplement() {
|
||||||
|
let dict = PersonalDictionary(entries: [
|
||||||
|
PersonalDictionary.Entry(term: "Cursor", category: .productName, source: .manual),
|
||||||
|
])
|
||||||
|
let merged = PolishingService.mergedDictionaryBlock(
|
||||||
|
dictionary: dict,
|
||||||
|
supplement: "内置技术词汇参考:SwiftUI"
|
||||||
|
)
|
||||||
|
XCTAssertTrue(merged.contains("Cursor"))
|
||||||
|
XCTAssertTrue(merged.contains("SwiftUI"))
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,88 @@
|
|||||||
|
// LocalASRModelCatalogTests.swift
|
||||||
|
// OSGKeyboardTests
|
||||||
|
|
||||||
|
import XCTest
|
||||||
|
@testable import OSGKeyboardShared
|
||||||
|
|
||||||
|
final class LocalASRModelCatalogTests: XCTestCase {
|
||||||
|
|
||||||
|
func testBundledCatalogLoads() throws {
|
||||||
|
let catalog = try LocalASRModelCatalog.loadBundled()
|
||||||
|
XCTAssertEqual(catalog.schemaVersion, 1)
|
||||||
|
XCTAssertFalse(catalog.models.isEmpty)
|
||||||
|
XCTAssertTrue(catalog.models.contains { $0.id == "qwen3-mlx-1.7b" })
|
||||||
|
XCTAssertTrue(catalog.models.contains { $0.id == "sherpa-qwen3-0.6b-int8" })
|
||||||
|
}
|
||||||
|
|
||||||
|
func testCapabilitiesForSherpaQwen3() throws {
|
||||||
|
let catalog = try LocalASRModelCatalog.loadBundled()
|
||||||
|
let model = try XCTUnwrap(LocalASRModelCatalog.model("sherpa-qwen3-0.6b-int8", in: catalog))
|
||||||
|
let caps = LocalASRModelCatalog.capabilities(for: model)
|
||||||
|
XCTAssertEqual(caps.hotwordMode, .recognizerScoped)
|
||||||
|
XCTAssertTrue(model.supportsHotwords)
|
||||||
|
}
|
||||||
|
|
||||||
|
func testManifestRoundTrip() throws {
|
||||||
|
let manifest = LocalASRInstalledManifest(
|
||||||
|
selectedModelId: "sherpa-qwen3-0.6b-int8",
|
||||||
|
installedModelIDs: ["sherpa-qwen3-0.6b-int8"]
|
||||||
|
)
|
||||||
|
let url = FileManager.default.temporaryDirectory
|
||||||
|
.appendingPathComponent("manifest-\(UUID().uuidString).json")
|
||||||
|
defer { try? FileManager.default.removeItem(at: url) }
|
||||||
|
|
||||||
|
let encoder = JSONEncoder()
|
||||||
|
encoder.dateEncodingStrategy = .iso8601
|
||||||
|
try encoder.encode(manifest).write(to: url)
|
||||||
|
|
||||||
|
let decoder = JSONDecoder()
|
||||||
|
decoder.dateDecodingStrategy = .iso8601
|
||||||
|
let loaded = try decoder.decode(LocalASRInstalledManifest.self, from: Data(contentsOf: url))
|
||||||
|
XCTAssertEqual(loaded.selectedModelId, manifest.selectedModelId)
|
||||||
|
XCTAssertEqual(loaded.installedModelIDs, manifest.installedModelIDs)
|
||||||
|
}
|
||||||
|
|
||||||
|
func testBiasDiagnosticsStoreRoundTrip() {
|
||||||
|
LocalASRBiasDiagnosticsStore.clear()
|
||||||
|
let payload = LocalASRBiasPayload(
|
||||||
|
hardHotwords: ["Cursor"],
|
||||||
|
promptBias: "test",
|
||||||
|
corpusContext: nil,
|
||||||
|
polishFragment: "fragment",
|
||||||
|
correctionPairs: [],
|
||||||
|
diagnostics: LocalASRBiasDiagnostics(userTermCount: 2, builtinTermCount: 3)
|
||||||
|
)
|
||||||
|
LocalASRBiasDiagnosticsStore.save(
|
||||||
|
payload: payload,
|
||||||
|
modelId: "qwen3-mlx-1.7b",
|
||||||
|
backendLabel: "MLX"
|
||||||
|
)
|
||||||
|
let snapshot = LocalASRBiasDiagnosticsStore.load()
|
||||||
|
XCTAssertEqual(snapshot?.modelId, "qwen3-mlx-1.7b")
|
||||||
|
XCTAssertEqual(snapshot?.diagnostics.userTermCount, 2)
|
||||||
|
XCTAssertEqual(snapshot?.hotwordCount, 1)
|
||||||
|
LocalASRBiasDiagnosticsStore.clear()
|
||||||
|
}
|
||||||
|
|
||||||
|
func testSherpaAdapterProducesHardHotwords() throws {
|
||||||
|
let fixtureURL = FileManager.default.temporaryDirectory
|
||||||
|
.appendingPathComponent("phrases-\(UUID().uuidString).tsv")
|
||||||
|
try "word\tpinyin\tsource\tweight\nSwiftUI\tswift ui\tcomputer_terms\t5\n"
|
||||||
|
.write(to: fixtureURL, atomically: true, encoding: .utf8)
|
||||||
|
defer { try? FileManager.default.removeItem(at: fixtureURL) }
|
||||||
|
|
||||||
|
var dict = PersonalDictionary.empty
|
||||||
|
_ = dict.upsertManual(term: "Kubernetes")
|
||||||
|
|
||||||
|
let payload = LocalASRBiasAdapter.adapt(
|
||||||
|
LocalASRBiasRequest(
|
||||||
|
dictionary: dict,
|
||||||
|
locale: Locale(identifier: "zh-CN"),
|
||||||
|
capabilities: .sherpaQwen3
|
||||||
|
),
|
||||||
|
lexicon: BuiltinLexiconIndex(fixtureURL: fixtureURL)
|
||||||
|
)
|
||||||
|
XCTAssertFalse(payload.hardHotwords.isEmpty)
|
||||||
|
XCTAssertTrue(payload.hardHotwords.contains("Kubernetes"))
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,585 @@
|
|||||||
|
# OSGKeyboard 本地 ASR 技术架构
|
||||||
|
|
||||||
|
> **文档状态**:架构规划(非实现规格)
|
||||||
|
> **适用范围**:macOS 本地听写;与 iOS 键盘扩展、云 ASR 路径的关系见各节说明。
|
||||||
|
> **核心结论**:短期不换主模型,优先打通 **词库感知管道**;中期用 POC 验证 **Sherpa Qwen3 hard hotwords** 是否值得成为热词主线。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Executive Summary
|
||||||
|
|
||||||
|
OSGKeyboard 的本地 ASR 竞争力不来自单一模型,而来自:
|
||||||
|
|
||||||
|
1. **用户 PersonalDictionary**(term / aliases / iCloud)
|
||||||
|
2. **内置技术词库**(`phrases.tsv` ≈ 1 万词,iOS 已用于 Apple CLM)
|
||||||
|
3. **分层 bias**:ASR 偏置 → 后处理纠错 → Polish 保真
|
||||||
|
4. **可替换的 Local ASR Provider**(Qwen3 MLX 主线,Sherpa / SenseVoice / Apple Speech 对照)
|
||||||
|
|
||||||
|
当前最大缺口:**macOS 本地路径未消费任何词库**;云路径已通过 `PersonalDictionary+ASRBias` 完整接线。
|
||||||
|
|
||||||
|
推荐路线:
|
||||||
|
|
||||||
|
| 阶段 | 动作 |
|
||||||
|
|------|------|
|
||||||
|
| **短期** | 保留 Qwen3 MLX;实现 `LocalASRBiasAdapter`;接 soft prompt + polish + aliases 后处理 |
|
||||||
|
| **中期** | ModelScope 优先的本地模型 catalog;Sherpa Qwen3 hotwords POC |
|
||||||
|
| **长期** | 按评测数据决定是否新增默认 provider 或保留 Qwen3 MLX |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. 背景与问题定义
|
||||||
|
|
||||||
|
### 2.1 为什么本地 ASR 不能只讨论模型
|
||||||
|
|
||||||
|
语音输入的「专有名词准确率」由多层共同决定:
|
||||||
|
|
||||||
|
- **ASR 层**:听出 `Claude`、`SwiftUI`、`Qwen3-ASR`
|
||||||
|
- **后处理层**:`克劳德` → `Claude`
|
||||||
|
- **润色层**:保留品牌名、变量名,不擅自改写
|
||||||
|
|
||||||
|
闭源产品(Typeless 等)常把词典效果归因于云端 ASR;开源竞品(OpenLess、Typeflux、SayIt)表明:**词典必须按 backend 能力分层注入**,不能假设「一个 hotwords 数组走天下」。
|
||||||
|
|
||||||
|
### 2.2 OSG 相对竞品的结构性优势
|
||||||
|
|
||||||
|
| 能力 | OSGKeyboard | 典型开源竞品 |
|
||||||
|
|------|-------------|----------------|
|
||||||
|
| 用户词库 | `PersonalDictionary`(term + aliases + category + iCloud) | 多为 phrase-only |
|
||||||
|
| 内置领域词库 | ~10k `phrases.tsv` + iOS CLM | OpenLess preset ~20 词;SayIt server hotwords.txt ~30 词 |
|
||||||
|
| 云 ASR bias | 智谱 / 阿里 vocabulary / Whisper prompt | 单云或单 provider |
|
||||||
|
| iOS 本地 CLM | `SFCustomLanguageModelData` | macOS 路径未等价 |
|
||||||
|
|
||||||
|
### 2.3 设计目标
|
||||||
|
|
||||||
|
- 离线、隐私友好的 macOS 本地听写
|
||||||
|
- 复用 `PersonalDictionary` 与 `phrases.tsv` **源数据**(非 iOS `.bin` 直用)
|
||||||
|
- Provider 可替换;能力矩阵诚实声明(尤其热词模式)
|
||||||
|
- 模型下载可管理(**ModelScope 优先**,HF / GitHub 备用)
|
||||||
|
- 可评测、可灰度、可回退
|
||||||
|
|
||||||
|
### 2.4 非目标
|
||||||
|
|
||||||
|
- 不立即将主路径切到 FunASR Python server 或 Sherpa
|
||||||
|
- 不把 1 万词全量塞入 ASR prompt
|
||||||
|
- 不把 Polish 当作唯一纠错层
|
||||||
|
- 不承诺未 POC 验证的模型效果
|
||||||
|
- 第一期不强制实现 Typeflux 式「自动词库学习」(仅作可选实验设计)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. 当前架构(代码事实)
|
||||||
|
|
||||||
|
### 3.1 端到端数据流
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart LR
|
||||||
|
subgraph macOS["macOS"]
|
||||||
|
Rec["MacAudioRecorder"]
|
||||||
|
Pipe["MacDictationPipeline"]
|
||||||
|
Local["MacLocalASRService"]
|
||||||
|
Cloud["CloudASRClient"]
|
||||||
|
Polish["PolishingService"]
|
||||||
|
Insert["MacTextInsertionService"]
|
||||||
|
end
|
||||||
|
Rec --> Pipe
|
||||||
|
Pipe -->|engineMode local| Local
|
||||||
|
Pipe -->|engineMode cloud| Cloud
|
||||||
|
Local --> Polish
|
||||||
|
Cloud --> Polish
|
||||||
|
Polish --> Insert
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3.2 云路径 vs 本地路径
|
||||||
|
|
||||||
|
| 环节 | 云 ASR | 本地 ASR(当前) |
|
||||||
|
|------|--------|------------------|
|
||||||
|
| 入口 | `MacDictationPipeline.run` | 同左 |
|
||||||
|
| ASR | `CloudASRClientFactory` + `dictionary: store.personalDictionary` | `MacLocalASRService.transcribe(samples, locale)` **无 dictionary** |
|
||||||
|
| 词库 bias | `PersonalDictionary+ASRBias`(按 provider) | **无** |
|
||||||
|
| 润色 | `PolishingService` + `promptFragment()` | 同左(仅用词典做 polish,不经 ASR) |
|
||||||
|
| 默认模型 | 用户所选云 provider | Qwen3 MLX 1.7B;缺权重 → Apple Speech |
|
||||||
|
|
||||||
|
关键代码:
|
||||||
|
|
||||||
|
- [`OSGKeyboardMac/MacDictationPipeline.swift`](../OSGKeyboardMac/MacDictationPipeline.swift) — 本地分支未传 `personalDictionary`
|
||||||
|
- [`OSGKeyboardMac/MacLocalASRService.swift`](../OSGKeyboardMac/MacLocalASRService.swift) — Qwen3 / Apple Speech 二选一
|
||||||
|
- [`OSGKeyboardShared/Models/PersonalDictionary+ASRBias.swift`](../OSGKeyboardShared/Models/PersonalDictionary+ASRBias.swift) — 云侧 `asrHotwords` / `asrPromptBias` / 阿里热词表
|
||||||
|
- [`OSGKeyboardShared/Services/PolishingService.swift`](../OSGKeyboardShared/Services/PolishingService.swift) — 润色层消费 `dictionary.promptFragment()`
|
||||||
|
|
||||||
|
### 3.3 iOS 词库资产(macOS 不可直接复用)
|
||||||
|
|
||||||
|
- **内置词库**:[`OSGKeyboard/Resources/CustomLanguageModel/v1/phrases.tsv`](../OSGKeyboard/Resources/CustomLanguageModel/v1/phrases.tsv)(约 10,301 行,含 `word` / `pinyin` / `source` / `weight`)
|
||||||
|
- **Apple CLM**:[`OSGKeyboardShared/Services/CustomLanguageModelManager.swift`](../OSGKeyboardShared/Services/CustomLanguageModelManager.swift) → `SFSpeechLanguageModel.Configuration`,仅 iOS 26+ 中文本地 ASR 路径
|
||||||
|
- **结论**:macOS 需从 TSV + `PersonalDictionary` **重新适配**为 Qwen3 prompt / Sherpa hotwords / polish fragment,不能加载 `OSGKeyboardCLM.bin`
|
||||||
|
|
||||||
|
### 3.4 本地模型现状
|
||||||
|
|
||||||
|
- 默认路径:`~/Library/Application Support/OSGKeyboard/models/qwen3-asr-1.7b-mlx`
|
||||||
|
- 校验:`config.json`、`model.safetensors`、`vocab.json`、`merges.txt`
|
||||||
|
- Qwen3 引擎:[`MacQwen3LocalASR.swift`](../OSGKeyboardMac/MacQwen3LocalASR.swift) / [`MacQwen3ASREngine.swift`](../OSGKeyboardMac/MacQwen3ASREngine.swift) — **无 prompt / hotword 入参**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. 开源竞品源码观察
|
||||||
|
|
||||||
|
基于 GitHub topic `typeless-alternative` 及关联仓库(2026-03 快照)。
|
||||||
|
|
||||||
|
### 4.1 对比总表
|
||||||
|
|
||||||
|
| 项目 | 技术栈 | 本地 ASR | 热词 / 词库 | 可借鉴 | 不宜照搬 |
|
||||||
|
|------|--------|----------|-------------|--------|----------|
|
||||||
|
| **OpenLess** | Tauri/Rust | Qwen3 C 引擎、Apple Speech、Sherpa(Win) | 火山 `context.hotwords`;Whisper prompt;Polish hotword block;**本地 Qwen3 未接词典** | 多 provider + polish 双层 | 本地热词叙事过度乐观 |
|
||||||
|
| **Typeflux** | Swift/macOS | SenseVoice、FunASR、Qwen3 Sherpa CLI、WhisperKit | `VocabularyStore` cap 500;Doubao hotwords;Whisper prompt;**Sherpa 本地无热词**;自动项目词 + 编辑后学习 | Swift 原生、词库排序、自动学习(作实验) | Sherpa 仅 CLI 离线,无 hotwords 接线 |
|
||||||
|
| **SayIt** | Tauri + FastAPI | sherpa-onnx Rust:**Qwen3 recognizer 创建时写 hotwords** | 内置主题词 + 自定义;云 Qwen corpus;server `hotwords.txt` | **本地 Qwen3 hard hotwords 实证**;跨 provider `StartOptions.hotwords` | 服务端 vLLM 非 macOS 客户端主线 |
|
||||||
|
| **VoiceSnap** | Go/Wails | SenseVoice + sherpa-onnx | **无个性化词库** | 离线体验:静音截断、剪贴板保护、填充词过滤 | 无万级词库场景 |
|
||||||
|
| **OpenBroca** | Electron | Sherpa 等 | Dictionary hotword/replacement | **one-shot `recognize` first**;model catalog、sha256、selected model | Electron 栈 |
|
||||||
|
|
||||||
|
### 4.2 对 OSG 的启示
|
||||||
|
|
||||||
|
1. **学架构,不学「本地已完整支持热词」的 README 叙事**(OpenLess 本地 Qwen3 缺口与 OSG 类似)。
|
||||||
|
2. **SayIt 证明**:sherpa-onnx `OfflineQwen3ASRModelConfig.hotwords` 可在 recognizer 创建时注入;热词变化需 **重建 recognizer**(缓存 key 含 hotwords 字符串)。
|
||||||
|
3. **Typeflux 证明**:`activeTerms()` 上限 500 + 动态排序;但 Sherpa Qwen3/SenseVoice 命令行路径**未传** vocabulary prompt。
|
||||||
|
4. **OpenBroca 证明**:runtime 不得静默选「目录里第一个模型」;须 `selectedModelId` + manifest。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. 目标架构
|
||||||
|
|
||||||
|
### 5.1 管道总览
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart TD
|
||||||
|
audioSamples["Audio Samples 16kHz"] --> macPipeline["MacDictationPipeline"]
|
||||||
|
personalDict["PersonalDictionary"] --> biasAdapter["LocalASRBiasAdapter"]
|
||||||
|
phrasesTSV["phrases.tsv Index"] --> biasAdapter
|
||||||
|
runtimeCtx["Runtime Context locale app recentHits"] --> biasAdapter
|
||||||
|
providerCap["Provider Capability"] --> biasAdapter
|
||||||
|
|
||||||
|
biasAdapter --> hotwords["hardHotwords"]
|
||||||
|
biasAdapter --> promptBias["promptBias"]
|
||||||
|
biasAdapter --> corpusText["corpusContext"]
|
||||||
|
biasAdapter --> correctionPairs["correctionPairs"]
|
||||||
|
biasAdapter --> polishFrag["polishFragment"]
|
||||||
|
biasAdapter --> diag["diagnostics"]
|
||||||
|
|
||||||
|
macPipeline --> recognize["LocalASRProvider.recognize"]
|
||||||
|
hotwords --> recognize
|
||||||
|
promptBias --> recognize
|
||||||
|
corpusText --> recognize
|
||||||
|
|
||||||
|
recognize --> rawTranscript["Raw Transcript"]
|
||||||
|
rawTranscript --> correctionLayer["Correction Layer"]
|
||||||
|
correctionPairs --> correctionLayer
|
||||||
|
correctionLayer --> polishingService["PolishingService"]
|
||||||
|
polishFrag --> polishingService
|
||||||
|
polishingService --> finalText["Final Text"]
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5.2 三层职责边界
|
||||||
|
|
||||||
|
| 层 | 职责 | 禁止 |
|
||||||
|
|----|------|------|
|
||||||
|
| **ASR bias** | 提高听写阶段专有名词概率 | 承担全文语法润色 |
|
||||||
|
| **Correction** | 高置信 `aliases → term` 替换 | 凭热词表改写普通句意 |
|
||||||
|
| **Polish** | 标点、口语转书面、热词保真 | 单独承担全部专名纠错 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Local ASR Provider 抽象
|
||||||
|
|
||||||
|
### 6.1 One-shot first
|
||||||
|
|
||||||
|
借鉴 OpenBroca:macOS 听写主路径为 **录完后一次性 `recognize`**;流式预览(`transcribe` / partial)为可选能力,非第一期必做。
|
||||||
|
|
||||||
|
建议协议(概念层):
|
||||||
|
|
||||||
|
```swift
|
||||||
|
protocol LocalASRProvider {
|
||||||
|
var capabilities: LocalASRCapabilities { get }
|
||||||
|
func recognize(
|
||||||
|
samples: [Float],
|
||||||
|
sampleRate: Int,
|
||||||
|
locale: Locale,
|
||||||
|
bias: LocalASRBiasPayload?,
|
||||||
|
options: LocalASRRecognizeOptions?
|
||||||
|
) async throws -> LocalASRResult
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 6.2 能力矩阵(须诚实声明)
|
||||||
|
|
||||||
|
除 `supportsStreaming`、`maxHotwordCount` 外,**必须区分热词模式**:
|
||||||
|
|
||||||
|
| 字段 | 含义 |
|
||||||
|
|------|------|
|
||||||
|
| `hotwordMode` | `none` / `promptOnly` / `perRequest` / `recognizerScoped` / `cloudVocabulary` |
|
||||||
|
| `hotwordStrength` | `weak` / `medium` / `strong`(产品文案用,非科学绝对值) |
|
||||||
|
| `hotwordReloadCost` | `none` / `recognizerReload` / `modelReload` |
|
||||||
|
| `maxPromptCharacters` | soft prompt 上限 |
|
||||||
|
| `maxHotwordCount` | hard hotwords 上限 |
|
||||||
|
| `supportsLanguageHint` | 是否接受 locale → language hint |
|
||||||
|
|
||||||
|
### 6.3 各 Backend 定位(规划)
|
||||||
|
|
||||||
|
| Provider | 角色 | hotwordMode(规划) | 备注 |
|
||||||
|
|----------|------|---------------------|------|
|
||||||
|
| **Qwen3 MLX** | 短期主线 | `promptOnly`(待接 `qwen_set_prompt` 等价 API) | 已有权重路径;改动面最小 |
|
||||||
|
| **Sherpa Qwen3** | 中期 POC | `recognizerScoped` | SayIt 同款;热词变更加载成本 |
|
||||||
|
| **SenseVoice** | 对照 | `none` 或弱 prompt | 速度/中文基线;非热词主线 |
|
||||||
|
| **Apple Speech** | Fallback | `none`(macOS CLM 待验证) | 系统稳定 |
|
||||||
|
| **Cloud ASR** | 质量上限 | 各云 `PersonalDictionary+ASRBias` | 非离线 |
|
||||||
|
|
||||||
|
**暂不主推**:Sherpa Paraformer 作为热词主线(官方不支持 Paraformer hotwords,与 transducer/Qwen3 不同)。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. LocalASRBiasAdapter 设计
|
||||||
|
|
||||||
|
### 7.1 输入
|
||||||
|
|
||||||
|
| 输入 | 说明 |
|
||||||
|
|------|------|
|
||||||
|
| `PersonalDictionary.effectiveEntries` | 用户词;最高优先级 |
|
||||||
|
| `BuiltinLexiconIndex` | 自 `phrases.tsv` 构建;按 weight / 场景筛选 |
|
||||||
|
| `locale` | `store.localeId` |
|
||||||
|
| `frontAppBundleId` | 可选;技术类 App 提升 IT 子集权重 |
|
||||||
|
| `recentHitTerms` | 历史命中统计(若已有) |
|
||||||
|
| `providerCapabilities` | 决定输出哪些字段、如何截断 |
|
||||||
|
|
||||||
|
### 7.2 输出 `LocalASRBiasPayload`
|
||||||
|
|
||||||
|
```swift
|
||||||
|
struct LocalASRBiasPayload {
|
||||||
|
var hardHotwords: [String] // Sherpa Qwen3、部分云 API
|
||||||
|
var promptBias: String? // Qwen3 MLX、Whisper 系
|
||||||
|
var corpusContext: String? // Qwen 云 corpus 风格(若将来统一)
|
||||||
|
var polishFragment: String // PolishingService 追加块
|
||||||
|
var correctionPairs: [(alias: String, term: String)]
|
||||||
|
var diagnostics: BiasDiagnostics // 供设置页 / 调试
|
||||||
|
}
|
||||||
|
|
||||||
|
struct BiasDiagnostics {
|
||||||
|
var userTermCount: Int
|
||||||
|
var builtinTermCount: Int
|
||||||
|
var truncated: Bool
|
||||||
|
var truncationReason: String?
|
||||||
|
var selectedSources: [String] // e.g. user, builtin-it, builtin-top
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 7.3 优先级与截断
|
||||||
|
|
||||||
|
```
|
||||||
|
用户高频 / 最近命中
|
||||||
|
> PersonalDictionary(全部有效 term)
|
||||||
|
> 当前 App 相关内置词(phrases 子集)
|
||||||
|
> 高 weight 内置技术词(Top-N)
|
||||||
|
> 其余内置词(仅 polish / 检索,不进 ASR)
|
||||||
|
```
|
||||||
|
|
||||||
|
默认建议(可 POC 调参):
|
||||||
|
|
||||||
|
| 输出 | 默认上限 |
|
||||||
|
|------|----------|
|
||||||
|
| `hardHotwords` | 100(Qwen3 Sherpa);对齐 `asrHotwords(maxCount: 100)` |
|
||||||
|
| `promptBias` | 800 字符;复用 `asrPromptBias(maxCharacters:)` 逻辑 |
|
||||||
|
| ASR 层内置词 | 200–500;**不全量 1 万** |
|
||||||
|
| `correctionPairs` | aliases 全量可进后处理,但仅 **整词 / 高置信** 替换 |
|
||||||
|
|
||||||
|
### 7.4 防污染规则
|
||||||
|
|
||||||
|
- 近静音、极短音频:减少或跳过内置词,保留用户词。
|
||||||
|
- 用户词始终优先于内置词。
|
||||||
|
- diagnostics 必须记录「为何丢弃」某批词(超 cap、provider 不支持、场景不匹配)。
|
||||||
|
|
||||||
|
### 7.5 与现有云代码复用
|
||||||
|
|
||||||
|
扩展 [`PersonalDictionary+ASRBias.swift`](../OSGKeyboardShared/Models/PersonalDictionary+ASRBias.swift) 为 **单一事实来源**,新增例如:
|
||||||
|
|
||||||
|
- `localPromptBias(maxCharacters:builtinTerms:)`
|
||||||
|
- `correctionPairs()`
|
||||||
|
- `rankedTermsForASR(limit:builtinBoost:)`
|
||||||
|
|
||||||
|
避免 macOS / iOS / Cloud 三套独立拼接逻辑。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. 词库策略
|
||||||
|
|
||||||
|
### 8.1 PersonalDictionary
|
||||||
|
|
||||||
|
| 字段 | ASR | Correction | Polish |
|
||||||
|
|------|-----|------------|--------|
|
||||||
|
| `term` | hotword / prompt | 标准写法 | 必须保留 |
|
||||||
|
| `aliases` | 可进 prompt 提示 | **主战场** | 语义纠错参考 |
|
||||||
|
| `category` | 排序权重 | — | 分组展示 |
|
||||||
|
| `usageCount` | 排序权重 | — | — |
|
||||||
|
|
||||||
|
iCloud:[`PersonalDictionaryCloudSync`](../OSGKeyboardShared/Services/PersonalDictionaryCloudSync/) 保证 Mac / iOS / Extension 一致;本地 ASR 只读 `AppGroupStore.personalDictionary`。
|
||||||
|
|
||||||
|
### 8.2 phrases.tsv 分层
|
||||||
|
|
||||||
|
**不全量进入 ASR prompt。**
|
||||||
|
|
||||||
|
| 层级 | 用途 | 规模建议 |
|
||||||
|
|------|------|----------|
|
||||||
|
| L1 ASR 高价值 | `weight >= 4` 或 curated IT 品牌缩写 | 200–500 |
|
||||||
|
| L2 场景相关 | 按 `frontApp` / 用户最近命中动态加入 | +0–100 |
|
||||||
|
| L3 全量索引 | 后处理模糊匹配、polish 检索 | ~10k |
|
||||||
|
|
||||||
|
TSV 列:`word`、`pinyin`、`source`、`weight` — 构建索引时保留 weight 用于排序。
|
||||||
|
|
||||||
|
### 8.3 iOS CLM 与 macOS 关系
|
||||||
|
|
||||||
|
- iOS:TSV → export script → `.bin` → `CustomLanguageModelManager`
|
||||||
|
- macOS:TSV → `BuiltinLexiconIndex` → `LocalASRBiasAdapter` → Qwen3 / Sherpa / Polish
|
||||||
|
- **同一 TSV 源**,两种消费格式;不尝试把 `.bin` 喂给 Sherpa/MLX
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. 自动词库学习(可选实验,非第一期)
|
||||||
|
|
||||||
|
借鉴 Typeflux `WorkflowController+AutomaticVocabulary`:
|
||||||
|
|
||||||
|
- 听写插入后,短时观察用户在前台可编辑框内的修改
|
||||||
|
- LLM 或规则判断是否为「专名 / 品牌 / 大小写修正」
|
||||||
|
- 候选进入 **待确认队列**,不直接写入 `PersonalDictionary`
|
||||||
|
|
||||||
|
**约束(必须写进隐私说明)**:
|
||||||
|
|
||||||
|
- 默认关闭
|
||||||
|
- 不自动 iCloud 同步待确认项
|
||||||
|
- 可一键清空、可审计来源
|
||||||
|
- 拒绝:整句改写、纯语法修正、过短词条、编辑幅度过大
|
||||||
|
|
||||||
|
OSG 已有 `PersonalDictionary.Entry.Source.recentEdit` 与合并逻辑,可与之对齐而非新建平行存储。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. 本地模型管理
|
||||||
|
|
||||||
|
### 10.1 原则
|
||||||
|
|
||||||
|
- **Catalog 与 Runtime 分离**:下载源只影响安装;推理只读本地 **已验证 manifest**
|
||||||
|
- **Selected model 显式**:禁止「扫描目录用第一个 onnx」
|
||||||
|
- **完整性**:sha256 或 size 校验 + staging 目录原子发布
|
||||||
|
|
||||||
|
### 10.2 存储布局(建议)
|
||||||
|
|
||||||
|
```
|
||||||
|
~/Library/Application Support/OSGKeyboard/
|
||||||
|
LocalASRModels/
|
||||||
|
manifest.json # 已安装模型、版本、backend、capabilities
|
||||||
|
qwen3-mlx-1.7b/ # 当前 MLX 布局(可与现路径兼容)
|
||||||
|
sherpa-qwen3-0.6b/
|
||||||
|
sherpa-sensevoice-small/
|
||||||
|
```
|
||||||
|
|
||||||
|
### 10.3 Catalog 条目(概念)
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"modelId": "sherpa-qwen3-asr-0.6b-int8",
|
||||||
|
"displayName": "Qwen3-ASR 0.6B (Sherpa)",
|
||||||
|
"backend": "sherpaQwen3",
|
||||||
|
"sizeBytes": 1200000000,
|
||||||
|
"recommendedLocales": ["zh-CN", "en-US"],
|
||||||
|
"supportsHotwords": true,
|
||||||
|
"hotwordMode": "recognizerScoped",
|
||||||
|
"sources": [
|
||||||
|
{
|
||||||
|
"type": "modelscope",
|
||||||
|
"url": "https://www.modelscope.cn/api/v1/models/.../repo?Revision=master&FilePath=...",
|
||||||
|
"sha256": "...",
|
||||||
|
"priority": 1
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "huggingface",
|
||||||
|
"url": "https://huggingface.co/...",
|
||||||
|
"priority": 2
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"type": "github",
|
||||||
|
"url": "https://github.com/k2-fsa/sherpa-onnx/releases/download/...",
|
||||||
|
"priority": 3
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 10.4 ModelScope 策略
|
||||||
|
|
||||||
|
| 场景 | 策略 |
|
||||||
|
|------|------|
|
||||||
|
| 中国大陆用户默认 | **ModelScope 优先**(Qwen3-ASR、SenseVoice、FunASR 相关 ONNX) |
|
||||||
|
| 国际 / ModelScope 失败 | Hugging Face → GitHub Releases |
|
||||||
|
| 企业内网 | `custom` mirror URL(用户配置) |
|
||||||
|
| 安装流程 | download → verify → extract → validate required files → rename staging → update manifest |
|
||||||
|
| 失败 | 清理 staging / 临时文件;不留下半安装状态 |
|
||||||
|
|
||||||
|
MLX Qwen3 权重:可继续支持用户自选目录(现状),逐步纳入统一 catalog 的 `type: mlx` 条目。
|
||||||
|
|
||||||
|
### 10.5 UI / 设置需求(规划)
|
||||||
|
|
||||||
|
- 模型列表:体积、语言、安装状态、是否支持热词
|
||||||
|
- 下载进度:phase(downloading / extracting / validating / finalizing)
|
||||||
|
- 切换模型:仅允许 **installed + manifest 合法** 的项为默认
|
||||||
|
- 诊断:当前 provider、capability、上次 bias diagnostics
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 11. 后端对比与决策矩阵
|
||||||
|
|
||||||
|
### 11.1 产品分层
|
||||||
|
|
||||||
|
```text
|
||||||
|
主线: Qwen3 MLX + LocalASRBiasAdapter + Polish/Correction
|
||||||
|
重点 POC: Sherpa Qwen3 hotwords
|
||||||
|
对照: SenseVoice(速度)、Apple Speech(fallback)
|
||||||
|
参考上限: Cloud ASR + PersonalDictionary
|
||||||
|
团队部署: Qwen3-ASR vLLM(SayIt 式,非客户端主线)
|
||||||
|
暂不主推: FunASR Paraformer hotwords、纯 OpenLess 本地词典叙事
|
||||||
|
```
|
||||||
|
|
||||||
|
### 11.2 详细对比
|
||||||
|
|
||||||
|
| 维度 | Qwen3 MLX | Sherpa Qwen3 | SenseVoice | Apple Speech | Cloud |
|
||||||
|
|------|-----------|--------------|------------|--------------|-------|
|
||||||
|
| 离线 | ✅ | ✅ | ✅ | ✅ | ❌ |
|
||||||
|
| 中英混合技术词 | 强(经验性) | 待 POC | 中 | 中 | 强 |
|
||||||
|
| Hard hotwords | ❌→prompt | ✅ recognizerScoped | ❌ | ❌ | ✅ 因 provider 异 |
|
||||||
|
| 实现成本 | 低(已有) | 高(runtime 体积) | 中 | 低 | 已有 |
|
||||||
|
| 模型体积 | ~1.3GB+ | 类似 | ~350MB 级 | 0 | N/A |
|
||||||
|
| 隐私 | 本地 | 本地 | 本地 | 本地 | 依配置 |
|
||||||
|
|
||||||
|
### 11.3 Sherpa POC 通过阈值(建议)
|
||||||
|
|
||||||
|
相对 **当前 Qwen3 MLX + 仅 polish** 基线:
|
||||||
|
|
||||||
|
| 指标 | 建议阈值 |
|
||||||
|
|------|----------|
|
||||||
|
| 用户热词召回率 | 提升 ≥ 20% |
|
||||||
|
| 误触发率(未说热词却被改成热词) | ≤ 2% |
|
||||||
|
| 30s 音频端到端延迟 | ≤ 基线 × 1.5 |
|
||||||
|
| 内存峰值(8GB Mac 目标机) | 可接受且无 OOM |
|
||||||
|
| 安装成功率 | 普通用户可完成 ModelScope/HF 下载 |
|
||||||
|
|
||||||
|
未达阈值:**保留 Qwen3 MLX 主线**,Sherpa 仅作高级选项。
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 12. POC 评测计划
|
||||||
|
|
||||||
|
### 12.1 测试集
|
||||||
|
|
||||||
|
| 类别 | 内容 | 目的 |
|
||||||
|
|------|------|------|
|
||||||
|
| A 普通中文 | 日常口语 50 句 | 基线 WER / 误触发 |
|
||||||
|
| B 技术术语 | SwiftUI、Cursor、Qwen3-ASR 等 50 句 | 专名召回 |
|
||||||
|
| C 用户词典 | 模拟 PersonalDictionary 20 词 × 多句 | 热词核心场景 |
|
||||||
|
| D 长句润色 | 30s+ 口语 | polish 兜底 |
|
||||||
|
| E 噪声 / 短句 | 低 SNR、<2s | 防污染规则 |
|
||||||
|
|
||||||
|
### 12.2 对照矩阵
|
||||||
|
|
||||||
|
| 配置 | 说明 |
|
||||||
|
|------|------|
|
||||||
|
| Baseline | Qwen3 MLX,无 bias |
|
||||||
|
| B1 | Qwen3 MLX + promptBias |
|
||||||
|
| B2 | B1 + polishFragment + correction |
|
||||||
|
| POC1 | Sherpa Qwen3 + hardHotwords |
|
||||||
|
| POC2 | SenseVoice,无 hotwords |
|
||||||
|
| Ref | 云 ASR + PersonalDictionary |
|
||||||
|
|
||||||
|
### 12.3 指标
|
||||||
|
|
||||||
|
- Raw CER/WER(中文可用字错误率)
|
||||||
|
- **Hotword recall**(用户词是否出现在 raw 或 final)
|
||||||
|
- **False hotword rate**
|
||||||
|
- Final accuracy(用户主观或编辑距离)
|
||||||
|
- Latency:record end → text inserted
|
||||||
|
- Memory / CPU、模型加载时间
|
||||||
|
- 离线可靠性(无网络完成全流程)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 13. 失败回退策略
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart TD
|
||||||
|
start["recognize 开始"] --> qwen{"Qwen3 MLX 可用?"}
|
||||||
|
qwen -->|是| qwenRun["Qwen3 + bias"]
|
||||||
|
qwen -->|否| apple["Apple Speech"]
|
||||||
|
qwenRun -->|失败| apple
|
||||||
|
qwenRun -->|成功| post["Correction + Polish"]
|
||||||
|
apple --> post
|
||||||
|
post -->|Polish 失败| raw["返回 raw transcript"]
|
||||||
|
post -->|成功| done["插入 final"]
|
||||||
|
```
|
||||||
|
|
||||||
|
| 条件 | 行为 |
|
||||||
|
|------|------|
|
||||||
|
| Qwen3 权重缺失 | Apple Speech(现状) |
|
||||||
|
| Qwen3 推理失败 | 可配置:重试一次 → Apple Speech |
|
||||||
|
| Sherpa 未安装 | 不回退云;提示下载 |
|
||||||
|
| 模型 manifest 损坏 | 标记 invalid,禁止设为默认 |
|
||||||
|
| Polish 失败 | 使用 raw(现状) |
|
||||||
|
| 用户禁用云 | 不静默切云 |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 14. 分阶段落地路线
|
||||||
|
|
||||||
|
| Phase | 内容 | 交付物 |
|
||||||
|
|-------|------|--------|
|
||||||
|
| **1** | 本文档定稿;`LocalASRCapabilities` + `LocalASRBiasPayload` 类型设计 | 架构文档 + ADR 可选 |
|
||||||
|
| **2** | `MacDictationPipeline` 接入 adapter;Qwen3 MLX `promptBias`;polish + correction | 实现 PR |
|
||||||
|
| **3** | `BuiltinLexiconIndex`;Top-N;diagnostics UI | 实现 PR |
|
||||||
|
| **4** | Model catalog + ModelScope 下载 + manifest | 实现 PR |
|
||||||
|
| **5** | Sherpa Qwen3 POC + 评测报告 | 决策是否默认切换 |
|
||||||
|
| **6** | 可选:自动词库学习实验(默认关) | 功能 flag |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 15. 风险与待确认问题
|
||||||
|
|
||||||
|
| 风险 | 缓解 |
|
||||||
|
|------|------|
|
||||||
|
| Qwen3 prompt bias 过弱 | POC 对比 Sherpa hard hotwords;保留 correction + polish |
|
||||||
|
| 热词过多污染识别 | cap + 场景筛选 + diagnostics |
|
||||||
|
| aliases 后处理误改 | 整词边界、低置信跳过 |
|
||||||
|
| Sherpa 分发体积 / 签名 / 公证 | 单独评估;可选按需下载 |
|
||||||
|
| Apple Speech macOS CLM | 调研 macOS 26+ 是否可接 CLM;否则仅 fallback |
|
||||||
|
| ModelScope API 变更 | 多 mirror;manifest 可更新 URL |
|
||||||
|
| 自动学习隐私 | 默认关、本地、待确认 |
|
||||||
|
| 低配 Mac 内存 | 单模型常驻策略;SenseVoice 作轻量选项 |
|
||||||
|
|
||||||
|
**待确认**:
|
||||||
|
|
||||||
|
1. MLX Swift API 是否暴露等价 `setPrompt`(对标 Open-Less/qwen-asr `qwen_set_prompt`)
|
||||||
|
2. Sherpa-ONNX Swift/SPM 与 App Store 公证路径
|
||||||
|
3. `phrases.tsv` Top-N 是否按 `weight` 静态裁剪即可,或需按 App 动态检索
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 16. 相关代码索引
|
||||||
|
|
||||||
|
| 主题 | 路径 |
|
||||||
|
|------|------|
|
||||||
|
| Mac 听写管道 | `OSGKeyboardMac/MacDictationPipeline.swift` |
|
||||||
|
| 本地 ASR 入口 | `OSGKeyboardMac/MacLocalASRService.swift` |
|
||||||
|
| Qwen3 MLX | `OSGKeyboardMac/MacQwen3LocalASR.swift`, `MacQwen3ASREngine.swift` |
|
||||||
|
| Apple Speech fallback | `OSGKeyboardMac/MacSpeechLocalASR.swift` |
|
||||||
|
| 用户词库 | `OSGKeyboardShared/Models/PersonalDictionary.swift` |
|
||||||
|
| 云 bias | `OSGKeyboardShared/Models/PersonalDictionary+ASRBias.swift` |
|
||||||
|
| 内置 TSV | `OSGKeyboard/Resources/CustomLanguageModel/v1/phrases.tsv` |
|
||||||
|
| iOS CLM | `OSGKeyboardShared/Services/CustomLanguageModelManager.swift` |
|
||||||
|
| 润色 | `OSGKeyboardShared/Services/PolishingService.swift` |
|
||||||
|
| 文本插入 | `OSGKeyboardMac/MacTextInsertionService.swift` |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 17. 修订记录
|
||||||
|
|
||||||
|
| 日期 | 说明 |
|
||||||
|
|------|------|
|
||||||
|
| 2026-03-31 | 初版:基于 OSG 代码审计 + typeless-alternative 竞品源码 + 计划评审(ModelScope、hotwordMode、回退策略) |
|
||||||
+17
-3
@@ -50,8 +50,8 @@ settings:
|
|||||||
ASSETCATALOG_COMPILER_GENERATE_SWIFT_ASSET_SYMBOL_EXTENSIONS: YES
|
ASSETCATALOG_COMPILER_GENERATE_SWIFT_ASSET_SYMBOL_EXTENSIONS: YES
|
||||||
STRING_CATALOG_GENERATE_SYMBOLS: YES
|
STRING_CATALOG_GENERATE_SYMBOLS: YES
|
||||||
CLANG_CXX_LANGUAGE_STANDARD: c++17
|
CLANG_CXX_LANGUAGE_STANDARD: c++17
|
||||||
MARKETING_VERSION: "0.5.0"
|
MARKETING_VERSION: "0.5.2"
|
||||||
CURRENT_PROJECT_VERSION: "18"
|
CURRENT_PROJECT_VERSION: "19"
|
||||||
# 签名配置来自 Signing.local.xcconfig(gitignored,不会被覆盖)
|
# 签名配置来自 Signing.local.xcconfig(gitignored,不会被覆盖)
|
||||||
|
|
||||||
# 项目级签名 xcconfig,适用于所有 target
|
# 项目级签名 xcconfig,适用于所有 target
|
||||||
@@ -327,6 +327,7 @@ targets:
|
|||||||
resources:
|
resources:
|
||||||
- path: OSGKeyboardShared/Resources/CustomLanguageModel/v1/OSGKeyboardCLM.bin
|
- path: OSGKeyboardShared/Resources/CustomLanguageModel/v1/OSGKeyboardCLM.bin
|
||||||
- path: OSGKeyboardShared/Resources/CustomLanguageModel/v1/compiled-manifest.json
|
- path: OSGKeyboardShared/Resources/CustomLanguageModel/v1/compiled-manifest.json
|
||||||
|
- path: OSGKeyboardShared/Resources/LocalASR/local-asr-catalog.json
|
||||||
info:
|
info:
|
||||||
path: OSGKeyboardShared/Info.plist
|
path: OSGKeyboardShared/Info.plist
|
||||||
settings:
|
settings:
|
||||||
@@ -429,10 +430,18 @@ targets:
|
|||||||
buildPhase: resources
|
buildPhase: resources
|
||||||
- path: OSGKeyboardShared/zh-Hans.lproj/Shared.strings
|
- path: OSGKeyboardShared/zh-Hans.lproj/Shared.strings
|
||||||
buildPhase: resources
|
buildPhase: resources
|
||||||
|
- path: OSGKeyboard/Resources/CustomLanguageModel/v1/phrases.tsv
|
||||||
|
buildPhase: resources
|
||||||
|
- path: OSGKeyboardShared/Resources/LocalASR/local-asr-catalog.json
|
||||||
|
buildPhase: resources
|
||||||
entitlements:
|
entitlements:
|
||||||
path: OSGKeyboardMac/OSGKeyboardMac.entitlements
|
path: OSGKeyboardMac/OSGKeyboardMac.entitlements
|
||||||
properties:
|
properties:
|
||||||
com.apple.security.app-sandbox: true
|
# Sandbox disabled on purpose: this menu-bar dictation utility uses the
|
||||||
|
# Accessibility API to monitor a global hotkey and inject ⌘V into other
|
||||||
|
# apps — both are forbidden by App Sandbox. Ship via Developer ID +
|
||||||
|
# notarization (not Mac App Store / TestFlight, which force the sandbox).
|
||||||
|
com.apple.security.app-sandbox: false
|
||||||
com.apple.security.device.audio-input: true
|
com.apple.security.device.audio-input: true
|
||||||
com.apple.security.network.client: true
|
com.apple.security.network.client: true
|
||||||
com.apple.developer.ubiquity-kvstore-identifier: $(TeamIdentifierPrefix)com.osgkeyboard.ios
|
com.apple.developer.ubiquity-kvstore-identifier: $(TeamIdentifierPrefix)com.osgkeyboard.ios
|
||||||
@@ -455,6 +464,9 @@ targets:
|
|||||||
ITSAppUsesNonExemptEncryption: false
|
ITSAppUsesNonExemptEncryption: false
|
||||||
settings:
|
settings:
|
||||||
base:
|
base:
|
||||||
|
PRODUCT_NAME: OSGKeyboard
|
||||||
|
INFOPLIST_KEY_CFBundleDisplayName: OSGKeyboard
|
||||||
|
INFOPLIST_KEY_CFBundleName: OSGKeyboard
|
||||||
PRODUCT_BUNDLE_IDENTIFIER: com.osgkeyboard.mac
|
PRODUCT_BUNDLE_IDENTIFIER: com.osgkeyboard.mac
|
||||||
MACOSX_DEPLOYMENT_TARGET: "15.0"
|
MACOSX_DEPLOYMENT_TARGET: "15.0"
|
||||||
ASSETCATALOG_COMPILER_APPICON_NAME: AppIcon
|
ASSETCATALOG_COMPILER_APPICON_NAME: AppIcon
|
||||||
@@ -462,6 +474,8 @@ targets:
|
|||||||
APPLICATION_EXTENSION_API_ONLY: NO
|
APPLICATION_EXTENSION_API_ONLY: NO
|
||||||
CODE_SIGN_STYLE: Automatic
|
CODE_SIGN_STYLE: Automatic
|
||||||
DEVELOPMENT_TEAM: X329MZU23S
|
DEVELOPMENT_TEAM: X329MZU23S
|
||||||
|
# Required for Developer ID distribution + notarization (outside App Store).
|
||||||
|
ENABLE_HARDENED_RUNTIME: YES
|
||||||
dependencies:
|
dependencies:
|
||||||
- sdk: Speech.framework
|
- sdk: Speech.framework
|
||||||
- sdk: AVFoundation.framework
|
- sdk: AVFoundation.framework
|
||||||
|
|||||||
Reference in New Issue
Block a user