rocky/OSGKeyboard
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor: drop Qwen3 CoreML ASR, add local-engine cloud polish toggle

Browse Source 
Rolls back the v0.2.0 Qwen3 CoreML on-device ASR stack and replaces the
'local engine' UX with iOS 26 SpeechAnalyzer + DictationTranscriber only.

The 'Cloud polish after ASR' toggle (ProviderConfig.localModeCloudPolishEnabled)
lets users opt into a post-ASR DeepSeek round-trip from the local engine.
Defaults to off so the local engine stays genuinely local. New PolishError.missingAPIError
surfaces an inline 'fill in your key' warning when the toggle is on but the
Keychain is empty. DeepSeek preset default model bumped to deepseek-v4-flash.

Deleted:
  - OSGKeyboard/ThirdParty/Qwen3Speech/ (74 files, ~16k LoC)
  - OSGKeyboard/Services/ModelManager.swift (492)
  - OSGKeyboard/Services/OnDeviceModelWarmup.swift (197)
  - OSGKeyboard/Services/Qwen3ASRService.swift (257)
  - OSGKeyboard/Services/ModelDownloadSourcePicker.swift (126)
  - OSGKeyboard/Views/OnDeviceModelsView.swift (184)
  - OSGKeyboard/Views/DownloadConfirmSheet.swift (96)
  - OSGKeyboardShared/Models/OnDeviceModel.swift (140)
  - OSGKeyboardShared/Services/OnDeviceModelStatus.swift (104)
  - Qwen3ASRServiceProvider registration in OSGKeyboardApp
  - Qwen3Speech package declaration in project.yml
  - 5 .qwen3ASR enum / branch reference sites in HomeView, OnboardingView,
    LocalEngineSettingsRows, FlowSessionManager, ASRService, EngineServiceLabel
  - Two pre-existing Swift 6 strict-concurrency errors in
    LiveDictationController + FlowSessionManager (the weak [weak self] in
    detached-task MainActor.run blocks) that were blocking clean builds

Added:
  - LocalModelsGroup: 'Built-in iOS SpeechAnalyzer' badge + 'Cloud polish
    after ASR' Switch toggle
  - PolishingService: honour localModeCloudPolishEnabled; new .missingAPIKey
    error case with localised warning
  - AppGroupStore.localModeCloudPolishEnabled (mirrored into App Group
    so the keyboard extension honours the toggle during live dictation)
  - SettingsView: show provider/api sections when local-mode cloud polish
    is on so the user can paste a DeepSeek key
  - FlowSessionManager: route through PolishingService for local + polish-on
    flow; translate missingAPIKey into a polished warning
  - KeyboardViewController: handle PolishingService.PolishError.missingAPIKey
    in the keyboard-side live polish path
  - CHANGELOG v0.2.1: documents the rollback + new toggle
  - README.md / README.zh.md: engine matrix section, data flow note

Verified: xcodebuild -scheme OSGKeyboard -destination 'generic/platform=iOS Simulator'
build succeeds under SWIFT_STRICT_CONCURRENCY=complete.
This commit is contained in:
Rocky
2026-06-24 01:51:34 +08:00
parent 39690c0a93
commit c07cf4db9f
119 changed files with 401 additions and 18858 deletions
Download Patch File Download Diff File Expand all files Collapse all files
OSGKeyboardShared/Services/ASRService.swift
+10 -46
View File
@@ -116,68 +116,32 @@ public enum ASREvent: Sendable, Equatable {
// MARK: - Factory
public enum ASRServiceFactory {
/// Registry of backend-specific providers. The host app installs
/// a provider for the Qwen3-ASR backend at launch time (the
/// `Qwen3ASRProvider` lives in the app target because linking
/// `Qwen3ASR` pulls in mlx-swift, which the shared framework
/// deliberately stays off to keep `APPLICATION_EXTENSION_API_ONLY`
/// clean). The shared framework always provides a built-in
/// `SpeechAnalyzer` provider; custom providers override it.
/// Returns the on-device ASR backend. As of v0.2.0 the only
/// supported `LocalASRBackend` is iOS 26 `SpeechAnalyzer` +
/// `DictationTranscriber` (always on-device, no asset download),
/// so the factory collapses to a single concrete type. We keep the
/// `localBackend` parameter on the signature so the next non-iOS
/// backend can slot in without touching every call site.
///
/// `nonisolated(unsafe)` because the only writer is
/// `OSGKeyboardApp.init` (single-threaded, runs once at launch).
/// After launch, all callers read the dictionary from any
/// actor.
public nonisolated(unsafe) static var providers: [LocalASRBackend: any ASRServiceProvider] = [
.speechAnalyzer: SpeechAnalyzerProvider()
]
/// Returns the ASR backend chosen by the user. The cloud engine
/// always uses the iOS `SpeechAnalyzer` path — it has the lowest
/// latency and never hits the network, which matches the user's
/// expectation that "ASR" is the local half of the pipeline
/// The cloud engine also routes through `SpeechAnalyzerASR`: the
/// user expectation is that ASR is the local half of the pipeline
/// regardless of where the LLM polish happens.
///
/// For the local engine, we honour `LocalASRBackend`:
/// - `.speechAnalyzer` (default) → on-device iOS pipeline.
/// - `.qwen3ASR` → CoreML-backed Qwen3-ASR via `soniqo/speech-swift` (host app only)
/// (registered by the host app at launch).
public static func make(
engineMode: String,
localBackend: LocalASRBackend = .speechAnalyzer
) -> ASRService {
if engineMode == "local" {
if let provider = providers[localBackend] {
return provider.make()
}
}
return SpeechAnalyzerASR()
SpeechAnalyzerASR()
}
/// Back-compat overload for callers that only ever want the
/// SpeechAnalyzer path. The previous single-backend build used
/// this signature; new code should pass the engine mode explicitly
/// so the user's selection is honoured.
/// so any future non-iOS backend is honoured.
public static func make() -> ASRService {
SpeechAnalyzerASR()
}
}
/// Backend-specific ASR factory. The shared framework ships a default
/// `SpeechAnalyzerProvider`; the host app installs a `Qwen3ASRProvider`
/// at launch time so the Qwen3 backend is wired in only where its
/// large MLX dependency is also linked.
public protocol ASRServiceProvider: Sendable {
var backend: LocalASRBackend { get }
func make() -> ASRService
}
/// Built-in provider for the iOS SpeechAnalyzer path. Always present.
struct SpeechAnalyzerProvider: ASRServiceProvider {
let backend: LocalASRBackend = .speechAnalyzer
func make() -> ASRService { SpeechAnalyzerASR() }
}
// MARK: - PCM format conversion (testable helpers)
//
// Extracted from the audio-thread hot path so the scaling + clipping
OSGKeyboardShared/Services/AppGroupStore.swift
+13
View File
@@ -37,6 +37,8 @@ public struct AppGroupStore: @unchecked Sendable {
static let engineMode = "config.engineMode"
static let localASRBackend = "config.localASRBackend"
static let uiLanguage = "config.uiLanguage"
// v0.2.0: opt-in cloud polish step after local-mode ASR.
static let localModeCloudPolishEnabled = "config.localModeCloudPolishEnabled"
}
// MARK: - Reads
@@ -86,6 +88,17 @@ public struct AppGroupStore: @unchecked Sendable {
return LocalASRBackend(rawValue: raw) ?? .speechAnalyzer
}
/// v0.2.0: whether the local engine should route its transcript
/// through the configured cloud LLM (DeepSeek by default) before
/// insertion. Defaults to `false`; the keyboard extension reads
/// this so Flow sessions honour the toggle.
public var localModeCloudPolishEnabled: Bool {
guard defaults.object(forKey: Key.localModeCloudPolishEnabled) != nil else {
return false
}
return defaults.bool(forKey: Key.localModeCloudPolishEnabled)
}
/// Host-app UI language override (`auto` / `en` / `zh-Hans`).
public var uiLanguage: AppUILanguage {
AppUILanguage.fromStored(defaults.string(forKey: Key.uiLanguage))
OSGKeyboardShared/Services/FlowSessionKeys.swift
+7 -3
View File
@@ -31,18 +31,22 @@ public enum FlowSessionKeys {
/// finishes most chunks during recording; this is a soft deadline before
/// blocking on `asrTask.value` (which waits until the pipeline exits).
public static let localASRWaitTimeout: TimeInterval = 120
public static let localQwen3ASRWaitTimeout: TimeInterval = 180
public static let cloudASRWaitTimeout: TimeInterval = 120
/// Keyboard watchdog after the user stops recording (not utterance max length).
/// Must cover worst-case post-stop backlog: remaining MLX/SpeechAnalyzer chunks
/// Must cover worst-case post-stop backlog: remaining SpeechAnalyzer chunks
/// plus cloud LLM polish (see `PolishingService.effectiveTimeout` cap).
///
/// As of v0.2.0 the local engine uses iOS `SpeechAnalyzer` only, so the
/// previous Qwen3-specific timeout (240 s) collapses into the shared
/// local path. We keep `localASRBackend` on the signature for symmetry
/// with other shared helpers.
public static func keyboardResultTimeout(
engineMode: String,
localASRBackend: LocalASRBackend
) -> TimeInterval {
if engineMode == "local" {
return localASRBackend == .qwen3ASR ? 240 : 180
return 180
}
return 240
}
OSGKeyboardShared/Services/KeyboardState.swift
+8 -3
View File
@@ -77,10 +77,15 @@ public final class KeyboardState: ObservableObject {
/// Mirrored from `ProviderConfig.localASRBackend` for UI display
/// and for `state` consumers that want a single source of truth.
@Published public var localASRBackend: LocalASRBackend = .speechAnalyzer
/// `false` when the local engine needs on-device models that are
/// not yet downloaded (mirrored from App Group by the extension).
/// v0.2.0: kept for source compatibility with the previous Qwen3
/// CoreML local engine. Always `true` now — iOS `SpeechAnalyzer`
/// ships with iOS 26 and has no per-user weights to download or
/// preload. Existing read sites will see `true` and behave the
/// same as the "stack ready" branch did.
@Published public var localModelsReady: Bool = true
/// `true` when host app has preloaded Qwen weights into memory.
/// v0.2.0: kept for source compatibility with the previous Qwen3
/// CoreML local engine. Always `false` now — there are no weights
/// for the host app to preload.
@Published public var localModelsLoaded: Bool = false
/// Convenience shorthand used by the pipeline and views.
OSGKeyboardShared/Services/LiveDictationController.swift
+5 -1
View File
@@ -358,7 +358,11 @@ public final class LiveDictationController: ObservableObject {
controller?.currentPartial = partial
}
}
await MainActor.run {
// Re-bind `controller` inside the `@MainActor` block so the
// weak reference is captured under the right isolation. Swift
// 6 strict concurrency otherwise complains about a
// task-isolated reference escaping into a main-actor closure.
await MainActor.run { [weak controller] in
guard let controller else { return }
switch outcome {
case .success(let success):
OSGKeyboardShared/Services/OnDeviceModelStatus.swift
-104
View File
@@ -1,104 +0,0 @@
// OnDeviceModelStatus.swift
// OSGKeyboard · Shared
//
// Mirrors on-device model download state into the App Group so the
// keyboard extension can show readiness hints without reading the
// host app's Caches directory.
import Foundation
public enum OnDeviceModelStatus {
private enum Key {
static func downloaded(_ model: OnDeviceModel) -> String {
"models.\(model.rawValue).downloaded"
}
static func progress(_ model: OnDeviceModel) -> String {
"models.\(model.rawValue).downloadProgress"
}
static let modelsLoadedInMemory = "models.loadedInMemory"
}
// MARK: - Writes (host app)
public static func setDownloaded(_ downloaded: Bool, for model: OnDeviceModel) {
guard AppGroup.isAvailable else { return }
AppGroup.defaults.set(downloaded, forKey: Key.downloaded(model))
if downloaded {
clearProgress(for: model)
}
}
public static func setProgress(_ progress: Double?, for model: OnDeviceModel) {
guard AppGroup.isAvailable else { return }
if let progress {
AppGroup.defaults.set(progress, forKey: Key.progress(model))
} else {
AppGroup.defaults.removeObject(forKey: Key.progress(model))
}
}
public static func clearProgress(for model: OnDeviceModel) {
guard AppGroup.isAvailable else { return }
AppGroup.defaults.removeObject(forKey: Key.progress(model))
}
public static func setModelsLoadedInMemory(_ loaded: Bool) {
guard AppGroup.isAvailable else { return }
AppGroup.defaults.set(loaded, forKey: Key.modelsLoadedInMemory)
}
public static func modelsLoadedInMemory(defaults: UserDefaults? = nil) -> Bool {
let store = defaults ?? (AppGroup.isAvailable ? AppGroup.defaults : .standard)
return store.bool(forKey: Key.modelsLoadedInMemory)
}
// MARK: - Reads (keyboard + host app)
public static func isDownloaded(_ model: OnDeviceModel, defaults: UserDefaults? = nil) -> Bool {
let store = defaults ?? (AppGroup.isAvailable ? AppGroup.defaults : .standard)
return store.bool(forKey: Key.downloaded(model))
}
public static func downloadProgress(_ model: OnDeviceModel, defaults: UserDefaults? = nil) -> Double? {
let store = defaults ?? (AppGroup.isAvailable ? AppGroup.defaults : .standard)
guard store.object(forKey: Key.progress(model)) != nil else { return nil }
return store.double(forKey: Key.progress(model))
}
/// Whether the currently selected local-engine stack has every
/// required on-device model downloaded.
public static func isLocalStackReady(
asrBackend: LocalASRBackend,
defaults: UserDefaults? = nil
) -> Bool {
if asrBackend == .qwen3ASR {
return isDownloaded(.qwen3ASR, defaults: defaults)
}
return true
}
/// First missing model for the active local stack, if any.
public static func firstMissingModel(
asrBackend: LocalASRBackend,
defaults: UserDefaults? = nil
) -> OnDeviceModel? {
if asrBackend == .qwen3ASR, !isDownloaded(.qwen3ASR, defaults: defaults) {
return .qwen3ASR
}
return nil
}
}
// MARK: - On-device Qwen3 runtime
/// CoreML ASR requires iOS 18+ / macOS 15+ (MLState KV cache).
public enum OnDeviceMLRuntime {
/// Whether Qwen3-ASR CoreML can run in this process.
public static var supportsOnDeviceQwen3: Bool {
if #available(iOS 18.0, *) {
return true
}
return false
}
}
OSGKeyboardShared/Services/PolishingService.swift
+26 -5
View File
@@ -5,9 +5,17 @@
// to produce polished, well-punctuated text. Falls back to the raw transcript
// if the LLM call fails or times out.
//
// Cloud engine always runs the LLM polish step (settings no longer expose
// off / transcribe). Local engine (`engineMode == "local"`) is ASR-only —
// the raw transcript is returned unchanged and cloud API settings are ignored.
// Engine matrix:
// - `engineMode == "cloud"` → always polish (cloud engine's whole point).
// - `engineMode == "local"`,
// `localModeCloudPolishEnabled == false` → ASR-only, return raw.
// - `engineMode == "local"`,
// `localModeCloudPolishEnabled == true` → polish via the user's LLM
// (DeepSeek by default). The local engine gains stronger accuracy on
// noisy / dialectal Chinese at the cost of one cloud round-trip.
// If the user hasn't entered an API key the call falls back to the
// raw transcript and surfaces a warning so the keyboard can show
// the "fill in your key" hint.
import Foundation
@@ -16,6 +24,11 @@ public actor PolishingService {
public enum PolishError: Error, Equatable {
case noTranscript
case timeout
/// v0.2.0: local engine + cloud-polish-on, but the user hasn't
/// saved an API key in the Keychain. Caller surfaces an Alert
/// telling them to fill it in; we deliver the raw transcript
/// so no data is lost.
case missingAPIKey
}
private let store: AppGroupStore
@@ -43,9 +56,17 @@ public actor PolishingService {
let trimmed = raw.trimmingCharacters(in: .whitespacesAndNewlines)
guard !trimmed.isEmpty else { throw PolishError.noTranscript }
// Local engine: ASR-only — no on-device or cloud polish.
// Local engine: ASR-only unless the user opted into cloud
// polish via `localModeCloudPolishEnabled`. The cloud polish
// path still requires an API key; if the Keychain is empty we
// fall back to the raw transcript and throw `missingAPIKey`
// so the UI can surface the "fill in your key" hint.
if store.engineMode == "local" {
return trimmed
guard store.localModeCloudPolishEnabled else { return trimmed }
guard !store.apiKey.isEmpty else {
throw PolishError.missingAPIKey
}
return try await polishRemote(trimmed)
}
return try await polishRemote(trimmed)