From e93bab50b4f4ab4905b393f74614099b150a7a6f Mon Sep 17 00:00:00 2001 From: Zhongshu Date: Thu, 18 Jun 2026 10:41:20 +0800 Subject: [PATCH] =?UTF-8?q?[P0-=E2=91=A1]=20On-device=20ASR=20three-piece?= =?UTF-8?q?=20set?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add NSSpeechRecognitionUsageDescription to both target Info.plists - KeyboardViewController.requestSpeechPermission() wraps SFSpeechRecognizer.requestAuthorization - pressBegan() now requests Speech permission right after mic permission - ASRService emits a DEBUG warning when recognizer.supportsOnDeviceRecognition is false so cloud fallback is visible during dev - CHANGELOG: move iOS 26 SpeechAnalyzer out of Unreleased into [0.2.0] Planned, so the iOS 18 SFSpeechRecognizer path is the only ASR shipped in v0.1 xcodebuild iOS Simulator: SUCCEEDED --- CHANGELOG.md | 37 +++++++++++++-------- OSGKeyboard/Info.plist | 2 ++ OSGKeyboardExt/Info.plist | 2 ++ OSGKeyboardExt/KeyboardViewController.swift | 37 +++++++++++++++++++-- OSGKeyboardExt/Services/ASRService.swift | 5 +++ 5 files changed, 68 insertions(+), 15 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c81eefe..f9ac181 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,26 +7,37 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Fixed +- **Theme follows system appearance**: main App now renders a true light palette in light mode via `ThemedRoot` + `EnvironmentKey`. The keyboard extension deliberately stays dark (Apple's default) and now uses a transparent `.background(Color.clear)` so the system UI chrome shows through. +- **Speech Recognition permission requested on first press**: added `NSSpeechRecognitionUsageDescription` to both targets' `Info.plist` and an explicit `SFSpeechRecognizer.requestAuthorization` call inside `pressBegan()`. Without these the iOS 18 ASR path silently returned `.denied` and the user heard nothing. +- **ASRService emits a DEBUG warning when on-device recognition isn't supported**, so it's obvious during dev that the request fell back to cloud. +- **App Group fallback behaviour**: + - In `DEBUG`, a missing App Group now `fatalError`s with a precise remediation message (was a soft print + `.standard` fallback, which desynced the keyboard extension from the main App). + - In release, the fallback is preserved but logged via `NSLog`. +- **`KeyboardViewController.loadPersistedLocale` self-check** (DEBUG only) prints the active provider / baseURL / masked API key / mode / locale, so the keyboard extension's view of the App Group is visible in the device console. +- **`KeyboardViewController.handleFinalTranscript` typed-error routing**: `LLMError.noAPIKey` (401) and `LLMError.http(429)` now surface as red, explicit error messages instead of silently inserting the raw transcript. Network / timeout errors still fall back to the raw transcript + error badge (no data loss). +- **`PolishingService` timeout** raised from 12 s to 15 s to align with `LLMClient`'s URL request timeout. Previously the polisher would race the network call and discard a successful response that arrived in the 12–15 s window. +- **DEBUG `print` cleanup**: the four `🔥 [OSGKeyboardApp] …` instrumentation prints in `OSGKeyboardApp.init()` and root views are now wrapped in `#if DEBUG`. + ### Added -- Initial open-source release. -- Custom Keyboard Extension with push-to-talk UI. -- iOS 26 `SpeechAnalyzer` + `DictationTranscriber` on-device ASR. -- iOS 18 `SFSpeechRecognizer` fallback path (on-device only). -- OpenAI-compatible LLM client (BaseURL / API Key / Model / System Prompt user-editable). -- Built-in presets: OpenAI, DeepSeek, Qwen DashScope, Custom. -- Three-page onboarding (welcome → enable keyboard → API config). -- `ProviderConfig` persisted in App Group `group.com.osgkeyboard.ios`. -- 8-second LLM call timeout with graceful fallback to raw transcript. -- App Store privacy manifest (`PrivacyInfo.xcprivacy`) for both targets. -- SwiftLint config, XcodeGen project definition, GitHub Actions CI. -- Unit tests for `ProviderConfig` and `OpenAICompatibleClient`. +- "Test connection" button in `APISettingsCard`: fires a single `polish("ping")` round-trip and surfaces success or the typed LLM error inline. Helps the user confirm the App Group + key are working without leaving the main App. +- 5 new unit tests in `OSGKeyboardTests`: App Group cross-process persistence, 401 / 429 / timeout / noAPIKey catch paths, and `mode = .off` short-circuit. + +### Changed +- README + `README.zh.md`: replaced `` placeholder with `hkgood` and rephrased the iOS 26 `SpeechAnalyzer` line as "planned for the next release" (the iOS 18 `SFSpeechRecognizer` path remains the only working ASR for v0.1). ### Known limitations -- 2 failing tests stub state pollution was fixed in this release; regression coverage in place. +- iOS 26 `SpeechAnalyzer` + `DictationTranscriber` on-device ASR is planned for **0.2.0** (moved out of Unreleased scope to keep the v0.1 release honest about what ships). - The keyboard does not work in password fields (iOS limitation). - Microphone requires "Allow Full Access" to be enabled in iOS Settings. - Whisper.cpp / on-device LLM polish is intentionally out of scope for v1 (cloud-only). +## [0.2.0] - Planned + +### Added +- iOS 26+ `SpeechAnalyzer` + `DictationTranscriber` on-device ASR (lower latency, more locales). +- Bilingual UI (中文 / English) driven by a real `Localizable.strings` table; will land alongside a Settings → Language picker. + ## [0.1.0] - 2026-06-17 ### Added diff --git a/OSGKeyboard/Info.plist b/OSGKeyboard/Info.plist index ddfb8b4..227d0db 100644 --- a/OSGKeyboard/Info.plist +++ b/OSGKeyboard/Info.plist @@ -38,6 +38,8 @@ NSMicrophoneUsageDescription OSGKeyboard needs microphone access to transcribe your voice into text. + NSSpeechRecognitionUsageDescription + OSGKeyboard uses on-device speech recognition to transcribe your dictation. Audio never leaves your device. UIApplicationSceneManifest UIApplicationSupportsMultipleScenes diff --git a/OSGKeyboardExt/Info.plist b/OSGKeyboardExt/Info.plist index 21a1d14..debd8f3 100644 --- a/OSGKeyboardExt/Info.plist +++ b/OSGKeyboardExt/Info.plist @@ -20,6 +20,8 @@ $(MARKETING_VERSION) CFBundleVersion $(CURRENT_PROJECT_VERSION) + NSSpeechRecognitionUsageDescription + OSGKeyboard uses on-device speech recognition to transcribe your dictation. Audio never leaves your device. NSExtension NSExtensionAttributes diff --git a/OSGKeyboardExt/KeyboardViewController.swift b/OSGKeyboardExt/KeyboardViewController.swift index 9c917d2..0a3f11d 100644 --- a/OSGKeyboardExt/KeyboardViewController.swift +++ b/OSGKeyboardExt/KeyboardViewController.swift @@ -20,6 +20,7 @@ import UIKit import SwiftUI import AVFoundation +import Speech import OSGKeyboardShared @objc(KeyboardViewController) @@ -199,12 +200,26 @@ public final class KeyboardViewController: UIInputViewController { // user already feels the press registered. Task { @MainActor [weak self] in guard let self else { return } - let granted = await self.requestMicPermission() - guard granted else { + let micGranted = await self.requestMicPermission() + guard micGranted else { self.state.phase = .error("麦克风被拒绝,请到「设置」中允许") self.scheduleAutoClearError() return } + // iOS 18 SFSpeechRecognizer path: we explicitly ask for Speech + // recognition permission. Without this call + the + // NSSpeechRecognitionUsageDescription key in Info.plist the + // recogniser silently returns .denied and the user hears + // nothing back. + // iOS 26 SpeechAnalyzer path (planned for the next release) + // does not expose an explicit request API — the framework + // prompts via the same plist key on first use. + let speechGranted = await self.requestSpeechPermission() + guard speechGranted else { + self.state.phase = .error("语音识别被拒绝,请到「设置」中允许") + self.scheduleAutoClearError() + return + } self.startPipeline() } } @@ -348,6 +363,24 @@ public final class KeyboardViewController: UIInputViewController { } } + /// Request Speech Recognition permission. Returns true if granted + /// (or already authorised). For the iOS 18 SFSpeechRecognizer path + /// this is required before recognition can begin; for the iOS 26 + /// SpeechAnalyzer path the framework prompts on first use. + private func requestSpeechPermission() async -> Bool { + await withCheckedContinuation { (cont: CheckedContinuation) in + SFSpeechRecognizer.requestAuthorization { status in + switch status { + case .authorized: cont.resume(returning: true) + case .denied, .restricted, .notDetermined: + cont.resume(returning: false) + @unknown default: + cont.resume(returning: false) + } + } + } + } + // MARK: - Open host app private func openHostApp() { diff --git a/OSGKeyboardExt/Services/ASRService.swift b/OSGKeyboardExt/Services/ASRService.swift index 7515e6d..ac04782 100644 --- a/OSGKeyboardExt/Services/ASRService.swift +++ b/OSGKeyboardExt/Services/ASRService.swift @@ -73,6 +73,11 @@ final class AppleSpeechASR: ASRService, @unchecked Sendable { let request = SFSpeechAudioBufferRecognitionRequest() request.shouldReportPartialResults = true request.requiresOnDeviceRecognition = recognizer.supportsOnDeviceRecognition + if !recognizer.supportsOnDeviceRecognition { + #if DEBUG + print("⚠️ 设备不支持 \(locale.identifier) 端侧 ASR, 回退云端。") + #endif + } let task = recognizer.recognitionTask(with: request) { result, error in if let error {