diff --git a/CHANGELOG.md b/CHANGELOG.md index 46635a2..63d440d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - **Host-only analytics recording**: the keyboard extension no longer opens the analytics or keyboard-usage SQLite queues. The host app is the only process that records and uploads events, so hiding the keyboard cannot hold WAL locks and get killed as `0xdead10cc`. Keyboard-activated and per-keystroke usage events are no longer collected. / **仅主 App 记埋点**：键盘扩展不再打开统计与键盘用量 SQLite 队列；开库、记账与上传只在主 App 进行，收起键盘时不会因 WAL 锁被系统以 `0xdead10cc` 终止。不再采集「键盘激活」和逐次按键用量。 ### Fixed +- **Faster managed dictation feedback**: begin the managed Volcengine streaming connection while the microphone touch primes audio, reuse it when recording starts, and keep the live transcript automatically scrolled to the newest recognition text. The early connection is reused only while it is fresh (under 10 seconds) and still open; otherwise it is closed and a new one opens, so an unused early connection can no longer make the next dictation fail. / **托管听写反馈提速**：在麦克风触摸预热音频时同步建立托管火山流式连接，录音开始后直接复用，并让实时转写自动滚动到最新识别文字。提前建立的连接仅在新鲜（10 秒内）且仍可用时复用，否则关闭并重新建立，未被使用的提前连接不会再导致下一次听写失败。 +- **Recording kept when live recognition fails**: a streaming recognition error while you were still speaking ended the dictation immediately and discarded the audio. Recording now continues, and when you finish, the recorded audio is retried with batch recognition before any error is shown. After a cancelled or failed dictation, that batch retry also stopped working for the rest of the session; it now works again from the next dictation. / **实时识别失败时保留录音**：说话过程中流式识别一旦出错，本次听写会立即结束并丢弃录音。现在会继续录音，说完后先用批量识别重试已录下的音频，仍失败才提示错误。此前一次听写被取消或失败后，本次会话内的批量重试也会失效；现在从下一次听写起恢复正常。 +- **Stalled cloud ASR losing the recording**: when the streaming recognizer went quiet and the result wait expired, the utterance ended as "no speech" even though the audio was captured. A stalled recognizer now counts as a failure, so the batch ASR fallback retries the recorded audio before giving up. / **云端识别卡住时丢失录音**：流式识别无响应、等待结果超时后，即便录音已采集，本次语音也会直接报「未检测到语音」。现在识别卡住按失败处理，会先用批量识别重试已录下的音频再放弃。 - **Speak as Me output shape**: the skill could answer with raw JSON variants or echo the personal-style XML tags into the result. The prompt now requires one ready-to-send plain-text rewrite, and length follows your natural voice instead of a mechanical ±20% cap. / **「像我所说」输出形态**：该技能可能返回原始 JSON 候选，或把个人风格的 XML 标记带进结果。现在提示词要求只输出一段可直接发送的文本，长度改为跟随你的自然表达节奏，不再机械限制 ±20%。 - **Clipboard history reverting deletes**: history is one shared blob, and the main App kept an in-memory copy that only refreshed when a screen appeared — so a delete made there could write back entries the keyboard had just captured or removed, and a clear done in the keyboard could be undone entirely. Both processes now announce their writes to each other, and the main App re-reads history whenever it returns to the foreground. / **剪贴板历史撤销删除**：历史记录是整块共享数据，而主 App 仅在页面出现时刷新内存副本——因此在主 App 中删除一条，可能把键盘刚采集或刚删除的条目重新写回，在键盘里执行的清空也可能被整体撤销。现在两端写入时会互相通知，主 App 每次回到前台也会重新读取历史。 - **Keyboard crash loop on memory relief**: recording a memory milestone could request relief while relief was already running, and the two sides called each other until the keyboard extension exhausted its stack (SIGSEGV grouped under `KeyboardViewController.viewDidLoad`). Reentrancy guards on both the controller and the telemetry side break the cycle. / **键盘内存降级递归崩溃**：记录内存里程碑时可能再次请求降级，两侧互相回调直至扩展栈溢出（崩溃聚合在 `KeyboardViewController.viewDidLoad`）。控制器与遥测两侧均加重入保护，打断循环。 diff --git a/OSGKeyboard/Services/FlowSessionManager.swift b/OSGKeyboard/Services/FlowSessionManager.swift index cca96c4..4c1dccc 100644 --- a/OSGKeyboard/Services/FlowSessionManager.swift +++ b/OSGKeyboard/Services/FlowSessionManager.swift @@ -2109,6 +2109,13 @@ final class FlowSessionManager: ObservableObject { !hasUnacknowledgedTerminalResult() else { return } + bindSessionASRIfNeeded() + if let cloudASR = sessionASR as? CloudASRService { + // Same locale source as `beginUtterance`, so the prime is adoptable. + cloudASR.primeUtteranceStreaming( + locale: SpeechLocaleResolver.resolve(store.localeId) + ) + } startAudioPrime(id: command.utteranceId, origin: "micTouch") } @@ -2171,6 +2178,7 @@ final class FlowSessionManager: ObservableObject { !isUtteranceProcessing else { return } + (sessionASR as? CloudASRService)?.cancelPrimedUtteranceStreaming() audioPrimeCancellationRequested = true // `capture.start()` is not cancellation-safe. Never stop it mid-start; // the completion waiter performs cleanup unless a real utterance adopts @@ -2613,11 +2621,14 @@ final class FlowSessionManager: ObservableObject { ] .map { $0.trimmingCharacters(in: .whitespacesAndNewlines) } .first(where: { !$0.isEmpty }) + // Without a partial, keep recording: capture is independent + // of ASR, and finalize retries the full PCM through batch + // ASR (`recognitionFailed`) before reporting the error. if let recovery { manager.lastFinal = recovery manager.debug("asr error recovered via partial len=\(recovery.count)") } else if manager.isUtteranceRecording { - manager.failUtterance(message: message, kind: .asrFailed) + manager.debug("asr error while recording — deferring to batch fallback") } case .cancelled: // A provider-side cancellation while the mic is still open @@ -2935,8 +2946,16 @@ final class FlowSessionManager: ObservableObject { try? await Task.sleep(nanoseconds: 100_000_000) } + // The recognizer never reported back within the wait: treat it as a + // failure so batch ASR can still recover the captured audio. + let asrStalled = lastFinal.isEmpty + && asrCompletedGeneration != finalizeGeneration + && !Task.isCancelled if lastFinal.isEmpty { - FlowDiagnostics.log("ASR wait elapsed — cancelling ASR task and using best available transcript") + FlowDiagnostics.log( + "ASR wait elapsed — cancelling ASR task and using best available transcript " + + "stalled=\(asrStalled ? 1 : 0)" + ) asrTask?.cancel() Task { await chunkedPipeline?.cancel() } } @@ -2973,11 +2992,13 @@ final class FlowSessionManager: ObservableObject { let wantsBatchFallback = UtteranceBatchFallbackPolicy.shouldRunBatchFallback( stitchedFinal: lastFinal, partialSnapshot: bestPartialSnapshot, - recognitionFailed: asrFailureMessage != nil + recognitionFailed: asrFailureMessage != nil, + recognitionStalled: asrStalled ) FlowTrace.pipeline( "batchFallback.decision", - "wanted=\(wantsBatchFallback ? 1 : 0) pcmSamples=\(utterancePCMSamples.count) " + "wanted=\(wantsBatchFallback ? 1 : 0) stalled=\(asrStalled ? 1 : 0) " + + "pcmSamples=\(utterancePCMSamples.count) " + "pcmRms=\(FlowTrace.rms(utterancePCMSamples)) " + "stitchedLen=\(lastFinal.count) partialLen=\(bestPartialSnapshot.count) " + "resolvedLen=\(text.count)" diff --git a/OSGKeyboardExt/Views/AIKeyboardView.swift b/OSGKeyboardExt/Views/AIKeyboardView.swift index 4b2f776..26caa6b 100644 --- a/OSGKeyboardExt/Views/AIKeyboardView.swift +++ b/OSGKeyboardExt/Views/AIKeyboardView.swift @@ -6,6 +6,7 @@ import OSGKeyboardShared import SwiftUI +import UIKit struct AIKeyboardView: View { private enum Layout { @@ -35,6 +36,7 @@ struct AIKeyboardView: View { static let pageDotSize: CGFloat = 5 static let skillPaginationBottomInset: CGFloat = 6 static let maximumSemanticSkills = 5 + static let statusViewportHeight = UIFont.systemFont(ofSize: 15).lineHeight * 3 } private struct SemanticBadgeContent { @@ -685,13 +687,37 @@ struct AIKeyboardView: View { } private func statusText(_ text: String, color: Color) -> some View { - Text(text) - .font(TypeStyle.body) - .foregroundStyle(color) - .lineLimit(2) - .multilineTextAlignment(.center) - .truncationMode(.tail) - .frame(maxWidth: .infinity) + GeometryReader { geometry in + ScrollViewReader { proxy in + ScrollView(.vertical, showsIndicators: false) { + // Match the viewport height while short, then grow upward one + // complete line at a time as the live transcript wraps. + Text(text) + .font(TypeStyle.body) + .foregroundStyle(color) + .multilineTextAlignment(.center) + .fixedSize(horizontal: false, vertical: true) + .frame( + maxWidth: .infinity, + minHeight: geometry.size.height, + alignment: .center + ) + .id("assistant.status.text") + } + .scrollDisabled(true) + .scrollEdgeEffectHidden() + .onAppear { + proxy.scrollTo("assistant.status.text", anchor: .bottom) + } + .onChange(of: text) { _, _ in + // ASR partials update per word. Jump only by the resulting + // layout delta instead of animating every partial update. + proxy.scrollTo("assistant.status.text", anchor: .bottom) + } + } + } + .frame(height: Layout.statusViewportHeight) + .frame(maxWidth: .infinity) } private var activeStatus: (text: String, color: Color)? { diff --git a/OSGKeyboardHostSupport/Services/CloudASR/CloudASRService.swift b/OSGKeyboardHostSupport/Services/CloudASR/CloudASRService.swift index c94e810..b4a55e8 100644 --- a/OSGKeyboardHostSupport/Services/CloudASR/CloudASRService.swift +++ b/OSGKeyboardHostSupport/Services/CloudASR/CloudASRService.swift @@ -11,11 +11,43 @@ import os import OSGKeyboardShared #endif +/// A primed streaming session is opened before the utterance has an owner. +/// Route its callbacks through a mutable relay, then bind the real partial +/// consumer when the mic gesture is committed to an utterance. +private final class CloudASRPartialRelay: @unchecked Sendable { + private let lock = OSAllocatedUnfairLock< + (@Sendable (String) -> Void)? + >(initialState: nil) + + func bind(_ consumer: @escaping @Sendable (String) -> Void) { + lock.withLock { $0 = consumer } + } + + func publish(_ text: String) { + let consumer = lock.withLock { $0 } + consumer?(text) + } +} + /// Uploads PCM only on the user-selected cloud engine path; provider clients /// reject missing credentials before network transmission. Personal dictionary /// entries are sent as recognition bias. Mutable client/cancellation state is /// lock-protected, which is the basis for `@unchecked Sendable`. public final class CloudASRService: ASRService, @unchecked Sendable { + private struct PrimedStreamingSession { + let id: UUID + let selection: String + let localeID: String + let createdAt: ContinuousClock.Instant + let relay: CloudASRPartialRelay + let task: Task + } + + /// The gateway closes a session that receives no audio for 15 s (counted + /// from WebSocket connect). A primed session older than this is never + /// adopted, leaving headroom for the first audio frame to arrive. + public static let defaultPrimedSessionMaxAge: Duration = .seconds(10) + private let store: any ConfigurationStore private let session: URLSession private let localFallback: ASRService @@ -23,24 +55,28 @@ public final class CloudASRService: ASRService, @unchecked Sendable { /// provider-specific local-fallback selection unchanged. private let managedClient: (any CloudASRTranscribing)? private let managedGrants: GatewayGrantCoordinator? + private let primedSessionMaxAge: Duration private let lock = OSAllocatedUnfairLock() private var client: CloudASRTranscribing? private var usesLocalFallback = false private var boundClientSelection: String? private var cancelled = false private var streamingPipeline: StreamingUtterancePipeline? + private var primedStreamingSession: PrimedStreamingSession? public init( store: any ConfigurationStore = AppGroupStore(), session: URLSession = .shared, localFallback: ASRService? = nil, managedClient: (any CloudASRTranscribing)? = nil, - managedGrants: GatewayGrantCoordinator? = nil + managedGrants: GatewayGrantCoordinator? = nil, + primedSessionMaxAge: Duration = CloudASRService.defaultPrimedSessionMaxAge ) { self.store = store self.session = session self.managedClient = managedClient self.managedGrants = managedGrants + self.primedSessionMaxAge = primedSessionMaxAge // `SpeechAnalyzerASR` is internal, so it can't appear in a public // default argument value — resolve the fallback in the body instead. self.localFallback = localFallback ?? SpeechAnalyzerASR() @@ -77,6 +113,143 @@ public final class CloudASRService: ASRService, @unchecked Sendable { } } + /// Starts the account-managed Volcengine reservation + WebSocket handshake + /// while the user's finger is still down. The utterance adopts this session + /// moments later, removing those network round trips from first-partial + /// latency without sending audio before recording actually begins. + public func primeUtteranceStreaming(locale: Locale) { + bindClientIfNeeded() + guard store.credentialSource == .managed, + let selection = lock.withLock({ boundClientSelection }), + let streamingClient = lock.withLock({ + client as? any CloudASRStreamingCapable + }) else { + return + } + + let localeID = locale.identifier(.bcp47) + let alreadyPrimed = lock.withLock { () -> Bool in + guard let current = primedStreamingSession else { return false } + return current.selection == selection + && current.localeID == localeID + && !isExpired(current) + } + guard !alreadyPrimed else { return } + + cancelPrimedUtteranceStreaming() + let id = UUID() + let relay = CloudASRPartialRelay() + let dictionary = store.personalDictionary + let task = Task.detached(priority: .userInitiated) { + try await streamingClient.openStreamingSession( + locale: locale, + dictionary: dictionary, + onPartial: { relay.publish($0) } + ) + } + lock.withLock { + primedStreamingSession = PrimedStreamingSession( + id: id, + selection: selection, + localeID: localeID, + createdAt: .now, + relay: relay, + task: task + ) + } + // Self-expire: the keyboard's cancel command may never arrive (the + // extension can be dismissed, or a start can fail before adoption). + let maxAge = primedSessionMaxAge + Task { [weak self] in + try? await Task.sleep(for: maxAge) + guard let self, let expired = self.takePrimedSession(id: id) else { return } + FlowTrace.asr("cloud.stream.primeDiscarded", "reason=expired") + Self.close(expired) + } + FlowTrace.asr( + "cloud.stream.primeStarted", + "locale=\(localeID) selection=\(selection)" + ) + } + + public func cancelPrimedUtteranceStreaming() { + guard let primed = takePrimedSession() else { return } + Self.close(primed) + } + + /// Removes the primed session; with an `id`, only if it is still that one. + private func takePrimedSession(id: UUID? = nil) -> PrimedStreamingSession? { + lock.withLock { + guard let current = primedStreamingSession, + id == nil || current.id == id else { + return nil + } + primedStreamingSession = nil + return current + } + } + + private func isExpired(_ primed: PrimedStreamingSession) -> Bool { + ContinuousClock.now - primed.createdAt >= primedSessionMaxAge + } + + /// Closes a primed session without cancelling its open task. Cancelling + /// mid-reservation can leave a reserved, never-connected gateway session + /// holding credits and a concurrency slot until its connect timeout; + /// letting the open finish and then closing releases it immediately. + private static func close(_ primed: PrimedStreamingSession) { + Task { + if let session = try? await primed.task.value { + session.cancel() + } + } + } + + /// Returns the primed session only if it still matches this utterance and + /// can accept audio. Anything else is closed and the caller opens a fresh + /// session, so priming can never make an utterance fail. + private func adoptPrimedSession( + _ primed: PrimedStreamingSession?, + selection: String?, + localeID: String, + onPartial: @escaping @Sendable (String) -> Void + ) async -> (any CloudASRStreamingSession)? { + guard let primed else { return nil } + guard primed.selection == selection, primed.localeID == localeID else { + FlowTrace.asr("cloud.stream.primeDiscarded", "reason=mismatch") + Self.close(primed) + return nil + } + guard !isExpired(primed) else { + FlowTrace.asr("cloud.stream.primeDiscarded", "reason=expired") + Self.close(primed) + return nil + } + + primed.relay.bind(onPartial) + let session: any CloudASRStreamingSession + do { + session = try await primed.task.value + } catch { + FlowTrace.asr( + "cloud.stream.primeDiscarded", + "reason=openFailed \(CloudASRLogMetadata.describe(error))" + ) + return nil + } + guard session.acceptsAudio else { + FlowTrace.asr("cloud.stream.primeDiscarded", "reason=closed") + session.cancel() + return nil + } + let ageMs = Int((ContinuousClock.now - primed.createdAt) / .milliseconds(1)) + FlowTrace.asr( + "cloud.stream.primeAdopted", + "locale=\(localeID) ageMs=\(ageMs)" + ) + return session + } + public func transcribeChunk(samples: [Float], locale: Locale) async -> ASRChunkResult { guard !samples.isEmpty else { return .success("") } if Task.isCancelled || lock.withLock({ cancelled }) { return .cancelled } @@ -129,23 +302,42 @@ public final class CloudASRService: ASRService, @unchecked Sendable { onPartial: @escaping @Sendable (String) -> Void ) async -> ChunkedUtterancePipelineOutcome { bindClientIfNeeded() + // A previous utterance's abort/failure leaves `cancelled` set, which + // would make this utterance's batch fallback (`transcribeChunk`) + // return `.cancelled` for the rest of the Flow session. + lock.withLock { cancelled = false } if usesLocalFallback { let pipeline = ChunkedUtterancePipeline(asr: localFallback, locale: locale) return await pipeline.transcribe(stream: stream, onPartial: onPartial) } guard let streamingClient = lock.withLock({ client as? CloudASRStreamingCapable }) else { + cancelPrimedUtteranceStreaming() let pipeline = ChunkedUtterancePipeline(asr: self, locale: locale) return await pipeline.transcribe(stream: stream, onPartial: onPartial) } + let selection = lock.withLock { boundClientSelection } + let localeID = locale.identifier(.bcp47) + // Every utterance either adopts the primed session or closes it. + let primed = takePrimedSession() + let session: any CloudASRStreamingSession do { - session = try await streamingClient.openStreamingSession( - locale: locale, - dictionary: store.personalDictionary, + if let adopted = await adoptPrimedSession( + primed, + selection: selection, + localeID: localeID, onPartial: onPartial - ) + ) { + session = adopted + } else { + session = try await streamingClient.openStreamingSession( + locale: locale, + dictionary: store.personalDictionary, + onPartial: onPartial + ) + } } catch where ProviderToolCancellation.matches(error) { return .cancelled } catch { @@ -249,6 +441,7 @@ public final class CloudASRService: ASRService, @unchecked Sendable { public func cancel() { lock.withLock { cancelled = true } + cancelPrimedUtteranceStreaming() let pipeline = lock.withLock { streamingPipeline } Task { await pipeline?.cancel() } localFallback.cancel() @@ -259,8 +452,10 @@ public final class CloudASRService: ASRService, @unchecked Sendable { let strategy = CloudASRModelCatalog.strategy(for: providerId) let credentialSource = store.credentialSource let selection = "\(credentialSource.rawValue):\(providerId)" - lock.withLock { - guard boundClientSelection != selection else { return } + let stalePrime = lock.withLock { () -> PrimedStreamingSession? in + guard boundClientSelection != selection else { return nil } + let stalePrime = primedStreamingSession + primedStreamingSession = nil boundClientSelection = selection if credentialSource == .managed { usesLocalFallback = false @@ -269,12 +464,16 @@ public final class CloudASRService: ASRService, @unchecked Sendable { session: session, managedGrants: managedGrants ) - return + return stalePrime } usesLocalFallback = strategy == .localFallback client = usesLocalFallback ? nil : CloudASRClientFactory.make(store: store, session: session) + return stalePrime + } + if let stalePrime { + Self.close(stalePrime) } } } diff --git a/OSGKeyboardHostSupport/Services/CloudASR/CloudASRStreaming.swift b/OSGKeyboardHostSupport/Services/CloudASR/CloudASRStreaming.swift index e3cd954..b9ba18f 100644 --- a/OSGKeyboardHostSupport/Services/CloudASR/CloudASRStreaming.swift +++ b/OSGKeyboardHostSupport/Services/CloudASR/CloudASRStreaming.swift @@ -51,6 +51,13 @@ public protocol CloudASRStreamingSession: Sendable { /// Signal end-of-audio and wait for the polish-ready final transcript. func finish() async throws -> String func cancel() + /// False once the session has failed, closed, or been cancelled, so a + /// pre-opened session is not adopted after its transport already died. + var acceptsAudio: Bool { get } +} + +public extension CloudASRStreamingSession { + var acceptsAudio: Bool { true } } /// Providers that can open an utterance-level streaming session. diff --git a/OSGKeyboardHostSupport/Services/CloudASR/ManagedVolcengineASRClient.swift b/OSGKeyboardHostSupport/Services/CloudASR/ManagedVolcengineASRClient.swift index ec60973..38ccd94 100644 --- a/OSGKeyboardHostSupport/Services/CloudASR/ManagedVolcengineASRClient.swift +++ b/OSGKeyboardHostSupport/Services/CloudASR/ManagedVolcengineASRClient.swift @@ -765,6 +765,12 @@ private final class ManagedVolcengineASRSession: CloudASRStreamingSession, @unch } } + var acceptsAudio: Bool { + lock.withLock { state in + !state.cancelled && state.failure == nil && !state.receiveClosed && !state.endSent + } + } + func cancel() { let changed = lock.withLock { state -> Bool in guard !state.cancelled else { return false } diff --git a/OSGKeyboardShared/Utilities/UtteranceBatchFallbackPolicy.swift b/OSGKeyboardShared/Utilities/UtteranceBatchFallbackPolicy.swift index 1e24178..60a259e 100644 --- a/OSGKeyboardShared/Utilities/UtteranceBatchFallbackPolicy.swift +++ b/OSGKeyboardShared/Utilities/UtteranceBatchFallbackPolicy.swift @@ -10,13 +10,18 @@ public enum UtteranceBatchFallbackPolicy { public static let defaultCharacterAdvantage = UtteranceTranscriptGuard.defaultPartialAdvantage /// True when chunked output likely lost trailing content vs the live partial. + /// + /// `recognitionStalled` means the ASR wait expired before the recognizer + /// finished (e.g. the streaming socket went quiet). That is a failure, not + /// silence, so the captured PCM still deserves a batch attempt. public static func shouldRunBatchFallback( stitchedFinal: String, partialSnapshot: String, recognitionFailed: Bool = false, + recognitionStalled: Bool = false, minimumCharacterAdvantage: Int = defaultCharacterAdvantage ) -> Bool { - if recognitionFailed { return true } + if recognitionFailed || recognitionStalled { return true } let final = stitchedFinal.trimmingCharacters(in: .whitespacesAndNewlines) let partial = partialSnapshot.trimmingCharacters(in: .whitespacesAndNewlines) diff --git a/OSGKeyboardTests/CloudASRServiceTests.swift b/OSGKeyboardTests/CloudASRServiceTests.swift index 106fe2f..88a1bc9 100644 --- a/OSGKeyboardTests/CloudASRServiceTests.swift +++ b/OSGKeyboardTests/CloudASRServiceTests.swift @@ -30,6 +30,80 @@ private final class RecordingFallbackASR: ASRService, @unchecked Sendable { } } +private final class RecordingStreamingSession: CloudASRStreamingSession, @unchecked Sendable { + private let onPartial: @Sendable (String) -> Void + /// Models a transport the gateway already closed (e.g. idle timeout). + private let isDead: Bool + + init(onPartial: @escaping @Sendable (String) -> Void, isDead: Bool = false) { + self.onPartial = onPartial + self.isDead = isDead + } + + var acceptsAudio: Bool { !isDead } + + func append(samples: [Float]) async throws { + if isDead { throw ManagedCloudASRError.idleTimeout } + if !samples.isEmpty { + onPartial("你好") + } + } + + func finish() async throws -> String { + if isDead { throw ManagedCloudASRError.idleTimeout } + return "你好" + } + + func cancel() {} +} + +/// Sessions are numbered by open order, starting at 1. +private final class RecordingStreamingClient: CloudASRStreamingCapable, @unchecked Sendable { + private let lock = OSAllocatedUnfairLock(initialState: 0) + private let deadSessions: Set + private let failingOpens: Set + + init(deadSessions: Set = [], failingOpens: Set = []) { + self.deadSessions = deadSessions + self.failingOpens = failingOpens + } + + var openCount: Int { + lock.withLock { $0 } + } + + func prepare(dictionary: PersonalDictionary) async throws {} + + func transcribe( + samples: [Float], + sampleRate: Int, + locale: Locale, + dictionary: PersonalDictionary + ) async throws -> String { + "batch" + } + + func openStreamingSession( + locale: Locale, + dictionary: PersonalDictionary, + onPartial: @escaping @Sendable (String) -> Void + ) async throws -> any CloudASRStreamingSession { + _ = locale + _ = dictionary + let index = lock.withLock { count -> Int in + count += 1 + return count + } + if failingOpens.contains(index) { + throw ManagedCloudASRError.sessionTransportFailed + } + return RecordingStreamingSession( + onPartial: onPartial, + isDead: deadSessions.contains(index) + ) + } +} + final class CloudASRServiceTests: XCTestCase { private var suiteName: String! @@ -93,4 +167,142 @@ final class CloudASRServiceTests: XCTestCase { XCTAssertEqual(text, "") XCTAssertEqual(fallback.calls, 0) } + + func testManagedStreamingAdoptsPrimedSession() async { + let client = RecordingStreamingClient() + let service = makeManagedService(client: client) + service.primeUtteranceStreaming(locale: zhLocale) + + let partials = OSAllocatedUnfairLock(initialState: [String]()) + let outcome = await service.transcribeUtteranceStreaming( + stream: oneSnapshotStream(), + locale: zhLocale + ) { partial in + partials.withLock { $0.append(partial) } + } + + guard case .success(let success) = outcome else { + return XCTFail("expected streaming success, got \(outcome)") + } + XCTAssertEqual(success.text, "你好") + XCTAssertEqual(partials.withLock { $0 }, ["你好"]) + XCTAssertEqual(client.openCount, 1) + } + + func testDeadPrimedSessionIsReplacedWithFreshSession() async { + let client = RecordingStreamingClient(deadSessions: [1]) + let service = makeManagedService(client: client) + service.primeUtteranceStreaming(locale: zhLocale) + await waitUntil { client.openCount == 1 } + + let outcome = await service.transcribeUtteranceStreaming( + stream: oneSnapshotStream(), + locale: zhLocale + ) { _ in } + + guard case .success(let success) = outcome else { + return XCTFail("expected fresh streaming success, got \(outcome)") + } + XCTAssertEqual(success.text, "你好") + XCTAssertEqual(client.openCount, 2) + } + + func testExpiredPrimedSessionIsNotAdopted() async { + let client = RecordingStreamingClient(deadSessions: [1]) + let service = makeManagedService(client: client, primedSessionMaxAge: .zero) + service.primeUtteranceStreaming(locale: zhLocale) + await waitUntil { client.openCount == 1 } + + let outcome = await service.transcribeUtteranceStreaming( + stream: oneSnapshotStream(), + locale: zhLocale + ) { _ in } + + guard case .success(let success) = outcome else { + return XCTFail("expected fresh streaming success, got \(outcome)") + } + XCTAssertEqual(success.text, "你好") + XCTAssertEqual(client.openCount, 2) + } + + func testFailedPrimeOpensFreshStreamingSession() async { + let client = RecordingStreamingClient(failingOpens: [1]) + let service = makeManagedService(client: client) + service.primeUtteranceStreaming(locale: zhLocale) + + let outcome = await service.transcribeUtteranceStreaming( + stream: oneSnapshotStream(), + locale: zhLocale + ) { _ in } + + // "你好" comes from streaming; the chunked batch fallback returns "batch". + guard case .success(let success) = outcome else { + return XCTFail("expected fresh streaming success, got \(outcome)") + } + XCTAssertEqual(success.text, "你好") + XCTAssertEqual(client.openCount, 2) + } + + func testStreamingUtteranceClearsCancellationForBatchFallback() async { + let client = RecordingStreamingClient() + let service = makeManagedService(client: client) + service.cancel() + + _ = await service.transcribeUtteranceStreaming( + stream: oneSnapshotStream(), + locale: zhLocale + ) { _ in } + let batch = await service.transcribeChunk( + samples: [Float](repeating: 0.1, count: 1_600), + locale: zhLocale + ) + + guard case .success(let text) = batch else { + return XCTFail("expected batch success after a new utterance, got \(batch)") + } + XCTAssertEqual(text, "batch") + } + + // MARK: - Helpers + + private let zhLocale = Locale(identifier: "zh-Hans") + + private func makeManagedService( + client: RecordingStreamingClient, + primedSessionMaxAge: Duration = CloudASRService.defaultPrimedSessionMaxAge + ) -> CloudASRService { + var config = AppGroupConfiguration.load(fromAvailable: defaults) + config.engineMode = "cloud" + config.credentialSource = .managed + config.save(to: defaults) + store = AppGroupStore(defaults: defaults) + return CloudASRService( + store: store, + session: .shared, + localFallback: RecordingFallbackASR(), + managedClient: client, + primedSessionMaxAge: primedSessionMaxAge + ) + } + + private func oneSnapshotStream() -> AsyncStream { + let (stream, continuation) = AsyncStream.makeStream() + continuation.yield( + AudioBufferSnapshot( + samples: [Float](repeating: 0.1, count: 1_600), + sampleRate: 16_000 + ) + ) + continuation.finish() + return stream + } + + /// Lets the detached prime task run before the utterance starts. + private func waitUntil(_ condition: () -> Bool) async { + var attempts = 0 + while !condition(), attempts < 200 { + attempts += 1 + try? await Task.sleep(for: .milliseconds(5)) + } + } } diff --git a/OSGKeyboardTests/UtteranceBatchFallbackPolicyTests.swift b/OSGKeyboardTests/UtteranceBatchFallbackPolicyTests.swift index 0f0285b..957ce90 100644 --- a/OSGKeyboardTests/UtteranceBatchFallbackPolicyTests.swift +++ b/OSGKeyboardTests/UtteranceBatchFallbackPolicyTests.swift @@ -43,6 +43,25 @@ final class UtteranceBatchFallbackPolicyTests: XCTestCase { ) } + func testShouldRunWhenRecognitionStalledWithoutAnyText() { + XCTAssertTrue( + UtteranceBatchFallbackPolicy.shouldRunBatchFallback( + stitchedFinal: "", + partialSnapshot: "", + recognitionStalled: true + ) + ) + } + + func testShouldNotRunForCompletedSilence() { + XCTAssertFalse( + UtteranceBatchFallbackPolicy.shouldRunBatchFallback( + stitchedFinal: "", + partialSnapshot: "" + ) + ) + } + func testPreferredTranscriptPicksLongestCandidate() { let resolved = UtteranceBatchFallbackPolicy.preferredTranscript( batch: "今天很好，我们一起去公园吧",