fix(asr): keep recordings when streaming recognition fails or stalls

- Keep recording after a streaming ASR error and retry the captured PCM
  through batch ASR before reporting a failure.
- Treat an expired ASR wait as a stall so batch fallback still runs.
- Prime the managed Volcengine stream on mic touch and adopt it only while
  fresh and open.
- Auto-scroll the live transcript to the newest recognized text.
This commit is contained in:
2026-10-20 20:05:08 +02:00
parent f7a031cc1d
commit be56d2210c
9 changed files with 518 additions and 20 deletions
+3
View File
@@ -13,6 +13,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **Host-only analytics recording**: the keyboard extension no longer opens the analytics or keyboard-usage SQLite queues. The host app is the only process that records and uploads events, so hiding the keyboard cannot hold WAL locks and get killed as `0xdead10cc`. Keyboard-activated and per-keystroke usage events are no longer collected. / **仅主 App 记埋点**:键盘扩展不再打开统计与键盘用量 SQLite 队列;开库、记账与上传只在主 App 进行,收起键盘时不会因 WAL 锁被系统以 `0xdead10cc` 终止。不再采集「键盘激活」和逐次按键用量。 - **Host-only analytics recording**: the keyboard extension no longer opens the analytics or keyboard-usage SQLite queues. The host app is the only process that records and uploads events, so hiding the keyboard cannot hold WAL locks and get killed as `0xdead10cc`. Keyboard-activated and per-keystroke usage events are no longer collected. / **仅主 App 记埋点**:键盘扩展不再打开统计与键盘用量 SQLite 队列;开库、记账与上传只在主 App 进行,收起键盘时不会因 WAL 锁被系统以 `0xdead10cc` 终止。不再采集「键盘激活」和逐次按键用量。
### Fixed ### Fixed
- **Faster managed dictation feedback**: begin the managed Volcengine streaming connection while the microphone touch primes audio, reuse it when recording starts, and keep the live transcript automatically scrolled to the newest recognition text. The early connection is reused only while it is fresh (under 10 seconds) and still open; otherwise it is closed and a new one opens, so an unused early connection can no longer make the next dictation fail. / **托管听写反馈提速**:在麦克风触摸预热音频时同步建立托管火山流式连接,录音开始后直接复用,并让实时转写自动滚动到最新识别文字。提前建立的连接仅在新鲜(10 秒内)且仍可用时复用,否则关闭并重新建立,未被使用的提前连接不会再导致下一次听写失败。
- **Recording kept when live recognition fails**: a streaming recognition error while you were still speaking ended the dictation immediately and discarded the audio. Recording now continues, and when you finish, the recorded audio is retried with batch recognition before any error is shown. After a cancelled or failed dictation, that batch retry also stopped working for the rest of the session; it now works again from the next dictation. / **实时识别失败时保留录音**:说话过程中流式识别一旦出错,本次听写会立即结束并丢弃录音。现在会继续录音,说完后先用批量识别重试已录下的音频,仍失败才提示错误。此前一次听写被取消或失败后,本次会话内的批量重试也会失效;现在从下一次听写起恢复正常。
- **Stalled cloud ASR losing the recording**: when the streaming recognizer went quiet and the result wait expired, the utterance ended as "no speech" even though the audio was captured. A stalled recognizer now counts as a failure, so the batch ASR fallback retries the recorded audio before giving up. / **云端识别卡住时丢失录音**:流式识别无响应、等待结果超时后,即便录音已采集,本次语音也会直接报「未检测到语音」。现在识别卡住按失败处理,会先用批量识别重试已录下的音频再放弃。
- **Speak as Me output shape**: the skill could answer with raw JSON variants or echo the personal-style XML tags into the result. The prompt now requires one ready-to-send plain-text rewrite, and length follows your natural voice instead of a mechanical ±20% cap. / **「像我所说」输出形态**:该技能可能返回原始 JSON 候选,或把个人风格的 XML 标记带进结果。现在提示词要求只输出一段可直接发送的文本,长度改为跟随你的自然表达节奏,不再机械限制 ±20%。 - **Speak as Me output shape**: the skill could answer with raw JSON variants or echo the personal-style XML tags into the result. The prompt now requires one ready-to-send plain-text rewrite, and length follows your natural voice instead of a mechanical ±20% cap. / **「像我所说」输出形态**:该技能可能返回原始 JSON 候选,或把个人风格的 XML 标记带进结果。现在提示词要求只输出一段可直接发送的文本,长度改为跟随你的自然表达节奏,不再机械限制 ±20%。
- **Clipboard history reverting deletes**: history is one shared blob, and the main App kept an in-memory copy that only refreshed when a screen appeared — so a delete made there could write back entries the keyboard had just captured or removed, and a clear done in the keyboard could be undone entirely. Both processes now announce their writes to each other, and the main App re-reads history whenever it returns to the foreground. / **剪贴板历史撤销删除**:历史记录是整块共享数据,而主 App 仅在页面出现时刷新内存副本——因此在主 App 中删除一条,可能把键盘刚采集或刚删除的条目重新写回,在键盘里执行的清空也可能被整体撤销。现在两端写入时会互相通知,主 App 每次回到前台也会重新读取历史。 - **Clipboard history reverting deletes**: history is one shared blob, and the main App kept an in-memory copy that only refreshed when a screen appeared — so a delete made there could write back entries the keyboard had just captured or removed, and a clear done in the keyboard could be undone entirely. Both processes now announce their writes to each other, and the main App re-reads history whenever it returns to the foreground. / **剪贴板历史撤销删除**:历史记录是整块共享数据,而主 App 仅在页面出现时刷新内存副本——因此在主 App 中删除一条,可能把键盘刚采集或刚删除的条目重新写回,在键盘里执行的清空也可能被整体撤销。现在两端写入时会互相通知,主 App 每次回到前台也会重新读取历史。
- **Keyboard crash loop on memory relief**: recording a memory milestone could request relief while relief was already running, and the two sides called each other until the keyboard extension exhausted its stack (SIGSEGV grouped under `KeyboardViewController.viewDidLoad`). Reentrancy guards on both the controller and the telemetry side break the cycle. / **键盘内存降级递归崩溃**:记录内存里程碑时可能再次请求降级,两侧互相回调直至扩展栈溢出(崩溃聚合在 `KeyboardViewController.viewDidLoad`)。控制器与遥测两侧均加重入保护,打断循环。 - **Keyboard crash loop on memory relief**: recording a memory milestone could request relief while relief was already running, and the two sides called each other until the keyboard extension exhausted its stack (SIGSEGV grouped under `KeyboardViewController.viewDidLoad`). Reentrancy guards on both the controller and the telemetry side break the cycle. / **键盘内存降级递归崩溃**:记录内存里程碑时可能再次请求降级,两侧互相回调直至扩展栈溢出(崩溃聚合在 `KeyboardViewController.viewDidLoad`)。控制器与遥测两侧均加重入保护,打断循环。
+25 -4
View File
@@ -2109,6 +2109,13 @@ final class FlowSessionManager: ObservableObject {
!hasUnacknowledgedTerminalResult() else { !hasUnacknowledgedTerminalResult() else {
return return
} }
bindSessionASRIfNeeded()
if let cloudASR = sessionASR as? CloudASRService {
// Same locale source as `beginUtterance`, so the prime is adoptable.
cloudASR.primeUtteranceStreaming(
locale: SpeechLocaleResolver.resolve(store.localeId)
)
}
startAudioPrime(id: command.utteranceId, origin: "micTouch") startAudioPrime(id: command.utteranceId, origin: "micTouch")
} }
@@ -2171,6 +2178,7 @@ final class FlowSessionManager: ObservableObject {
!isUtteranceProcessing else { !isUtteranceProcessing else {
return return
} }
(sessionASR as? CloudASRService)?.cancelPrimedUtteranceStreaming()
audioPrimeCancellationRequested = true audioPrimeCancellationRequested = true
// `capture.start()` is not cancellation-safe. Never stop it mid-start; // `capture.start()` is not cancellation-safe. Never stop it mid-start;
// the completion waiter performs cleanup unless a real utterance adopts // the completion waiter performs cleanup unless a real utterance adopts
@@ -2613,11 +2621,14 @@ final class FlowSessionManager: ObservableObject {
] ]
.map { $0.trimmingCharacters(in: .whitespacesAndNewlines) } .map { $0.trimmingCharacters(in: .whitespacesAndNewlines) }
.first(where: { !$0.isEmpty }) .first(where: { !$0.isEmpty })
// Without a partial, keep recording: capture is independent
// of ASR, and finalize retries the full PCM through batch
// ASR (`recognitionFailed`) before reporting the error.
if let recovery { if let recovery {
manager.lastFinal = recovery manager.lastFinal = recovery
manager.debug("asr error recovered via partial len=\(recovery.count)") manager.debug("asr error recovered via partial len=\(recovery.count)")
} else if manager.isUtteranceRecording { } else if manager.isUtteranceRecording {
manager.failUtterance(message: message, kind: .asrFailed) manager.debug("asr error while recording — deferring to batch fallback")
} }
case .cancelled: case .cancelled:
// A provider-side cancellation while the mic is still open // A provider-side cancellation while the mic is still open
@@ -2935,8 +2946,16 @@ final class FlowSessionManager: ObservableObject {
try? await Task.sleep(nanoseconds: 100_000_000) try? await Task.sleep(nanoseconds: 100_000_000)
} }
// The recognizer never reported back within the wait: treat it as a
// failure so batch ASR can still recover the captured audio.
let asrStalled = lastFinal.isEmpty
&& asrCompletedGeneration != finalizeGeneration
&& !Task.isCancelled
if lastFinal.isEmpty { if lastFinal.isEmpty {
FlowDiagnostics.log("ASR wait elapsed — cancelling ASR task and using best available transcript") FlowDiagnostics.log(
"ASR wait elapsed — cancelling ASR task and using best available transcript "
+ "stalled=\(asrStalled ? 1 : 0)"
)
asrTask?.cancel() asrTask?.cancel()
Task { await chunkedPipeline?.cancel() } Task { await chunkedPipeline?.cancel() }
} }
@@ -2973,11 +2992,13 @@ final class FlowSessionManager: ObservableObject {
let wantsBatchFallback = UtteranceBatchFallbackPolicy.shouldRunBatchFallback( let wantsBatchFallback = UtteranceBatchFallbackPolicy.shouldRunBatchFallback(
stitchedFinal: lastFinal, stitchedFinal: lastFinal,
partialSnapshot: bestPartialSnapshot, partialSnapshot: bestPartialSnapshot,
recognitionFailed: asrFailureMessage != nil recognitionFailed: asrFailureMessage != nil,
recognitionStalled: asrStalled
) )
FlowTrace.pipeline( FlowTrace.pipeline(
"batchFallback.decision", "batchFallback.decision",
"wanted=\(wantsBatchFallback ? 1 : 0) pcmSamples=\(utterancePCMSamples.count) " "wanted=\(wantsBatchFallback ? 1 : 0) stalled=\(asrStalled ? 1 : 0) "
+ "pcmSamples=\(utterancePCMSamples.count) "
+ "pcmRms=\(FlowTrace.rms(utterancePCMSamples)) " + "pcmRms=\(FlowTrace.rms(utterancePCMSamples)) "
+ "stitchedLen=\(lastFinal.count) partialLen=\(bestPartialSnapshot.count) " + "stitchedLen=\(lastFinal.count) partialLen=\(bestPartialSnapshot.count) "
+ "resolvedLen=\(text.count)" + "resolvedLen=\(text.count)"
+33 -7
View File
@@ -6,6 +6,7 @@
import OSGKeyboardShared import OSGKeyboardShared
import SwiftUI import SwiftUI
import UIKit
struct AIKeyboardView: View { struct AIKeyboardView: View {
private enum Layout { private enum Layout {
@@ -35,6 +36,7 @@ struct AIKeyboardView: View {
static let pageDotSize: CGFloat = 5 static let pageDotSize: CGFloat = 5
static let skillPaginationBottomInset: CGFloat = 6 static let skillPaginationBottomInset: CGFloat = 6
static let maximumSemanticSkills = 5 static let maximumSemanticSkills = 5
static let statusViewportHeight = UIFont.systemFont(ofSize: 15).lineHeight * 3
} }
private struct SemanticBadgeContent { private struct SemanticBadgeContent {
@@ -685,13 +687,37 @@ struct AIKeyboardView: View {
} }
private func statusText(_ text: String, color: Color) -> some View { private func statusText(_ text: String, color: Color) -> some View {
Text(text) GeometryReader { geometry in
.font(TypeStyle.body) ScrollViewReader { proxy in
.foregroundStyle(color) ScrollView(.vertical, showsIndicators: false) {
.lineLimit(2) // Match the viewport height while short, then grow upward one
.multilineTextAlignment(.center) // complete line at a time as the live transcript wraps.
.truncationMode(.tail) Text(text)
.frame(maxWidth: .infinity) .font(TypeStyle.body)
.foregroundStyle(color)
.multilineTextAlignment(.center)
.fixedSize(horizontal: false, vertical: true)
.frame(
maxWidth: .infinity,
minHeight: geometry.size.height,
alignment: .center
)
.id("assistant.status.text")
}
.scrollDisabled(true)
.scrollEdgeEffectHidden()
.onAppear {
proxy.scrollTo("assistant.status.text", anchor: .bottom)
}
.onChange(of: text) { _, _ in
// ASR partials update per word. Jump only by the resulting
// layout delta instead of animating every partial update.
proxy.scrollTo("assistant.status.text", anchor: .bottom)
}
}
}
.frame(height: Layout.statusViewportHeight)
.frame(maxWidth: .infinity)
} }
private var activeStatus: (text: String, color: Color)? { private var activeStatus: (text: String, color: Color)? {
@@ -11,11 +11,43 @@ import os
import OSGKeyboardShared import OSGKeyboardShared
#endif #endif
/// A primed streaming session is opened before the utterance has an owner.
/// Route its callbacks through a mutable relay, then bind the real partial
/// consumer when the mic gesture is committed to an utterance.
private final class CloudASRPartialRelay: @unchecked Sendable {
private let lock = OSAllocatedUnfairLock<
(@Sendable (String) -> Void)?
>(initialState: nil)
func bind(_ consumer: @escaping @Sendable (String) -> Void) {
lock.withLock { $0 = consumer }
}
func publish(_ text: String) {
let consumer = lock.withLock { $0 }
consumer?(text)
}
}
/// Uploads PCM only on the user-selected cloud engine path; provider clients /// Uploads PCM only on the user-selected cloud engine path; provider clients
/// reject missing credentials before network transmission. Personal dictionary /// reject missing credentials before network transmission. Personal dictionary
/// entries are sent as recognition bias. Mutable client/cancellation state is /// entries are sent as recognition bias. Mutable client/cancellation state is
/// lock-protected, which is the basis for `@unchecked Sendable`. /// lock-protected, which is the basis for `@unchecked Sendable`.
public final class CloudASRService: ASRService, @unchecked Sendable { public final class CloudASRService: ASRService, @unchecked Sendable {
private struct PrimedStreamingSession {
let id: UUID
let selection: String
let localeID: String
let createdAt: ContinuousClock.Instant
let relay: CloudASRPartialRelay
let task: Task<any CloudASRStreamingSession, Error>
}
/// The gateway closes a session that receives no audio for 15 s (counted
/// from WebSocket connect). A primed session older than this is never
/// adopted, leaving headroom for the first audio frame to arrive.
public static let defaultPrimedSessionMaxAge: Duration = .seconds(10)
private let store: any ConfigurationStore private let store: any ConfigurationStore
private let session: URLSession private let session: URLSession
private let localFallback: ASRService private let localFallback: ASRService
@@ -23,24 +55,28 @@ public final class CloudASRService: ASRService, @unchecked Sendable {
/// provider-specific local-fallback selection unchanged. /// provider-specific local-fallback selection unchanged.
private let managedClient: (any CloudASRTranscribing)? private let managedClient: (any CloudASRTranscribing)?
private let managedGrants: GatewayGrantCoordinator? private let managedGrants: GatewayGrantCoordinator?
private let primedSessionMaxAge: Duration
private let lock = OSAllocatedUnfairLock() private let lock = OSAllocatedUnfairLock()
private var client: CloudASRTranscribing? private var client: CloudASRTranscribing?
private var usesLocalFallback = false private var usesLocalFallback = false
private var boundClientSelection: String? private var boundClientSelection: String?
private var cancelled = false private var cancelled = false
private var streamingPipeline: StreamingUtterancePipeline? private var streamingPipeline: StreamingUtterancePipeline?
private var primedStreamingSession: PrimedStreamingSession?
public init( public init(
store: any ConfigurationStore = AppGroupStore(), store: any ConfigurationStore = AppGroupStore(),
session: URLSession = .shared, session: URLSession = .shared,
localFallback: ASRService? = nil, localFallback: ASRService? = nil,
managedClient: (any CloudASRTranscribing)? = nil, managedClient: (any CloudASRTranscribing)? = nil,
managedGrants: GatewayGrantCoordinator? = nil managedGrants: GatewayGrantCoordinator? = nil,
primedSessionMaxAge: Duration = CloudASRService.defaultPrimedSessionMaxAge
) { ) {
self.store = store self.store = store
self.session = session self.session = session
self.managedClient = managedClient self.managedClient = managedClient
self.managedGrants = managedGrants self.managedGrants = managedGrants
self.primedSessionMaxAge = primedSessionMaxAge
// `SpeechAnalyzerASR` is internal, so it can't appear in a public // `SpeechAnalyzerASR` is internal, so it can't appear in a public
// default argument value — resolve the fallback in the body instead. // default argument value — resolve the fallback in the body instead.
self.localFallback = localFallback ?? SpeechAnalyzerASR() self.localFallback = localFallback ?? SpeechAnalyzerASR()
@@ -77,6 +113,143 @@ public final class CloudASRService: ASRService, @unchecked Sendable {
} }
} }
/// Starts the account-managed Volcengine reservation + WebSocket handshake
/// while the user's finger is still down. The utterance adopts this session
/// moments later, removing those network round trips from first-partial
/// latency without sending audio before recording actually begins.
public func primeUtteranceStreaming(locale: Locale) {
bindClientIfNeeded()
guard store.credentialSource == .managed,
let selection = lock.withLock({ boundClientSelection }),
let streamingClient = lock.withLock({
client as? any CloudASRStreamingCapable
}) else {
return
}
let localeID = locale.identifier(.bcp47)
let alreadyPrimed = lock.withLock { () -> Bool in
guard let current = primedStreamingSession else { return false }
return current.selection == selection
&& current.localeID == localeID
&& !isExpired(current)
}
guard !alreadyPrimed else { return }
cancelPrimedUtteranceStreaming()
let id = UUID()
let relay = CloudASRPartialRelay()
let dictionary = store.personalDictionary
let task = Task.detached(priority: .userInitiated) {
try await streamingClient.openStreamingSession(
locale: locale,
dictionary: dictionary,
onPartial: { relay.publish($0) }
)
}
lock.withLock {
primedStreamingSession = PrimedStreamingSession(
id: id,
selection: selection,
localeID: localeID,
createdAt: .now,
relay: relay,
task: task
)
}
// Self-expire: the keyboard's cancel command may never arrive (the
// extension can be dismissed, or a start can fail before adoption).
let maxAge = primedSessionMaxAge
Task { [weak self] in
try? await Task.sleep(for: maxAge)
guard let self, let expired = self.takePrimedSession(id: id) else { return }
FlowTrace.asr("cloud.stream.primeDiscarded", "reason=expired")
Self.close(expired)
}
FlowTrace.asr(
"cloud.stream.primeStarted",
"locale=\(localeID) selection=\(selection)"
)
}
public func cancelPrimedUtteranceStreaming() {
guard let primed = takePrimedSession() else { return }
Self.close(primed)
}
/// Removes the primed session; with an `id`, only if it is still that one.
private func takePrimedSession(id: UUID? = nil) -> PrimedStreamingSession? {
lock.withLock {
guard let current = primedStreamingSession,
id == nil || current.id == id else {
return nil
}
primedStreamingSession = nil
return current
}
}
private func isExpired(_ primed: PrimedStreamingSession) -> Bool {
ContinuousClock.now - primed.createdAt >= primedSessionMaxAge
}
/// Closes a primed session without cancelling its open task. Cancelling
/// mid-reservation can leave a reserved, never-connected gateway session
/// holding credits and a concurrency slot until its connect timeout;
/// letting the open finish and then closing releases it immediately.
private static func close(_ primed: PrimedStreamingSession) {
Task {
if let session = try? await primed.task.value {
session.cancel()
}
}
}
/// Returns the primed session only if it still matches this utterance and
/// can accept audio. Anything else is closed and the caller opens a fresh
/// session, so priming can never make an utterance fail.
private func adoptPrimedSession(
_ primed: PrimedStreamingSession?,
selection: String?,
localeID: String,
onPartial: @escaping @Sendable (String) -> Void
) async -> (any CloudASRStreamingSession)? {
guard let primed else { return nil }
guard primed.selection == selection, primed.localeID == localeID else {
FlowTrace.asr("cloud.stream.primeDiscarded", "reason=mismatch")
Self.close(primed)
return nil
}
guard !isExpired(primed) else {
FlowTrace.asr("cloud.stream.primeDiscarded", "reason=expired")
Self.close(primed)
return nil
}
primed.relay.bind(onPartial)
let session: any CloudASRStreamingSession
do {
session = try await primed.task.value
} catch {
FlowTrace.asr(
"cloud.stream.primeDiscarded",
"reason=openFailed \(CloudASRLogMetadata.describe(error))"
)
return nil
}
guard session.acceptsAudio else {
FlowTrace.asr("cloud.stream.primeDiscarded", "reason=closed")
session.cancel()
return nil
}
let ageMs = Int((ContinuousClock.now - primed.createdAt) / .milliseconds(1))
FlowTrace.asr(
"cloud.stream.primeAdopted",
"locale=\(localeID) ageMs=\(ageMs)"
)
return session
}
public func transcribeChunk(samples: [Float], locale: Locale) async -> ASRChunkResult { public func transcribeChunk(samples: [Float], locale: Locale) async -> ASRChunkResult {
guard !samples.isEmpty else { return .success("") } guard !samples.isEmpty else { return .success("") }
if Task.isCancelled || lock.withLock({ cancelled }) { return .cancelled } if Task.isCancelled || lock.withLock({ cancelled }) { return .cancelled }
@@ -129,23 +302,42 @@ public final class CloudASRService: ASRService, @unchecked Sendable {
onPartial: @escaping @Sendable (String) -> Void onPartial: @escaping @Sendable (String) -> Void
) async -> ChunkedUtterancePipelineOutcome { ) async -> ChunkedUtterancePipelineOutcome {
bindClientIfNeeded() bindClientIfNeeded()
// A previous utterance's abort/failure leaves `cancelled` set, which
// would make this utterance's batch fallback (`transcribeChunk`)
// return `.cancelled` for the rest of the Flow session.
lock.withLock { cancelled = false }
if usesLocalFallback { if usesLocalFallback {
let pipeline = ChunkedUtterancePipeline(asr: localFallback, locale: locale) let pipeline = ChunkedUtterancePipeline(asr: localFallback, locale: locale)
return await pipeline.transcribe(stream: stream, onPartial: onPartial) return await pipeline.transcribe(stream: stream, onPartial: onPartial)
} }
guard let streamingClient = lock.withLock({ client as? CloudASRStreamingCapable }) else { guard let streamingClient = lock.withLock({ client as? CloudASRStreamingCapable }) else {
cancelPrimedUtteranceStreaming()
let pipeline = ChunkedUtterancePipeline(asr: self, locale: locale) let pipeline = ChunkedUtterancePipeline(asr: self, locale: locale)
return await pipeline.transcribe(stream: stream, onPartial: onPartial) return await pipeline.transcribe(stream: stream, onPartial: onPartial)
} }
let selection = lock.withLock { boundClientSelection }
let localeID = locale.identifier(.bcp47)
// Every utterance either adopts the primed session or closes it.
let primed = takePrimedSession()
let session: any CloudASRStreamingSession let session: any CloudASRStreamingSession
do { do {
session = try await streamingClient.openStreamingSession( if let adopted = await adoptPrimedSession(
locale: locale, primed,
dictionary: store.personalDictionary, selection: selection,
localeID: localeID,
onPartial: onPartial onPartial: onPartial
) ) {
session = adopted
} else {
session = try await streamingClient.openStreamingSession(
locale: locale,
dictionary: store.personalDictionary,
onPartial: onPartial
)
}
} catch where ProviderToolCancellation.matches(error) { } catch where ProviderToolCancellation.matches(error) {
return .cancelled return .cancelled
} catch { } catch {
@@ -249,6 +441,7 @@ public final class CloudASRService: ASRService, @unchecked Sendable {
public func cancel() { public func cancel() {
lock.withLock { cancelled = true } lock.withLock { cancelled = true }
cancelPrimedUtteranceStreaming()
let pipeline = lock.withLock { streamingPipeline } let pipeline = lock.withLock { streamingPipeline }
Task { await pipeline?.cancel() } Task { await pipeline?.cancel() }
localFallback.cancel() localFallback.cancel()
@@ -259,8 +452,10 @@ public final class CloudASRService: ASRService, @unchecked Sendable {
let strategy = CloudASRModelCatalog.strategy(for: providerId) let strategy = CloudASRModelCatalog.strategy(for: providerId)
let credentialSource = store.credentialSource let credentialSource = store.credentialSource
let selection = "\(credentialSource.rawValue):\(providerId)" let selection = "\(credentialSource.rawValue):\(providerId)"
lock.withLock { let stalePrime = lock.withLock { () -> PrimedStreamingSession? in
guard boundClientSelection != selection else { return } guard boundClientSelection != selection else { return nil }
let stalePrime = primedStreamingSession
primedStreamingSession = nil
boundClientSelection = selection boundClientSelection = selection
if credentialSource == .managed { if credentialSource == .managed {
usesLocalFallback = false usesLocalFallback = false
@@ -269,12 +464,16 @@ public final class CloudASRService: ASRService, @unchecked Sendable {
session: session, session: session,
managedGrants: managedGrants managedGrants: managedGrants
) )
return return stalePrime
} }
usesLocalFallback = strategy == .localFallback usesLocalFallback = strategy == .localFallback
client = usesLocalFallback client = usesLocalFallback
? nil ? nil
: CloudASRClientFactory.make(store: store, session: session) : CloudASRClientFactory.make(store: store, session: session)
return stalePrime
}
if let stalePrime {
Self.close(stalePrime)
} }
} }
} }
@@ -51,6 +51,13 @@ public protocol CloudASRStreamingSession: Sendable {
/// Signal end-of-audio and wait for the polish-ready final transcript. /// Signal end-of-audio and wait for the polish-ready final transcript.
func finish() async throws -> String func finish() async throws -> String
func cancel() func cancel()
/// False once the session has failed, closed, or been cancelled, so a
/// pre-opened session is not adopted after its transport already died.
var acceptsAudio: Bool { get }
}
public extension CloudASRStreamingSession {
var acceptsAudio: Bool { true }
} }
/// Providers that can open an utterance-level streaming session. /// Providers that can open an utterance-level streaming session.
@@ -765,6 +765,12 @@ private final class ManagedVolcengineASRSession: CloudASRStreamingSession, @unch
} }
} }
var acceptsAudio: Bool {
lock.withLock { state in
!state.cancelled && state.failure == nil && !state.receiveClosed && !state.endSent
}
}
func cancel() { func cancel() {
let changed = lock.withLock { state -> Bool in let changed = lock.withLock { state -> Bool in
guard !state.cancelled else { return false } guard !state.cancelled else { return false }
@@ -10,13 +10,18 @@ public enum UtteranceBatchFallbackPolicy {
public static let defaultCharacterAdvantage = UtteranceTranscriptGuard.defaultPartialAdvantage public static let defaultCharacterAdvantage = UtteranceTranscriptGuard.defaultPartialAdvantage
/// True when chunked output likely lost trailing content vs the live partial. /// True when chunked output likely lost trailing content vs the live partial.
///
/// `recognitionStalled` means the ASR wait expired before the recognizer
/// finished (e.g. the streaming socket went quiet). That is a failure, not
/// silence, so the captured PCM still deserves a batch attempt.
public static func shouldRunBatchFallback( public static func shouldRunBatchFallback(
stitchedFinal: String, stitchedFinal: String,
partialSnapshot: String, partialSnapshot: String,
recognitionFailed: Bool = false, recognitionFailed: Bool = false,
recognitionStalled: Bool = false,
minimumCharacterAdvantage: Int = defaultCharacterAdvantage minimumCharacterAdvantage: Int = defaultCharacterAdvantage
) -> Bool { ) -> Bool {
if recognitionFailed { return true } if recognitionFailed || recognitionStalled { return true }
let final = stitchedFinal.trimmingCharacters(in: .whitespacesAndNewlines) let final = stitchedFinal.trimmingCharacters(in: .whitespacesAndNewlines)
let partial = partialSnapshot.trimmingCharacters(in: .whitespacesAndNewlines) let partial = partialSnapshot.trimmingCharacters(in: .whitespacesAndNewlines)
+212
View File
@@ -30,6 +30,80 @@ private final class RecordingFallbackASR: ASRService, @unchecked Sendable {
} }
} }
private final class RecordingStreamingSession: CloudASRStreamingSession, @unchecked Sendable {
private let onPartial: @Sendable (String) -> Void
/// Models a transport the gateway already closed (e.g. idle timeout).
private let isDead: Bool
init(onPartial: @escaping @Sendable (String) -> Void, isDead: Bool = false) {
self.onPartial = onPartial
self.isDead = isDead
}
var acceptsAudio: Bool { !isDead }
func append(samples: [Float]) async throws {
if isDead { throw ManagedCloudASRError.idleTimeout }
if !samples.isEmpty {
onPartial("你好")
}
}
func finish() async throws -> String {
if isDead { throw ManagedCloudASRError.idleTimeout }
return "你好"
}
func cancel() {}
}
/// Sessions are numbered by open order, starting at 1.
private final class RecordingStreamingClient: CloudASRStreamingCapable, @unchecked Sendable {
private let lock = OSAllocatedUnfairLock(initialState: 0)
private let deadSessions: Set<Int>
private let failingOpens: Set<Int>
init(deadSessions: Set<Int> = [], failingOpens: Set<Int> = []) {
self.deadSessions = deadSessions
self.failingOpens = failingOpens
}
var openCount: Int {
lock.withLock { $0 }
}
func prepare(dictionary: PersonalDictionary) async throws {}
func transcribe(
samples: [Float],
sampleRate: Int,
locale: Locale,
dictionary: PersonalDictionary
) async throws -> String {
"batch"
}
func openStreamingSession(
locale: Locale,
dictionary: PersonalDictionary,
onPartial: @escaping @Sendable (String) -> Void
) async throws -> any CloudASRStreamingSession {
_ = locale
_ = dictionary
let index = lock.withLock { count -> Int in
count += 1
return count
}
if failingOpens.contains(index) {
throw ManagedCloudASRError.sessionTransportFailed
}
return RecordingStreamingSession(
onPartial: onPartial,
isDead: deadSessions.contains(index)
)
}
}
final class CloudASRServiceTests: XCTestCase { final class CloudASRServiceTests: XCTestCase {
private var suiteName: String! private var suiteName: String!
@@ -93,4 +167,142 @@ final class CloudASRServiceTests: XCTestCase {
XCTAssertEqual(text, "") XCTAssertEqual(text, "")
XCTAssertEqual(fallback.calls, 0) XCTAssertEqual(fallback.calls, 0)
} }
func testManagedStreamingAdoptsPrimedSession() async {
let client = RecordingStreamingClient()
let service = makeManagedService(client: client)
service.primeUtteranceStreaming(locale: zhLocale)
let partials = OSAllocatedUnfairLock(initialState: [String]())
let outcome = await service.transcribeUtteranceStreaming(
stream: oneSnapshotStream(),
locale: zhLocale
) { partial in
partials.withLock { $0.append(partial) }
}
guard case .success(let success) = outcome else {
return XCTFail("expected streaming success, got \(outcome)")
}
XCTAssertEqual(success.text, "你好")
XCTAssertEqual(partials.withLock { $0 }, ["你好"])
XCTAssertEqual(client.openCount, 1)
}
func testDeadPrimedSessionIsReplacedWithFreshSession() async {
let client = RecordingStreamingClient(deadSessions: [1])
let service = makeManagedService(client: client)
service.primeUtteranceStreaming(locale: zhLocale)
await waitUntil { client.openCount == 1 }
let outcome = await service.transcribeUtteranceStreaming(
stream: oneSnapshotStream(),
locale: zhLocale
) { _ in }
guard case .success(let success) = outcome else {
return XCTFail("expected fresh streaming success, got \(outcome)")
}
XCTAssertEqual(success.text, "你好")
XCTAssertEqual(client.openCount, 2)
}
func testExpiredPrimedSessionIsNotAdopted() async {
let client = RecordingStreamingClient(deadSessions: [1])
let service = makeManagedService(client: client, primedSessionMaxAge: .zero)
service.primeUtteranceStreaming(locale: zhLocale)
await waitUntil { client.openCount == 1 }
let outcome = await service.transcribeUtteranceStreaming(
stream: oneSnapshotStream(),
locale: zhLocale
) { _ in }
guard case .success(let success) = outcome else {
return XCTFail("expected fresh streaming success, got \(outcome)")
}
XCTAssertEqual(success.text, "你好")
XCTAssertEqual(client.openCount, 2)
}
func testFailedPrimeOpensFreshStreamingSession() async {
let client = RecordingStreamingClient(failingOpens: [1])
let service = makeManagedService(client: client)
service.primeUtteranceStreaming(locale: zhLocale)
let outcome = await service.transcribeUtteranceStreaming(
stream: oneSnapshotStream(),
locale: zhLocale
) { _ in }
// "你好" comes from streaming; the chunked batch fallback returns "batch".
guard case .success(let success) = outcome else {
return XCTFail("expected fresh streaming success, got \(outcome)")
}
XCTAssertEqual(success.text, "你好")
XCTAssertEqual(client.openCount, 2)
}
func testStreamingUtteranceClearsCancellationForBatchFallback() async {
let client = RecordingStreamingClient()
let service = makeManagedService(client: client)
service.cancel()
_ = await service.transcribeUtteranceStreaming(
stream: oneSnapshotStream(),
locale: zhLocale
) { _ in }
let batch = await service.transcribeChunk(
samples: [Float](repeating: 0.1, count: 1_600),
locale: zhLocale
)
guard case .success(let text) = batch else {
return XCTFail("expected batch success after a new utterance, got \(batch)")
}
XCTAssertEqual(text, "batch")
}
// MARK: - Helpers
private let zhLocale = Locale(identifier: "zh-Hans")
private func makeManagedService(
client: RecordingStreamingClient,
primedSessionMaxAge: Duration = CloudASRService.defaultPrimedSessionMaxAge
) -> CloudASRService {
var config = AppGroupConfiguration.load(fromAvailable: defaults)
config.engineMode = "cloud"
config.credentialSource = .managed
config.save(to: defaults)
store = AppGroupStore(defaults: defaults)
return CloudASRService(
store: store,
session: .shared,
localFallback: RecordingFallbackASR(),
managedClient: client,
primedSessionMaxAge: primedSessionMaxAge
)
}
private func oneSnapshotStream() -> AsyncStream<AudioBufferSnapshot> {
let (stream, continuation) = AsyncStream<AudioBufferSnapshot>.makeStream()
continuation.yield(
AudioBufferSnapshot(
samples: [Float](repeating: 0.1, count: 1_600),
sampleRate: 16_000
)
)
continuation.finish()
return stream
}
/// Lets the detached prime task run before the utterance starts.
private func waitUntil(_ condition: () -> Bool) async {
var attempts = 0
while !condition(), attempts < 200 {
attempts += 1
try? await Task.sleep(for: .milliseconds(5))
}
}
} }
@@ -43,6 +43,25 @@ final class UtteranceBatchFallbackPolicyTests: XCTestCase {
) )
} }
func testShouldRunWhenRecognitionStalledWithoutAnyText() {
XCTAssertTrue(
UtteranceBatchFallbackPolicy.shouldRunBatchFallback(
stitchedFinal: "",
partialSnapshot: "",
recognitionStalled: true
)
)
}
func testShouldNotRunForCompletedSilence() {
XCTAssertFalse(
UtteranceBatchFallbackPolicy.shouldRunBatchFallback(
stitchedFinal: "",
partialSnapshot: ""
)
)
}
func testPreferredTranscriptPicksLongestCandidate() { func testPreferredTranscriptPicksLongestCandidate() {
let resolved = UtteranceBatchFallbackPolicy.preferredTranscript( let resolved = UtteranceBatchFallbackPolicy.preferredTranscript(
batch: "今天很好,我们一起去公园吧", batch: "今天很好,我们一起去公园吧",