feat(app): add resilient Flow recovery and privacy-safe analytics
CI / Validate manifests (push) Has been cancelled
CI / SwiftLint (push) Has been cancelled
CI / iOS / Extension (push) Has been cancelled
CI / macOS (push) Has been cancelled

Unify PiP recovery across startup and foreground transitions, surface actionable status, and add privacy-safe analytics plus the refreshed onboarding and account experience. Update documentation assets and advance the release build to 84.
This commit is contained in:
Rocky
2026-08-20 20:22:20 +08:00
parent ee7de5c934
commit 10bc457c72
69 changed files with 7891 additions and 598 deletions
+4
View File
@@ -115,6 +115,10 @@
<array>
<string>audio</string>
</array>
<key>BGTaskSchedulerPermittedIdentifiers</key>
<array>
<string>com.osgkeyboard.ios.analytics-sync</string>
</array>
<key>UILaunchScreen</key>
<dict>
<key>UIColorName</key>
+39
View File
@@ -73,6 +73,45 @@
<string>NSPrivacyCollectedDataTypePurposeAppFunctionality</string>
</array>
</dict>
<dict>
<!-- App-scoped random installation UUID used for first-party analytics. -->
<key>NSPrivacyCollectedDataType</key>
<string>NSPrivacyCollectedDataTypeDeviceID</string>
<key>NSPrivacyCollectedDataTypeLinked</key>
<true/>
<key>NSPrivacyCollectedDataTypeTracking</key>
<false/>
<key>NSPrivacyCollectedDataTypePurposes</key>
<array>
<string>NSPrivacyCollectedDataTypePurposeAnalytics</string>
</array>
</dict>
<dict>
<!-- Fixed app, keyboard, AI, purchase-page and referral events. -->
<key>NSPrivacyCollectedDataType</key>
<string>NSPrivacyCollectedDataTypeProductInteraction</string>
<key>NSPrivacyCollectedDataTypeLinked</key>
<true/>
<key>NSPrivacyCollectedDataTypeTracking</key>
<false/>
<key>NSPrivacyCollectedDataTypePurposes</key>
<array>
<string>NSPrivacyCollectedDataTypePurposeAnalytics</string>
</array>
</dict>
<dict>
<!-- Coarse feature result, failure category and duration bucket only. -->
<key>NSPrivacyCollectedDataType</key>
<string>NSPrivacyCollectedDataTypeOtherUsageData</string>
<key>NSPrivacyCollectedDataTypeLinked</key>
<true/>
<key>NSPrivacyCollectedDataTypeTracking</key>
<false/>
<key>NSPrivacyCollectedDataTypePurposes</key>
<array>
<string>NSPrivacyCollectedDataTypePurposeAnalytics</string>
</array>
</dict>
</array>
<key>NSPrivacyAccessedAPITypes</key>
<array>
+8 -4
View File
@@ -22,7 +22,7 @@
<p class="lang"><a href="#zh">中文</a> · <a href="#top">English</a></p>
<div id="top">
<h1>OSGKeyboard Privacy Policy</h1>
<p><strong>Last updated:</strong> August 19, 2026 · <em>v2.0</em></p>
<p><strong>Last updated:</strong> August 20, 2026 · <em>v2.0</em></p>
<p>OSGKeyboard provides an iOS custom keyboard and a macOS menu-bar dictation app. iOS uses Apple's on-device speech APIs by default; macOS can use a downloaded local Qwen3-ASR model. The core local and user-configured-provider features do not require an OSGKeyboard account. On iOS, you may optionally sign in with Apple and choose managed credits instead of supplying provider credentials. This policy explains both paths.</p>
<h2>What we collect</h2>
@@ -35,6 +35,7 @@
<li><strong>API credentials</strong> — stored in the iOS Keychain and shared between the main app and keyboard extension. When iCloud settings sync is enabled, keys replicate through iCloud Keychain (not iCloud KVS JSON).</li>
<li><strong>Optional OSGKeyboard account and purchases</strong> — no account is required for local transcription or user-configured providers. If you choose Sign in with Apple, the account service processes the Apple identity token, a pseudonymous OSGKeyboard account identifier, optional display name, device-integrity evidence, credit balance, referral state, and StoreKit transaction identifiers needed to verify consumable credit purchases. Account session tokens stay in the main app's private Keychain; the keyboard extension receives only a short-lived, scope-limited managed-service grant. You can sign out or delete the account in the app.</li>
<li><strong>App preferences</strong> — engine mode, language, and keyboard settings stored in App Group UserDefaults. Optional iCloud sync mirrors eligible preferences, usage statistics, and voice history through your private iCloud account. Clipboard-history consent and its suggestion-strip switch stay device-local and are not activated by iCloud settings sync.</li>
<li><strong>Optional first-party product analytics</strong> — enabled by default and removable at any time in Settings. OSGKeyboard sends only fixed event names and fixed categories about app/keyboard sessions, AI feature outcomes and duration ranges, purchase-page interactions, and invitation actions, together with an app-scoped random installation identifier and safe app/OS version labels. It never includes keyboard input, audio, transcripts, cursor context, prompts, model output, contacts, file or device names, Apple identifiers, email, credentials, advertising identifiers, or free-form properties. Turning analytics off deletes queued events; turning it back on creates a new installation identifier.</li>
<li><strong>Optional clipboard history</strong> — off by default. When you turn it on, the keyboard may read text from the clipboard on this device or from Universal Clipboard; iOS does not provide a reliable way to distinguish those sources. Up to 15 accepted text items are stored only in the local App Group shared by this devices host app and keyboard extension. Turning history off stops capture, turns off the suggestion strip, and keeps existing items. Reset Settings also keeps them; only the separate confirmed “Clear clipboard history” action removes them. There is no fixed expiry. Secure fields immediately hide clipboard UI and are not captured. Conservative filters reject common OTP shapes, private-key headers, JWTs, Bearer tokens, recognizable provider-key prefixes, and common Luhn-valid 16-digit card numbers, but cannot identify every password or secret. A rejected item can still be pasted through iOS; it is simply not added to history. Clipboard history is not automatically sent to AI. If you insert an item and then actively use polish, the inserted text may be included as context sent to the provider you configured.</li>
<li><strong>Personal dictionary and typing learning</strong> — personal terms may sync through your private iCloud account when enabled. Chinese candidate frequencies and English suggestion choices remain in the local App Group and can be cleared separately without deleting the personal dictionary. OSGKeyboard does not upload this learning data.</li>
<li><strong>macOS local speech models</strong> — when selected, Qwen3-ASR model files are downloaded from Hugging Face and remain in the Mac app's local application-support directory until you remove them. Local inference does not send your audio to Hugging Face or OSGKeyboard.</li>
@@ -44,7 +45,7 @@
<h2>What we do not collect</h2>
<ul>
<li>We do <strong>not</strong> log or upload ordinary keystrokes.</li>
<li>We do <strong>not</strong> operate analytics, crash-reporting, or advertising SDKs.</li>
<li>We do <strong>not</strong> use third-party analytics, crash-reporting, advertising SDKs, ATT, or IDFA.</li>
<li>We do <strong>not</strong> sell personal data.</li>
</ul>
@@ -62,6 +63,7 @@
<p>Settings remain on your device until you delete the app or reset settings. With iCloud sync enabled, API keys use iCloud Keychain; eligible preferences, statistics, dictionary data, and voice history may sync through your private iCloud account.</p>
<p><strong>Voice history</strong> — successful transcripts may be saved in the main apps History tab (up to 300 entries). With iCloud settings sync enabled, history may also sync across your devices.</p>
<p><strong>Clipboard history</strong> — stays in this devices App Group, is capped at 15 entries, does not sync through iCloud, and has no fixed expiry. Turning the feature off or resetting settings keeps existing items. Use the separately confirmed clear action to delete them.</p>
<p><strong>Product analytics</strong> — queued events expire locally after 34 days and the service accepts events no older than 35 days. Account deletion and analytics opt-out delete the local queue. Analytics identifiers are not used for cross-app tracking or advertising.</p>
<p><strong>Typing learning and Mac models</strong> — Chinese and English learning data stays local and can be cleared from typing settings. Downloaded Mac speech models remain until you remove the model or delete its local files.</p>
<p><strong>Optional account</strong> — account profile, credit ledger, StoreKit audit records, and limited anti-abuse records are retained by the account service while the account is active. Account deletion removes the active profile and credentials; pseudonymous ledger, purchase-audit, and time-limited anti-abuse records may remain where required to prevent replay, fraud, or abuse.</p>
@@ -71,7 +73,7 @@
<hr id="zh">
<h1>OSGKeyboard 隐私政策</h1>
<p><strong>更新日期:</strong>2026 年 8 月 19 日 · <em>v2.0</em></p>
<p><strong>更新日期:</strong>2026 年 8 月 20 日 · <em>v2.0</em></p>
<p>OSGKeyboard 包含 iOS 自定义键盘与 macOS 菜单栏听写应用。iOS 默认使用 Apple 端侧语音能力;macOS 可使用下载到本机的 Qwen3-ASR 模型。本地能力和用户自备服务商能力无需登录 OSGKeyboard 账号。iOS 用户也可以选择 Apple 登录并使用托管积分,无需自行填写服务商凭证。本政策同时说明这两条可选路径。</p>
<h2>我们处理的数据</h2>
@@ -84,6 +86,7 @@
<li><strong>API 凭证</strong> — 保存在设备 Keychain,在主 App 与键盘扩展间共享。开启 iCloud 设置同步后,经 iCloud 钥匙串同步(非 iCloud KVS JSON)。</li>
<li><strong>可选 OSGKeyboard 账号与购买</strong> — 本地转写和用户自备服务商无需账号。若你选择 Apple 登录,账号服务会处理 Apple 身份令牌、匿名化的 OSGKeyboard 账号标识、可选昵称、设备完整性证明、积分余额、邀请状态,以及核验消耗型积分包所需的 StoreKit 交易标识。账号会话令牌只保存在主 App 私有 Keychain;键盘扩展仅获得短时、限权的托管服务凭证。你可以在 App 内退出或注销账号。</li>
<li><strong>应用偏好</strong> — 引擎、语言等设置保存在 App Group。可选 iCloud 同步经私有 iCloud 账户镜像可同步的偏好、统计与语音历史。剪贴板历史采集许可与建议条开关仅属于本机,不会被 iCloud 设置同步开启。</li>
<li><strong>可选第一方产品分析</strong> — 默认开启,可随时在设置中关闭。OSGKeyboard 只发送固定事件名和固定分类,用于了解 App/键盘会话、AI 功能结果与耗时区间、购买页交互及邀请操作,并携带 App 范围内随机生成的安装标识和安全的 App/系统版本标识。绝不包含键盘输入、音频、转录、光标上下文、Prompt、模型输出、联系人、文件名、设备名、Apple 标识、邮箱、凭证、广告标识符或自由形式属性。关闭后会删除待发送事件;重新开启会生成新的安装标识。</li>
<li><strong>可选剪贴板历史</strong> — 默认关闭。开启后,键盘可能读取本机剪贴板或通用剪贴板中的文字;iOS 无法可靠区分两者来源。最多 15 条通过规则的文本仅保存在本机主 App 与键盘扩展共享的 App Group。关闭历史只会停止采集、关闭建议条并保留已有记录;重置设置同样不会清除,只有单独确认的「清空剪贴板历史」操作会删除。历史没有固定过期时间。进入安全输入框会立即隐藏剪贴板入口与正文,且不会采集。保守过滤会拒绝常见 OTP 形态、私钥头、JWT、Bearer Token、具有明确服务商前缀的密钥以及常见的通过 Luhn 校验的 16 位卡号,但无法识别所有密码或秘密。被拒绝的内容仍可通过 iOS 一次性粘贴,只是不进入历史。剪贴板历史不会自动发送给 AI;插入后若主动使用润色,已插入文字可能作为上下文发送给你配置的服务商。</li>
<li><strong>个性词库与本地输入学习</strong> — 个性词条可在开启后经你的私有 iCloud 账户同步。中文候选频率与英文候选选择偏好保存在本机 App Group,可在输入设置中单独清除且不影响个性词库,OSGKeyboard 不会上传这些学习数据。</li>
<li><strong>macOS 本地语音模型</strong> — 选择后,Qwen3-ASR 模型会从 Hugging Face 下载到 Mac 本地应用支持目录,直到你主动移除。本地推理不会把录音发送给 Hugging Face 或 OSGKeyboard。</li>
@@ -93,7 +96,7 @@
<h2>我们不收集的内容</h2>
<ul>
<li>我们<strong>不会</strong>记录或上传你平时在键盘上的击键内容。</li>
<li>我们<strong>不会</strong>集成分析、崩溃上报或广告 SDK。</li>
<li>我们<strong>不会</strong>集成第三方分析、崩溃上报或广告 SDK,也不使用 ATT 或 IDFA</li>
<li>我们<strong>不会</strong>出售个人数据。</li>
</ul>
@@ -111,6 +114,7 @@
<p>设置保留在设备上,直至卸载或重置。开启 iCloud 同步后,API 密钥走 iCloud 钥匙串;可同步的偏好、统计、词库与语音历史可能经私有 iCloud 账户同步。</p>
<p><strong>语音历史</strong> — 成功转写可保存在主 App「历史」页(最多 300 条)。开启 iCloud 设置同步后,历史也可能在多设备间同步。</p>
<p><strong>剪贴板历史</strong> — 仅保存在本机 App Group,上限 15 条,不经 iCloud 同步,也没有固定过期时间。关闭功能或重置设置会保留已有记录;需使用单独确认的清空操作才能删除。</p>
<p><strong>产品分析</strong> — 待发送事件会在本地满 34 天后过期,服务端只接受 35 天以内的事件。注销账号或关闭产品分析会删除本地队列。分析标识不会用于跨 App 跟踪或广告。</p>
<p><strong>输入学习与 Mac 模型</strong> — 中英文输入学习数据仅留本机,可在输入设置中单独清除。下载的 Mac 语音模型会保留到你移除模型或删除本地文件。</p>
<p><strong>可选账号</strong> — 账号有效期间,账号服务会保留资料、积分账本、StoreKit 审计记录及有限的反滥用记录。注销会删除有效资料与凭证;为防止交易重放、欺诈或滥用,匿名账本、购买审计及有时限的反滥用记录可能在必要范围内继续保留。</p>
@@ -0,0 +1,284 @@
// AnalyticsHostService.swift
// OSGKeyboard · Main App
//
// Host-only analytics lifecycle. The shared module owns the wire contract,
// SQLite queue and uploader; this type only connects iOS lifecycle signals.
import BackgroundTasks
import Foundation
import Network
import OSGKeyboardHostSupport
import OSGKeyboardShared
import OSLog
import UIKit
actor AnalyticsUploadSignal: AnalyticsUploadTriggering {
typealias Action = @Sendable () async -> Void
private var action: Action?
private var isScheduled = false
private let debounce: Duration
init(debounce: Duration = .milliseconds(250)) {
self.debounce = debounce
}
func install(_ action: @escaping Action) {
self.action = action
}
func requestUpload() {
guard !isScheduled, let action else { return }
isScheduled = true
Task {
try? await Task.sleep(for: debounce)
guard !Task.isCancelled else {
uploadFinished()
return
}
await action()
uploadFinished()
}
}
private func uploadFinished() {
isScheduled = false
}
}
final class HostAnalyticsBearerBridge: AnalyticsBearerProviding, @unchecked Sendable {
static let shared = HostAnalyticsBearerBridge()
private let lock = NSLock()
private var provider: (any AnalyticsBearerProviding)?
func install(_ provider: any AnalyticsBearerProviding) {
lock.lock()
self.provider = provider
lock.unlock()
}
func bearerToken() async throws -> String? {
try await currentProvider()?.bearerToken()
}
func refreshBearerToken(
afterUnauthorizedAccessToken failedToken: String?
) async throws -> String? {
try await currentProvider()?.refreshBearerToken(
afterUnauthorizedAccessToken: failedToken
)
}
private func currentProvider() -> (any AnalyticsBearerProviding)? {
lock.lock()
defer { lock.unlock() }
return provider
}
}
struct AccountAnalyticsBearerProvider: AnalyticsBearerProviding {
let apiClient: AccountAPIClient
func bearerToken() async throws -> String? {
guard try await apiClient.currentSession() != nil else { return nil }
return try await apiClient.accessTokenForAuthorizedRequest()
}
func refreshBearerToken(
afterUnauthorizedAccessToken failedToken: String?
) async throws -> String? {
guard let failedToken, !failedToken.isEmpty else { return nil }
return try await apiClient.refreshAccessToken(
afterUnauthorizedAccessToken: failedToken
)
}
}
struct HostAnalyticsLogger: AnalyticsLogging {
private let logger = Logger(
subsystem: Bundle.main.bundleIdentifier ?? "com.osgkeyboard.ios",
category: "analytics"
)
func log(_ entry: AnalyticsUploadLogEntry) {
let statusCode = entry.statusCode ?? 0
let errorCategory = entry.errorCategory?.rawValue ?? "none"
logger.info(
"outcome=\(entry.outcome.rawValue, privacy: .public) count=\(entry.eventCount, privacy: .public) status=\(statusCode, privacy: .public) attempt=\(entry.attempt, privacy: .public) error=\(errorCategory, privacy: .public)"
)
}
}
@MainActor
final class AnalyticsHostService: ObservableObject {
static let shared = AnalyticsHostService()
static let backgroundTaskIdentifier = "com.osgkeyboard.ios.analytics-sync"
@Published private(set) var isEnabled = true
let client: any AnalyticsClient
private let runtime: AnalyticsRuntime
private let uploadSignal: AnalyticsUploadSignal
private let pathMonitor = NWPathMonitor()
private let monitorQueue = DispatchQueue(
label: "com.osgkeyboard.analytics.network",
qos: .utility
)
private var isMonitoringNetwork = false
private var foregroundUploadTask: Task<Void, Never>?
init(
bearerProvider: any AnalyticsBearerProviding = HostAnalyticsBearerBridge.shared,
network: any AnalyticsNetworking = URLSessionAnalyticsNetwork()
) {
let signal = AnalyticsUploadSignal()
uploadSignal = signal
let runtime = AnalyticsRuntime.mainApp(
environment: Self.environment,
uploadConfiguration: AnalyticsUploadConfiguration(endpoint: Self.endpoint),
network: network,
bearerProvider: bearerProvider,
trigger: signal,
logger: HostAnalyticsLogger()
)
self.runtime = runtime
client = runtime.client
Task {
await signal.install {
await runtime.uploadCoordinator.uploadAvailableEvents(maximumBatches: 4)
}
}
}
func prepare(firstOpenAcquisitionChannel: AnalyticsAcquisitionChannel) {
startNetworkMonitoringIfNeeded()
Task {
await runtime.prepare(
firstOpenAcquisitionChannel: firstOpenAcquisitionChannel
)
let enabled = await runtime.isEnabled()
await MainActor.run {
self.isEnabled = enabled
}
client.recordSessionActivity()
await runtime.uploadCoordinator.uploadAvailableEvents(maximumBatches: 8)
}
}
func appDidBecomeActive() {
client.recordSessionActivity()
requestImmediateUpload(maximumBatches: 8)
}
func appDidEnterBackground() {
Self.scheduleBackgroundRefresh()
let identifier = UIApplication.shared.beginBackgroundTask(
withName: "analytics-sync"
)
guard identifier != .invalid else { return }
let task = Task {
await runtime.uploadCoordinator.uploadAvailableEvents(maximumBatches: 2)
await MainActor.run {
UIApplication.shared.endBackgroundTask(identifier)
}
}
foregroundUploadTask = task
}
func observeAuthenticatedAccount(_ accountID: UUID) async {
_ = await runtime.observeAccount(stableIdentifier: accountID.uuidString)
await runtime.uploadCoordinator.uploadAvailableEvents(maximumBatches: 8)
}
func handleAccountDeletion() async {
await runtime.handleAccountDeletion()
}
func setEnabled(_ enabled: Bool) {
Task {
await runtime.setEnabled(enabled)
let current = await runtime.isEnabled()
await MainActor.run {
self.isEnabled = current
}
if current {
await runtime.uploadCoordinator.uploadAvailableEvents(maximumBatches: 4)
}
}
}
func refreshEnabledState() {
Task {
let current = await runtime.isEnabled()
await MainActor.run {
self.isEnabled = current
}
}
}
func requestImmediateUpload(maximumBatches: Int = 4) {
foregroundUploadTask?.cancel()
foregroundUploadTask = Task {
await runtime.uploadCoordinator.uploadAvailableEvents(
maximumBatches: maximumBatches
)
}
}
static func registerBackgroundTask() {
BGTaskScheduler.shared.register(
forTaskWithIdentifier: backgroundTaskIdentifier,
using: nil
) { task in
guard let refreshTask = task as? BGAppRefreshTask else {
task.setTaskCompleted(success: false)
return
}
let operation = Task {
await AnalyticsHostService.shared.runtime.uploadCoordinator
.uploadAvailableEvents(maximumBatches: 4)
refreshTask.setTaskCompleted(success: !Task.isCancelled)
scheduleBackgroundRefresh()
}
refreshTask.expirationHandler = {
operation.cancel()
}
}
}
static func scheduleBackgroundRefresh() {
let request = BGAppRefreshTaskRequest(identifier: backgroundTaskIdentifier)
request.earliestBeginDate = Date(timeIntervalSinceNow: 15 * 60)
try? BGTaskScheduler.shared.submit(request)
}
private func startNetworkMonitoringIfNeeded() {
guard !isMonitoringNetwork else { return }
isMonitoringNetwork = true
pathMonitor.pathUpdateHandler = { path in
guard path.status == .satisfied else { return }
Task { @MainActor in
AnalyticsHostService.shared.requestImmediateUpload(maximumBatches: 8)
}
}
pathMonitor.start(queue: monitorQueue)
}
private static let endpoint = URL(
string: "https://account.osglab.com/v1/analytics/events"
)!
private static var environment: AnalyticsEnvironment {
let appVersion = Bundle.main.object(
forInfoDictionaryKey: "CFBundleShortVersionString"
) as? String ?? "unknown"
let version = ProcessInfo.processInfo.operatingSystemVersion
return AnalyticsEnvironment(
appVersion: appVersion,
osVersion: "\(version.majorVersion).\(version.minorVersion).\(version.patchVersion)"
)
}
}
+10
View File
@@ -43,6 +43,16 @@ final class AppOpenURLRouter {
}
final class AppURLHandler: NSObject, UIApplicationDelegate {
func application(
_ application: UIApplication,
didFinishLaunchingWithOptions launchOptions: [
UIApplication.LaunchOptionsKey: Any
]? = nil
) -> Bool {
AnalyticsHostService.registerBackgroundTask()
return true
}
/// SwiftUI `@main` apps get no scene delegate by default. Attach ours so
/// scene-based URL delivery the only iOS 26 path to `sourceApplication`
/// reaches `AppSceneDelegate`. We deliberately do NOT create a window here;
@@ -36,14 +36,45 @@ enum FlowPiPStartOutcome: Equatable, Sendable {
case failed(FlowPiPStartFailure)
}
/// User-facing lifecycle published by `FlowSessionManager`.
enum FlowPiPLifecycleState: Equatable, Sendable {
case inactive
case preparing(attempt: Int, total: Int)
case recovering(attempt: Int, total: Int)
case active
case waitingForForeground
case failed(FlowPiPStartFailure)
}
@MainActor
final class FlowPictureInPictureController: NSObject {
/// User closed the PiP window host should end the Flow session.
var onUserDismissed: (() -> Void)?
protocol FlowPictureInPictureControlling: AnyObject {
var onUnexpectedStop: (() -> Void)? { get set }
var isPictureInPictureActive: Bool { get }
var generation: UInt64 { get }
func attachHostView(_ view: UIView)
func startAndWait(
hostTimeout: TimeInterval,
activeTimeout: TimeInterval
) async -> FlowPiPStartOutcome
func stop()
func resetGeneration()
func prepareForBackgroundAutoStart() async
func reassertKeepAliveAudioSession() async -> Bool
func updateWaveformLevels(_ levels: [Float])
}
@MainActor
final class FlowPictureInPictureController: NSObject, FlowPictureInPictureControlling {
/// PiP disappeared without an in-process teardown. AVKit deliberately does
/// not distinguish a user close from a system stop or another app takeover.
var onUnexpectedStop: (() -> Void)?
private(set) var isPictureInPictureActive = false
/// True once a host UIView has been attached (may still be awaiting a window).
private(set) var hasHostView = false
/// Monotonic identity for the currently configured AVKit controller.
private(set) var generation: UInt64 = 0
let displayLayer = AVSampleBufferDisplayLayer()
@@ -115,29 +146,7 @@ final class FlowPictureInPictureController: NSObject {
@discardableResult
func start() async -> Bool {
lastSystemStartFailure = nil
guard AVPictureInPictureController.isPictureInPictureSupported() else {
return false
}
guard hasHostView else { return false }
// Required before constructing the controller; without an active
// session, `isPictureInPicturePossible` stays false forever.
guard await activateAudioSessionForPiP() else {
return false
}
// If a controller was somehow created before audio activation, rebuild.
if pipController != nil, !didActivateAudioSessionBeforeController {
pipController = nil
}
configureControllerIfNeeded()
if !usesLowPowerVideoCallPiP {
warmLogoCacheIfNeeded()
animationStartedAt = CACurrentMediaTime()
startFramePump()
}
guard pipController != nil else { return false }
guard await prepareControllerForStart() else { return false }
if pipController?.isPictureInPictureActive == true {
isPictureInPictureActive = true
@@ -145,6 +154,7 @@ final class FlowPictureInPictureController: NSObject {
return true
}
guard pipController?.isPictureInPicturePossible == true else { return false }
if !usesLowPowerVideoCallPiP {
// Prime a few frames before asking the system to start PiP.
enqueueGuideFrame()
@@ -164,17 +174,23 @@ final class FlowPictureInPictureController: NSObject {
) async -> FlowPiPStartOutcome {
if isPictureInPictureActive { return .started }
guard AVPictureInPictureController.isPictureInPictureSupported() else {
let deadline = Date().addingTimeInterval(activeTimeout)
let supportsPiP = await waitForPictureInPictureSupport(
timeout: min(max(0, deadline.timeIntervalSinceNow), 0.8)
)
guard supportsPiP else {
return .failed(.unsupported)
}
let hostReady = await waitForHostInWindow(timeout: hostTimeout)
let hostReady = await waitForHostInWindow(
timeout: min(hostTimeout, max(0, deadline.timeIntervalSinceNow))
)
guard hostReady else {
return .failed(.hostNotReady)
}
lastSystemStartFailure = nil
guard await start() else {
guard await prepareControllerForStart() else {
stopFramePump()
if lastSystemStartFailure != nil {
return .failed(.systemRejected)
@@ -182,17 +198,49 @@ final class FlowPictureInPictureController: NSObject {
return .failed(hasHostView ? .notPossible : .hostNotReady)
}
let deadline = Date().addingTimeInterval(activeTimeout)
while Date() < deadline {
if isPictureInPictureActive { return .started }
if pipController?.isPictureInPictureActive == true {
isPictureInPictureActive = true
return .started
}
if let pipController, pipController.isPictureInPicturePossible {
pipController.startPictureInPicture()
} else {
if pipController?.isPictureInPicturePossible == true {
break
}
try? await Task.sleep(nanoseconds: 50_000_000)
}
guard pipController?.isPictureInPicturePossible == true else {
FlowDiagnostics.log(
"PiP generation \(generation) never became possible"
)
return .failed(.notPossible)
}
if !usesLowPowerVideoCallPiP {
// Prime a few frames before asking the system to start PiP.
enqueueGuideFrame()
enqueueGuideFrame()
pipController?.invalidatePlaybackState()
}
// Issue immediately when possible. iOS 27 beta can silently ignore a
// request without sending either delegate callback, so retry at a
// conservative cadence while keeping this controller generation alive.
pipController?.startPictureInPicture()
var lastStartRequestAt = Date()
var startRequestCount = 1
while Date() < deadline {
if isPictureInPictureActive { return .started }
if pipController?.isPictureInPictureActive == true {
isPictureInPictureActive = true
return .started
}
if pipController?.isPictureInPicturePossible == true,
Date().timeIntervalSince(lastStartRequestAt) >= 0.25 {
pipController?.startPictureInPicture()
lastStartRequestAt = Date()
startRequestCount += 1
}
try? await Task.sleep(nanoseconds: 50_000_000)
}
@@ -213,16 +261,24 @@ final class FlowPictureInPictureController: NSObject {
failure = .timedOut
}
FlowDiagnostics.log(
"PiP startAndWait failed: \(failure) possible=\(pipController?.isPictureInPicturePossible == true)"
"PiP startAndWait failed: \(failure) possible=\(pipController?.isPictureInPicturePossible == true) "
+ "requests=\(startRequestCount)"
)
stop()
return .failed(failure)
}
func stop() {
resetGeneration()
}
/// Fully releases the AVKit content source so the next attempt cannot
/// inherit a controller that the system has already rejected.
func resetGeneration() {
isStoppingProgrammatically = true
stopFramePump()
pipController?.stopPictureInPicture()
let controller = pipController
controller?.delegate = nil
controller?.stopPictureInPicture()
if !usesLowPowerVideoCallPiP {
displayLayer.sampleBufferRenderer.flush(
removingDisplayedImage: true,
@@ -232,6 +288,10 @@ final class FlowPictureInPictureController: NSObject {
isPictureInPictureActive = false
animationStartedAt = nil
lastSystemStartFailure = nil
pipController = nil
videoCallContentController = nil
transparentContentView = nil
didActivateAudioSessionBeforeController = false
isStoppingProgrammatically = false
}
@@ -277,6 +337,32 @@ final class FlowPictureInPictureController: NSObject {
/// Set once we successfully activate audio before building the controller.
private var didActivateAudioSessionBeforeController = false
private func prepareControllerForStart() async -> Bool {
lastSystemStartFailure = nil
guard AVPictureInPictureController.isPictureInPictureSupported() else {
return false
}
guard hasHostView else { return false }
// Required before constructing the controller; without an active
// session, `isPictureInPicturePossible` stays false forever.
guard await activateAudioSessionForPiP() else {
return false
}
// If a controller was somehow created before audio activation, rebuild.
if pipController != nil, !didActivateAudioSessionBeforeController {
resetGeneration()
}
configureControllerIfNeeded()
if !usesLowPowerVideoCallPiP {
warmLogoCacheIfNeeded()
animationStartedAt = CACurrentMediaTime()
startFramePump()
}
return pipController != nil
}
@discardableResult
private func activateAudioSessionForPiP() async -> Bool {
// The first PiP start uses playback. After capture establishes a
@@ -289,6 +375,24 @@ final class FlowPictureInPictureController: NSObject {
return false
}
private func waitForPictureInPictureSupport(timeout: TimeInterval) async -> Bool {
if AVPictureInPictureController.isPictureInPictureSupported() {
return true
}
#if targetEnvironment(simulator)
return false
#else
let deadline = Date().addingTimeInterval(timeout)
while Date() < deadline {
try? await Task.sleep(nanoseconds: 200_000_000)
if AVPictureInPictureController.isPictureInPictureSupported() {
return true
}
}
return AVPictureInPictureController.isPictureInPictureSupported()
#endif
}
private func waitForHostInWindow(timeout: TimeInterval) async -> Bool {
if isHostInWindowHierarchy { return true }
let deadline = Date().addingTimeInterval(timeout)
@@ -341,7 +445,9 @@ final class FlowPictureInPictureController: NSObject {
controller.canStartPictureInPictureAutomaticallyFromInline = true
controller.requiresLinearPlayback = true
pipController = controller
generation &+= 1
didActivateAudioSessionBeforeController = true
FlowDiagnostics.log("PiP controller generation \(generation) configured")
}
private func releaseAudioSessionForLowPowerPiP() {
@@ -642,6 +748,7 @@ extension FlowPictureInPictureController: @preconcurrency AVPictureInPictureCont
func pictureInPictureControllerDidStartPictureInPicture(
_ pictureInPictureController: AVPictureInPictureController
) {
guard pictureInPictureController === pipController else { return }
isPictureInPictureActive = true
lastSystemStartFailure = nil
releaseAudioSessionForLowPowerPiP()
@@ -650,16 +757,18 @@ extension FlowPictureInPictureController: @preconcurrency AVPictureInPictureCont
func pictureInPictureControllerDidStopPictureInPicture(
_ pictureInPictureController: AVPictureInPictureController
) {
guard pictureInPictureController === pipController else { return }
isPictureInPictureActive = false
stopFramePump()
guard !isStoppingProgrammatically else { return }
onUserDismissed?()
onUnexpectedStop?()
}
func pictureInPictureController(
_ pictureInPictureController: AVPictureInPictureController,
failedToStartPictureInPictureWithError error: Error
) {
guard pictureInPictureController === pipController else { return }
// Sample-buffer fallback may need warm-up retries. VideoCall PiP uses
// the automatic-inline path plus the bounded startAndWait fallback.
lastSystemStartFailure = error
+440 -88
View File
@@ -31,6 +31,67 @@ enum FlowUtteranceLifecyclePolicy {
}
}
enum FlowPiPActivationTrigger: String, Sendable {
case coldStart
case foreground
case existingSession
case startupRecovery
case unexpectedStop
case healthCheck
case manualRetry
}
enum FlowPiPReconciliationDecision: Equatable, Sendable {
case noSessionIntent
case alreadyActive
case waitForForeground
case startRecovery
}
enum FlowPiPRecoveryPolicy {
static let maxAttempts = 3
static let totalBudget: TimeInterval = 5
static func reconciliationDecision(
wantsActiveSession: Bool,
isAppForeground: Bool,
isPictureInPictureActive: Bool
) -> FlowPiPReconciliationDecision {
guard wantsActiveSession else { return .noSessionIntent }
if isPictureInPictureActive { return .alreadyActive }
return isAppForeground ? .startRecovery : .waitForForeground
}
static func canContinue(
operation: UInt64,
currentOperation: UInt64,
wantsActiveSession: Bool,
isAppForeground: Bool,
taskIsCancelled: Bool
) -> Bool {
operation == currentOperation
&& wantsActiveSession
&& isAppForeground
&& !taskIsCancelled
}
static func retryDelay(beforeAttempt attempt: Int) -> TimeInterval {
switch attempt {
case 1: return 0
case 2: return 0.25
default: return 0.6
}
}
static func hostTimeout(attempt: Int, remainingBudget: TimeInterval) -> TimeInterval {
min(attempt == 1 ? 1.5 : 0.25, max(0, remainingBudget))
}
static func activeTimeout(remainingBudget: TimeInterval) -> TimeInterval {
min(1.4, max(0, remainingBudget))
}
}
@MainActor
final class FlowSessionManager: ObservableObject {
private struct PrefilledAIRequest {
@@ -45,12 +106,14 @@ final class FlowSessionManager: ObservableObject {
@Published private(set) var sessionExpiresAt: Date?
/// Non-nil when continuous capture failed or permissions are missing.
@Published private(set) var sessionWarning: String?
@Published private(set) var pipLifecycleState: FlowPiPLifecycleState = .inactive
private let capture = FlowContinuousCapture()
private let pipController = FlowPictureInPictureController()
private let pipController: any FlowPictureInPictureControlling
private let store = AppGroupStore()
private let analyticsClient: any AnalyticsClient
/// Cloud / local polish via the user-configured LLM provider.
private let polisher = PolishingService()
private let polisher: PolishingService
/// AI-mode turns are intentionally process-local and never persisted.
private let aiConversations = AIConversationStore()
/// Cached ASR instance. The only on-device backend is iOS
@@ -98,6 +161,7 @@ final class FlowSessionManager: ObservableObject {
private var pendingSourceHistoryEntryID: UUID?
private var pendingSourceHistoryEntryRevision: Int64?
private var pendingAIConversationID: UUID?
private var pendingManagedRequestPurpose: ManagedGatewayRequestPurpose?
private var pendingProcessingDeadlineAt: TimeInterval?
private var pendingStopUtteranceId: UUID?
private var currentCommandSeq: Int64 = 0
@@ -110,6 +174,7 @@ final class FlowSessionManager: ObservableObject {
@Published private(set) var isUtteranceProcessing = false
private var finalizeTask: Task<Void, Never>?
private var asrTask: Task<Void, Never>?
private var transcriptionAnalyticsOperation: (any AnalyticsAIOperation)?
private var utteranceSafetyTask: Task<Void, Never>?
private var chunkedPipeline: ChunkedUtterancePipeline?
private var currentPartial = ""
@@ -131,7 +196,10 @@ final class FlowSessionManager: ObservableObject {
private var isAppForeground = false
/// True while handling a keyboard-initiated `startflow` cold start.
private var isColdStartHandoff = false
private var coldStartRecoveryTask: Task<Void, Never>?
/// User intent outlives a transient PiP loss; only explicit teardown clears it.
private var wantsActiveSession = false
/// Invalidates late async completions after retry, stop, or process teardown.
private var pipRecoveryOperation: UInt64 = 0
var shouldDeferHostHeavyWork: Bool {
isStarting
|| startingUtteranceId != nil
@@ -148,7 +216,13 @@ final class FlowSessionManager: ObservableObject {
/// recreate the `@StateObject`-owned manager within the same process).
private static var didRunLaunchReconciliation = false
init() {
init(
analyticsClient: any AnalyticsClient = NoopAnalyticsClient(),
pipController: any FlowPictureInPictureControlling = FlowPictureInPictureController()
) {
self.analyticsClient = analyticsClient
self.polisher = PolishingService(analyticsClient: analyticsClient)
self.pipController = pipController
// Sessions are (re)started explicitly on app foreground via
// `activateOnForeground()`. We deliberately do NOT silently reattach a
// stored session here after a force-quit that would resurrect capture
@@ -194,10 +268,9 @@ final class FlowSessionManager: ObservableObject {
kind: .recognitionInterrupted
)
}
pipController.onUserDismissed = { [weak self] in
guard let self, self.isActive else { return }
self.debug("PiP dismissed by user — ending Flow session")
self.endSession()
pipController.onUnexpectedStop = { [weak self] in
guard let self, self.wantsActiveSession else { return }
self.handleUnexpectedPiPStop()
}
FlowTerminationCoordinator.register(self)
}
@@ -257,25 +330,25 @@ final class FlowSessionManager: ObservableObject {
}
if isActive {
wantsActiveSession = true
extendSession(duration: duration)
refreshHostReady()
if pipController.isPictureInPictureActive {
pipLifecycleState = .active
refreshHostReady()
} else {
ensurePiPActive(
trigger: coldStart ? .coldStart : .existingSession,
duration: duration
)
}
return
}
guard !isStarting else {
traceState("startSession.ignored", extra: "reason=alreadyStarting")
return
}
// Claim the flag synchronously: on a cold start the URL router and
// `activateOnForeground()` both fire in the same runloop turn, and
// setting it inside the async body let two start bodies interleave.
isStarting = true
startTask?.cancel()
startTask = Task { @MainActor [weak self] in
await self?.startSessionAsync(duration: duration)
self?.handleColdStartAfterSessionReady()
}
wantsActiveSession = true
ensurePiPActive(
trigger: coldStart ? .coldStart : .foreground,
duration: duration
)
}
/// Clears App Group Flow state left behind when the host process was killed
@@ -339,9 +412,19 @@ final class FlowSessionManager: ObservableObject {
)
}
/// User-requested recovery always discards the rejected AVKit generation.
func retryPiPRecovery() {
guard AppGroup.isAvailable else { return }
guard AppPermissions.flowRequirementsMet else {
sessionWarning = permissionWarningMessage()
FlowSessionBridge.setHostReady(false)
return
}
wantsActiveSession = true
ensurePiPActive(trigger: .manualRetry, duration: nil, forceReset: true)
}
private func completeColdStartHandoff() {
coldStartRecoveryTask?.cancel()
coldStartRecoveryTask = nil
isColdStartHandoff = false
if isActive {
refreshHostReady()
@@ -360,9 +443,10 @@ final class FlowSessionManager: ObservableObject {
)
}
wantsActiveSession = false
pipRecoveryOperation &+= 1
pipLifecycleState = .inactive
isColdStartHandoff = false
coldStartRecoveryTask?.cancel()
coldStartRecoveryTask = nil
startTask?.cancel()
startTask = nil
startupAudioHealthTask?.cancel()
@@ -433,7 +517,7 @@ final class FlowSessionManager: ObservableObject {
}
func endSession() {
guard isActive else { return }
guard isActive || isStarting || wantsActiveSession else { return }
debug("Flow session ended")
if isUtteranceRecording || isUtteranceProcessing,
claimTerminal(utteranceId: currentUtteranceId) {
@@ -444,9 +528,10 @@ final class FlowSessionManager: ObservableObject {
)
}
wantsActiveSession = false
pipRecoveryOperation &+= 1
pipLifecycleState = .inactive
isColdStartHandoff = false
coldStartRecoveryTask?.cancel()
coldStartRecoveryTask = nil
startTask?.cancel()
startTask = nil
startupAudioHealthTask?.cancel()
@@ -493,6 +578,7 @@ final class FlowSessionManager: ObservableObject {
FlowSessionBridge.markSessionInactive()
FlowSessionDarwin.postSessionChanged()
isActive = false
isStarting = false
sessionExpiresAt = nil
sessionWarning = nil
currentPartial = ""
@@ -516,19 +602,27 @@ final class FlowSessionManager: ObservableObject {
case .active:
setAppForeground(true)
resumeAfterForeground()
if wantsActiveSession, !pipController.isPictureInPictureActive {
ensurePiPActive(trigger: .foreground, duration: nil)
}
case .inactive:
setAppForeground(false)
writeHeartbeatIfActive()
if isActive {
if isActive, pipController.isPictureInPictureActive {
Task { @MainActor [weak self] in
await self?.pipController.prepareForBackgroundAutoStart()
}
} else if wantsActiveSession {
pausePiPRecoveryUntilForeground()
}
case .background:
setAppForeground(false)
if isActive {
if isActive, pipController.isPictureInPictureActive {
Task { @MainActor [weak self] in
await self?.pipController.prepareForBackgroundAutoStart()
}
} else if wantsActiveSession {
pausePiPRecoveryUntilForeground()
}
if isColdStartHandoff {
completeColdStartHandoff()
@@ -549,6 +643,33 @@ final class FlowSessionManager: ObservableObject {
FlowSessionBridge.writeHeartbeat()
}
private func handleUnexpectedPiPStop() {
guard wantsActiveSession else { return }
traceState("pip.unexpectedStop")
pipLifecycleState = .waitingForForeground
sessionWarning = nil
if isUtteranceRecording {
failUtterance(
message: AppL10n.string("flow.error.recognitionInterrupted"),
kind: .recognitionInterrupted
)
}
FlowSessionBridge.setHostReady(false)
refreshHostReady()
// Do not immediately fight another app for the system-owned PiP slot.
// The scene-active path or the existing heartbeat performs reconciliation.
}
private func pausePiPRecoveryUntilForeground() {
startTask?.cancel()
startTask = nil
pipRecoveryOperation &+= 1
pipController.resetGeneration()
isStarting = false
pipLifecycleState = .waitingForForeground
refreshHostReady()
}
private func resumeAfterForeground() {
guard isActive else {
return
@@ -723,36 +844,192 @@ final class FlowSessionManager: ObservableObject {
// MARK: - Session start
private func startSessionAsync(duration: TimeInterval?) async {
// `isStarting` was claimed synchronously in `startSession()`.
defer { isStarting = false }
private func ensurePiPActive(
trigger: FlowPiPActivationTrigger,
duration: TimeInterval?,
forceReset: Bool = false
) {
switch FlowPiPRecoveryPolicy.reconciliationDecision(
wantsActiveSession: wantsActiveSession,
isAppForeground: isAppForeground,
isPictureInPictureActive: pipController.isPictureInPictureActive
) {
case .noSessionIntent:
return
case .alreadyActive:
pipLifecycleState = .active
sessionWarning = nil
if isActive {
refreshHostReady()
} else {
activateFlowSessionAfterPiPProof(duration: duration)
}
handleColdStartAfterSessionReady()
return
case .waitForForeground:
pipLifecycleState = .waitingForForeground
FlowSessionBridge.setHostReady(false)
refreshHostReady()
return
case .startRecovery:
break
}
guard startTask == nil else {
traceState(
"pipRecovery.coalesced",
extra: "trigger=\(trigger.rawValue)"
)
return
}
// Claim the flag synchronously: on a cold start the URL router and
// `activateOnForeground()` both fire in the same runloop turn, and
// setting it inside the async body let two start bodies interleave.
isStarting = true
pipRecoveryOperation &+= 1
let operation = pipRecoveryOperation
let recoveringExistingSession = isActive
let resetFirstGeneration = forceReset
|| recoveringExistingSession
|| pipController.generation > 0
pipLifecycleState = recoveringExistingSession
? .recovering(attempt: 1, total: FlowPiPRecoveryPolicy.maxAttempts)
: .preparing(attempt: 1, total: FlowPiPRecoveryPolicy.maxAttempts)
startTask = Task { @MainActor [weak self] in
await self?.startSessionAsync(
duration: duration,
trigger: trigger,
operation: operation,
resetFirstGeneration: resetFirstGeneration
)
}
}
private func startSessionAsync(
duration: TimeInterval?,
trigger: FlowPiPActivationTrigger,
operation: UInt64,
resetFirstGeneration: Bool
) async {
// `isStarting` was claimed synchronously in `ensurePiPActive()`.
defer {
if operation == pipRecoveryOperation {
isStarting = false
startTask = nil
}
}
guard !Task.isCancelled else { return }
traceState("startSessionAsync.begin")
traceState("startSessionAsync.begin", extra: "trigger=\(trigger.rawValue)")
sessionWarning = nil
guard AppPermissions.flowRequirementsMet else {
sessionWarning = permissionWarningMessage()
pipLifecycleState = .failed(.notPossible)
traceState("startSessionAsync.blocked", extra: "reason=permissions")
FlowSessionBridge.setHostReady(false)
isColdStartHandoff = false
return
}
switch await pipController.startAndWait() {
case .started:
activateFlowSessionAfterPiPProof(duration: duration)
traceState("startSessionAsync.ready")
debug("Flow session started (PiP keep-alive), mic released between utterances")
case .failed(let failure):
let message = AppL10n.string(failure.localizationKey)
sessionWarning = message
traceState("startSessionAsync.failed", extra: "reason=pipUnavailable failure=\(failure)")
FlowSessionBridge.setHostReady(false)
if isColdStartHandoff {
scheduleColdStartRecovery(duration: duration)
let deadline = Date().addingTimeInterval(FlowPiPRecoveryPolicy.totalBudget)
var lastFailure: FlowPiPStartFailure = .timedOut
for attempt in 1...FlowPiPRecoveryPolicy.maxAttempts {
guard FlowPiPRecoveryPolicy.canContinue(
operation: operation,
currentOperation: pipRecoveryOperation,
wantsActiveSession: wantsActiveSession,
isAppForeground: isAppForeground,
taskIsCancelled: Task.isCancelled
) else {
if operation == pipRecoveryOperation,
wantsActiveSession,
!Task.isCancelled,
!isAppForeground {
pipLifecycleState = .waitingForForeground
FlowSessionBridge.setHostReady(false)
}
return
}
let delay = FlowPiPRecoveryPolicy.retryDelay(beforeAttempt: attempt)
if delay > 0 {
try? await Task.sleep(
nanoseconds: UInt64(delay * 1_000_000_000)
)
}
guard FlowPiPRecoveryPolicy.canContinue(
operation: operation,
currentOperation: pipRecoveryOperation,
wantsActiveSession: wantsActiveSession,
isAppForeground: isAppForeground,
taskIsCancelled: Task.isCancelled
) else { return }
if resetFirstGeneration || attempt > 1 {
pipController.resetGeneration()
}
let remaining = deadline.timeIntervalSinceNow
guard remaining > 0 else { break }
pipLifecycleState = isActive
? .recovering(attempt: attempt, total: FlowPiPRecoveryPolicy.maxAttempts)
: .preparing(attempt: attempt, total: FlowPiPRecoveryPolicy.maxAttempts)
traceState(
"pipRecovery.attempt",
extra: "trigger=\(trigger.rawValue) attempt=\(attempt) generation=\(pipController.generation)"
)
let outcome = await pipController.startAndWait(
hostTimeout: FlowPiPRecoveryPolicy.hostTimeout(
attempt: attempt,
remainingBudget: remaining
),
activeTimeout: FlowPiPRecoveryPolicy.activeTimeout(
remainingBudget: deadline.timeIntervalSinceNow
)
)
guard FlowPiPRecoveryPolicy.canContinue(
operation: operation,
currentOperation: pipRecoveryOperation,
wantsActiveSession: wantsActiveSession,
isAppForeground: isAppForeground,
taskIsCancelled: Task.isCancelled
) else { return }
switch outcome {
case .started:
sessionWarning = nil
pipLifecycleState = .active
if isActive {
refreshHostReady()
} else {
activateFlowSessionAfterPiPProof(duration: duration)
}
traceState(
"startSessionAsync.ready",
extra: "trigger=\(trigger.rawValue) attempt=\(attempt) generation=\(pipController.generation)"
)
handleColdStartAfterSessionReady()
debug("Flow session started (PiP keep-alive), mic released between utterances")
return
case .failed(let failure):
lastFailure = failure
traceState(
"startSessionAsync.failed",
extra: "reason=pipUnavailable trigger=\(trigger.rawValue) attempt=\(attempt) failure=\(failure)"
)
}
debug("PiP keep-alive failed to start: \(failure)")
}
pipController.resetGeneration()
pipLifecycleState = .failed(lastFailure)
sessionWarning = AppL10n.string(lastFailure.localizationKey)
FlowSessionBridge.setHostReady(false)
refreshHostReady()
debug("PiP keep-alive failed after bounded recovery: \(lastFailure)")
}
private func activateFlowSessionAfterPiPProof(duration: TimeInterval?) {
@@ -762,6 +1039,8 @@ final class FlowSessionManager: ObservableObject {
FlowSessionBridge.markSessionActivePersistent(sessionId: sessionId)
FlowSessionDarwin.postSessionChanged()
isActive = true
wantsActiveSession = true
pipLifecycleState = .active
// Low-profile PiP is a system-owned keep-alive surface. Keeping the
// display awake wastes far more power than the static PiP itself.
ScreenWakeLock.release()
@@ -862,18 +1141,10 @@ final class FlowSessionManager: ObservableObject {
guard isColdStartHandoff, isActive else { return }
sessionWarning = nil
if !pipController.isPictureInPictureActive {
switch await pipController.startAndWait() {
case .started:
break
case .failed(let failure):
let message = AppL10n.string(failure.localizationKey)
sessionWarning = message
FlowSessionBridge.setHostReady(false)
scheduleColdStartRecovery(duration: nil)
debug("existing PiP session failed cold-start restart: \(failure)")
return
}
ensurePiPActive(trigger: .existingSession, duration: nil)
return
}
pipLifecycleState = .active
refreshHostReady()
handleColdStartAfterSessionReady()
}
@@ -895,7 +1166,9 @@ final class FlowSessionManager: ObservableObject {
let message = AppL10n.string("flow.session.error.notPossible")
sessionWarning = message
FlowSessionBridge.setHostReady(false)
scheduleColdStartRecovery(duration: nil)
if !pipController.isPictureInPictureActive {
scheduleColdStartRecovery(duration: nil)
}
debug("cold-start blocked: host ready contract not published")
return
}
@@ -927,31 +1200,14 @@ final class FlowSessionManager: ObservableObject {
/// bounce the audio session and rebuild. Force-quit relaunches routinely
/// inherit stale mediaserverd state that only a rebuild clears.
private func scheduleColdStartRecovery(duration: TimeInterval?) {
coldStartRecoveryTask?.cancel()
coldStartRecoveryTask = Task { @MainActor [weak self] in
guard let self else { return }
// The failed controller has been fully torn down. Give the app
// scene and Pegasus service a short settling interval before the
// one bounded cold-start retry creates a new controller generation.
try? await Task.sleep(nanoseconds: 400_000_000)
guard !Task.isCancelled, self.isColdStartHandoff else { return }
let outcome = await self.pipController.startAndWait()
self.traceState("coldStartRecovery.pip", extra: "outcome=\(outcome)")
guard !Task.isCancelled, self.isColdStartHandoff else { return }
switch outcome {
case .started:
if self.isActive {
self.sessionWarning = nil
self.refreshHostReady()
self.handleColdStartAfterSessionReady()
} else {
self.activateFlowSessionAfterPiPProof(duration: duration)
self.handleColdStartAfterSessionReady()
}
case .failed(let failure):
self.sessionWarning = AppL10n.string(failure.localizationKey)
}
}
// The failed controller has been fully torn down. Give the app
// scene and Pegasus service a short settling interval before the
// one bounded cold-start retry creates a new controller generation.
ensurePiPActive(
trigger: .startupRecovery,
duration: duration,
forceReset: true
)
}
private func bindSessionASRIfNeeded(force: Bool = false) {
@@ -1245,6 +1501,7 @@ final class FlowSessionManager: ObservableObject {
)
)
currentUtteranceMode = command.resolvedUtteranceMode
pendingManagedRequestPurpose = command.managedRequestPurpose
pendingAIConversationID = currentUtteranceMode == .aiQuestion
? command.aiConversationID
: nil
@@ -1357,6 +1614,7 @@ final class FlowSessionManager: ObservableObject {
commandSeq: command.commandSeq
) else { return }
currentUtteranceMode = .aiQuestion
pendingManagedRequestPurpose = command.managedRequestPurpose
pendingAIConversationID = command.aiConversationID
pendingEditSourceText = nil
pendingSourceHistoryEntryID = nil
@@ -1435,7 +1693,9 @@ final class FlowSessionManager: ObservableObject {
store: pipelineStore,
conversations: aiConversations,
taskKind: request.taskKind,
thinkingEnabled: request.thinkingEnabled
thinkingEnabled: request.thinkingEnabled,
analyticsClient: analyticsClient,
analyticsFeature: Self.analyticsFeature(for: request.taskKind)
)
aiAnswerStreamThrottle = AIAnswerStreamThrottle()
let answer = try await service.answer(
@@ -1610,6 +1870,13 @@ final class FlowSessionManager: ObservableObject {
kind: FlowSessionKeys.TranscriptionErrorKind = .generic,
status: FlowResult.Status = .error
) {
finishTranscriptionAnalytics(
success: false,
failureCategory: Self.transcriptionFailureCategory(
kind: kind,
status: status
)
)
guard let activeSessionId, let currentUtteranceId else { return }
FlowSessionBridge.writeResult(
FlowResult(
@@ -1901,6 +2168,11 @@ final class FlowSessionManager: ObservableObject {
)
let cloudASRForStreaming = useStreaming ? cloudService : nil
transcriptionAnalyticsOperation?.cancel()
transcriptionAnalyticsOperation = analyticsClient.startAIFeature(
.transcription,
executionMode: analyticsExecutionMode
)
asrTask = Task.detached(priority: .userInitiated) { [weak manager = self] in
let outcome: ChunkedUtterancePipelineOutcome
@@ -2220,6 +2492,7 @@ final class FlowSessionManager: ObservableObject {
pendingSourceHistoryEntryID = nil
pendingSourceHistoryEntryRevision = nil
pendingAIConversationID = nil
pendingManagedRequestPurpose = nil
pendingProcessingDeadlineAt = nil
currentUtteranceMode = .dictation
}
@@ -2237,6 +2510,7 @@ final class FlowSessionManager: ObservableObject {
let sourceHistoryEntryID = pendingSourceHistoryEntryID
let sourceHistoryEntryRevision = pendingSourceHistoryEntryRevision
let aiConversationID = pendingAIConversationID
let managedRequestPurpose = pendingManagedRequestPurpose
let processingDeadlineAt = pendingProcessingDeadlineAt
let asrEngineMode = sessionASREngineMode ?? store.engineMode
// ALWAYS clear the processing gate for this utterance. The previous
@@ -2250,6 +2524,7 @@ final class FlowSessionManager: ObservableObject {
pendingSourceHistoryEntryID = nil
pendingSourceHistoryEntryRevision = nil
pendingAIConversationID = nil
pendingManagedRequestPurpose = nil
pendingProcessingDeadlineAt = nil
if currentUtteranceMode == utteranceMode {
currentUtteranceMode = .dictation
@@ -2366,6 +2641,7 @@ final class FlowSessionManager: ObservableObject {
)
return
}
finishTranscriptionAnalytics(success: true)
// Re-read App Group at finalize so dictionary and translation changes
// from the keyboard extension are visible before local correction,
@@ -2427,7 +2703,9 @@ final class FlowSessionManager: ObservableObject {
let service = try AIQuestionService.configured(
store: pipelineStore,
conversations: aiConversations,
taskKind: .aiQuestion
taskKind: .aiQuestion,
analyticsClient: analyticsClient,
analyticsFeature: .aiAssistant
)
aiAnswerStreamThrottle = AIAnswerStreamThrottle()
let publishUtteranceId = finalizeUtteranceId
@@ -2564,6 +2842,7 @@ final class FlowSessionManager: ObservableObject {
systemPrompt: instructionPrompt?.system,
providerIdOverride: pipelineStore.polishProviderIdOverride,
taskKind: isEditLastInput ? .editLastInput : nil,
requestPurpose: managedRequestPurpose,
context: isInstructionMode ? nil : polishContext,
timeoutLimit: processingDeadlineAt.map {
max(0.1, $0 - Date().timeIntervalSince1970)
@@ -2868,6 +3147,13 @@ final class FlowSessionManager: ObservableObject {
status: FlowResult.Status = .error,
aiConversationID: UUID? = nil
) {
finishTranscriptionAnalytics(
success: false,
failureCategory: Self.transcriptionFailureCategory(
kind: kind,
status: status
)
)
guard let sessionId, let utteranceId else { return }
FlowTrace.warn(
"host.deliveredError",
@@ -2892,6 +3178,59 @@ final class FlowSessionManager: ObservableObject {
)
}
private static func analyticsFeature(
for taskKind: ManagedGatewayTaskKind
) -> AnalyticsFeature {
switch taskKind {
case .aiQuestion:
return .hotword
case .clipboardTransform, .customSkill, .agentPlanning:
return .agent
case .dictationPolish, .translation, .editLastInput:
return .other
}
}
private var analyticsExecutionMode: AnalyticsExecutionMode {
if store.engineMode == "local" {
return .local
}
return store.credentialSource == .managed ? .managed : .byok
}
private func finishTranscriptionAnalytics(
success: Bool,
failureCategory: AnalyticsFailureCategory = .unknown
) {
guard let operation = transcriptionAnalyticsOperation else { return }
transcriptionAnalyticsOperation = nil
if success {
operation.succeed()
} else {
operation.fail(category: failureCategory)
}
}
private static func transcriptionFailureCategory(
kind: FlowSessionKeys.TranscriptionErrorKind,
status: FlowResult.Status
) -> AnalyticsFailureCategory {
if status == .aborted {
return .cancelled
}
if status == .timeout {
return .timeout
}
switch kind {
case .noSpeech, .discardedEmpty:
return .validation
case .recognitionInterrupted:
return .cancelled
case .asrFailed, .audioUnavailable, .generic:
return .provider
}
}
private static var lastResultRevision: Int64 = 0
private static func resultRevision() -> Int64 {
@@ -3030,6 +3369,7 @@ final class FlowSessionManager: ObservableObject {
systemPrompt: String? = nil,
providerIdOverride: String?,
taskKind: ManagedGatewayTaskKind? = nil,
requestPurpose: ManagedGatewayRequestPurpose? = nil,
context: PolishContext?,
timeoutLimit: TimeInterval? = nil
) async throws -> PolishingService.PolishOutcome {
@@ -3042,6 +3382,7 @@ final class FlowSessionManager: ObservableObject {
systemPrompt: systemPrompt,
providerIdOverride: providerIdOverride,
taskKind: taskKind,
requestPurpose: requestPurpose,
context: context
)
}
@@ -3081,6 +3422,14 @@ final class FlowSessionManager: ObservableObject {
self.isUtteranceRecording || self.isUtteranceProcessing {
await self.reactivateCaptureIfNeeded()
}
if self.wantsActiveSession,
!self.pipController.isPictureInPictureActive {
if self.isAppForeground {
self.ensurePiPActive(trigger: .healthCheck, duration: nil)
} else {
self.pipLifecycleState = .waitingForForeground
}
}
FlowSessionBridge.writeHeartbeat()
self.refreshHostReady()
try? await Task.sleep(nanoseconds: 1_000_000_000)
@@ -3123,6 +3472,9 @@ final class FlowSessionManager: ObservableObject {
"event=\(event)",
"active=\(isActive)",
"starting=\(isStarting)",
"wantsActive=\(wantsActiveSession)",
"pipState=\(String(describing: pipLifecycleState))",
"pipGeneration=\(pipController.generation)",
"coldStart=\(isColdStartHandoff)",
"sessionId=\(sessionId)",
"utteranceId=\(utteranceId)",
@@ -505,6 +505,7 @@ private struct AccountCreditsSection: View {
private struct AccountCreditPurchaseSection: View {
@Environment(\.themePalette) private var palette
@ObservedObject var manager: AccountCreditPurchaseManager
@State private var didRecordPurchaseView = false
let accountID: UUID
let onGranted: () -> Void
@@ -540,6 +541,11 @@ private struct AccountCreditPurchaseSection: View {
guard current != nil, current != previous else { return }
onGranted()
}
.onAppear {
guard !didRecordPurchaseView else { return }
didRecordPurchaseView = true
manager.recordPurchaseViewed()
}
}
private func purchaseOption(_ option: AccountCreditPurchaseOption) -> some View {
@@ -777,10 +783,17 @@ private struct AccountInvitationActivityView: UIViewControllerRepresentable {
let invitationURL: URL
func makeUIViewController(context: Context) -> UIActivityViewController {
UIActivityViewController(
let controller = UIActivityViewController(
activityItems: [invitationURL],
applicationActivities: nil
)
controller.completionWithItemsHandler = { _, completed, _, _ in
guard completed else { return }
Task { @MainActor in
AnalyticsHostService.shared.client.recordReferralShared()
}
}
return controller
}
func updateUIViewController(
@@ -7,6 +7,7 @@
import Combine
import Foundation
import OSGKeyboardHostSupport
import OSGKeyboardShared
struct AccountCreditPurchaseOption: Identifiable, Equatable {
var id: String { productID }
@@ -43,6 +44,7 @@ final class AccountCreditPurchaseManager: ObservableObject {
private let service: any AccountCenterServicing
private let store: any AccountCreditStore
private let analyticsClient: any AnalyticsClient
private var updatesTask: Task<Void, Never>?
private var activeAccountID: UUID?
private var configuredProductIDs: Set<String> = []
@@ -50,10 +52,12 @@ final class AccountCreditPurchaseManager: ObservableObject {
init(
service: any AccountCenterServicing,
store: any AccountCreditStore = LiveAccountCreditStore()
store: any AccountCreditStore = LiveAccountCreditStore(),
analyticsClient: any AnalyticsClient = NoopAnalyticsClient()
) {
self.service = service
self.store = store
self.analyticsClient = analyticsClient
}
deinit {
@@ -100,6 +104,7 @@ final class AccountCreditPurchaseManager: ObservableObject {
return false
}
state = .purchasing(productID: productID)
analyticsClient.recordPurchaseStarted()
do {
switch try await store.purchase(productID: productID, accountID: accountID) {
case .success(let verification):
@@ -109,6 +114,7 @@ final class AccountCreditPurchaseManager: ObservableObject {
state = .pending
case .userCancelled:
state = .idle
analyticsClient.recordPurchaseCancelled()
@unknown default:
state = .failed(messageKey: "account.storekit.error.unknown")
}
@@ -118,6 +124,10 @@ final class AccountCreditPurchaseManager: ObservableObject {
return false
}
func recordPurchaseViewed() {
analyticsClient.recordPurchaseViewed()
}
func clearTransientState() {
switch state {
case .succeeded, .failed, .pending:
@@ -6,6 +6,7 @@
import Combine
import Foundation
import OSGKeyboardShared
@MainActor
final class AccountSessionCoordinator: ObservableObject {
@@ -46,6 +47,9 @@ final class AccountSessionCoordinator: ObservableObject {
private let sessionService: any AccountSessionServicing
private let centerService: any AccountCenterServicing
private let pendingReferralStore: any PendingReferralCodeStoring
private let analyticsClient: any AnalyticsClient
private let onAccountAuthenticated: (UUID) async -> Void
private let onAccountDeleted: () async -> Void
private let accountRefreshInterval: TimeInterval
private let now: () -> Date
private var didAttemptRestore = false
@@ -57,15 +61,22 @@ final class AccountSessionCoordinator: ObservableObject {
pendingReferralStore: any PendingReferralCodeStoring =
UserDefaultsPendingReferralCodeStore(),
accountRefreshInterval: TimeInterval = 10 * 60,
now: @escaping () -> Date = Date.init
now: @escaping () -> Date = Date.init,
analyticsClient: any AnalyticsClient = NoopAnalyticsClient(),
onAccountAuthenticated: @escaping (UUID) async -> Void = { _ in },
onAccountDeleted: @escaping () async -> Void = {}
) {
sessionService = dependencies.sessionService
centerService = dependencies.centerService
creditPurchases = AccountCreditPurchaseManager(
service: dependencies.centerService,
store: creditStore
store: creditStore,
analyticsClient: analyticsClient
)
self.pendingReferralStore = pendingReferralStore
self.analyticsClient = analyticsClient
self.onAccountAuthenticated = onAccountAuthenticated
self.onAccountDeleted = onAccountDeleted
self.accountRefreshInterval = accountRefreshInterval
self.now = now
pendingReferralCode = pendingReferralStore.code
@@ -94,6 +105,7 @@ final class AccountSessionCoordinator: ObservableObject {
sessionPhase = .signedOut
return
}
await onAccountAuthenticated(session.accountID)
sessionPhase = .signedIn(session)
await redeemPendingReferralIfNeeded()
await refreshAccountData(force: true)
@@ -114,6 +126,7 @@ final class AccountSessionCoordinator: ObservableObject {
pendingReferralCode = code
shouldPresentAccountCenter = true
operationErrorKey = nil
analyticsClient.recordInviteOpened()
if isSignedIn {
Task {
@@ -138,6 +151,7 @@ final class AccountSessionCoordinator: ObservableObject {
do {
let session = try await sessionService.signIn(with: payload)
await onAccountAuthenticated(session.accountID)
sessionPhase = .signedIn(session)
await redeemPendingReferralIfNeeded()
await refreshAccountData(force: true)
@@ -260,6 +274,7 @@ final class AccountSessionCoordinator: ObservableObject {
do {
try await sessionService.deleteAccount(with: payload)
await onAccountDeleted()
creditPurchases.reset()
clearAccountRefreshState()
sessionPhase = .signedOut
@@ -22,6 +22,9 @@ enum LiveAccountDependencyFactory {
let sessionVault = HostPrivateAccountKeychain(descriptor: descriptor)
let apiClient = AccountAPIClient(sessionVault: sessionVault)
HostAnalyticsBearerBridge.shared.install(
AccountAnalyticsBearerProvider(apiClient: apiClient)
)
let integrity = DeviceIntegrityCoordinator(
apiClient: apiClient,
keyStateStore: sessionVault
+130 -51
View File
@@ -14,6 +14,68 @@ private enum HomeRoute: Hashable {
case dictionary
}
enum FlowHomePiPStatusDescriptor: Equatable {
case text(localizationKey: String)
case progress(localizationKey: String, attempt: Int, total: Int)
}
enum FlowHomePiPStatusPolicy {
static func descriptor(
lifecycle: FlowPiPLifecycleState,
isStarting: Bool,
isRecording: Bool,
isProcessing: Bool,
isActive: Bool,
isHostReady: Bool
) -> FlowHomePiPStatusDescriptor {
switch lifecycle {
case .preparing(let attempt, let total):
return .progress(
localizationKey: "home.flow.preparingProgress",
attempt: attempt,
total: total
)
case .recovering(let attempt, let total):
return .progress(
localizationKey: "home.flow.recoveringProgress",
attempt: attempt,
total: total
)
case .waitingForForeground:
return .text(localizationKey: "home.flow.waitingForForeground")
case .failed:
return .text(localizationKey: "home.flow.recoveryFailed")
case .inactive, .active:
break
}
if isStarting {
return .text(localizationKey: "home.flow.starting")
}
if isRecording {
return .text(localizationKey: "home.flow.recording")
}
if isProcessing {
return .text(localizationKey: "home.flow.processing")
}
if isActive, isHostReady {
return .text(localizationKey: "home.flow.label")
}
if isActive {
// Session flag is up but the ready contract is not do not lie.
return .text(localizationKey: "home.flow.notReady")
}
return .text(localizationKey: "home.flow.inactive")
}
static func canRetry(
lifecycle: FlowPiPLifecycleState,
needsPermissionSetup: Bool
) -> Bool {
guard case .failed = lifecycle else { return false }
return !needsPermissionSetup
}
}
struct HomeView: View {
@Environment(\.themePalette) private var palette: ThemePalette
@Environment(\.scenePhase) private var scenePhase
@@ -55,6 +117,23 @@ struct HomeView: View {
!sessionIsLive && !needsPermissionSetup
}
private var canRetryPiP: Bool {
FlowHomePiPStatusPolicy.canRetry(
lifecycle: flowManager.pipLifecycleState,
needsPermissionSetup: needsPermissionSetup
)
}
private var canEndFlowSession: Bool {
if flowManager.isActive || flowManager.isStarting { return true }
switch flowManager.pipLifecycleState {
case .waitingForForeground, .failed:
return true
case .inactive, .preparing, .recovering, .active:
return false
}
}
private var shouldShowKeyboardHint: Bool {
!keyboardHintDismissed
&& !KeyboardSetupBridge.isReadyForOnboardingSkip
@@ -125,11 +204,11 @@ struct HomeView: View {
.padding(.top, logoTopPadding)
.padding(.bottom, logoBottomPadding)
if showsFlowSessionExtras {
flowSessionExtras
.padding(.horizontal, Spacing.lg)
.padding(.bottom, extrasBottomPadding)
}
// Engine + Flow status now stays directly below the
// logo so startup and recovery are always visible.
scrollStatusFooter
.padding(.horizontal, Spacing.lg)
.padding(.bottom, extrasBottomPadding)
HomeUsageStatsSection(layout: .stacked, compact: isCompact)
.padding(.horizontal, Spacing.lg)
@@ -138,11 +217,6 @@ struct HomeView: View {
homeLibrarySection
.padding(.horizontal, Spacing.lg)
.padding(.bottom, Spacing.xl)
// Scrolls with the page (not pinned); clearance comes from
// `tabBarScrollBottomPadding` so the dock never covers it.
scrollStatusFooter
.padding(.horizontal, Spacing.lg)
}
.frame(maxWidth: .infinity)
.tabBarScrollBottomPadding()
@@ -152,14 +226,14 @@ struct HomeView: View {
}
}
/// Engine + Flow status last content in the scroll stack.
/// Engine + Flow status visible near the logo instead of at scroll bottom.
private var scrollStatusFooter: some View {
VStack(spacing: Spacing.xs) {
setupGuidanceCard {
engineStatusLine
flowStatusFooter
Divider()
flowSessionExtras
}
.frame(maxWidth: .infinity, alignment: .center)
.multilineTextAlignment(.center)
}
// MARK: - Wide layout (iPad / regular width)
@@ -169,16 +243,11 @@ struct HomeView: View {
VStack(alignment: .leading, spacing: Spacing.lg) {
wideHeroHeader
if showsFlowSessionExtras {
flowSessionExtras
}
scrollStatusFooter
HomeUsageStatsSection(layout: .split)
homeLibrarySection
scrollStatusFooter
.frame(maxWidth: .infinity)
}
.padding(.horizontal, WideLayoutMetrics.pageHorizontalInset)
.padding(.top, Spacing.sm)
@@ -466,7 +535,26 @@ struct HomeView: View {
if needsAPIKeySetup {
// API Key
EmptyView()
} else if flowManager.isActive {
} else if canRetryPiP {
Button {
flowManager.retryPiPRecovery()
} label: {
Text("home.flow.retry")
.font(TypeStyle.caption2)
.foregroundStyle(palette.accent)
}
.buttonStyle(.plain)
.padding(.leading, Spacing.xs)
Button {
flowManager.endSession()
} label: {
Text("home.flow.endShort")
.font(TypeStyle.caption2)
.foregroundStyle(palette.textSecondary)
}
.buttonStyle(.plain)
.padding(.leading, Spacing.xs)
} else if canEndFlowSession {
Button {
flowManager.endSession()
} label: {
@@ -477,8 +565,8 @@ struct HomeView: View {
.buttonStyle(.plain)
.padding(.leading, Spacing.xs)
} else if canManuallyStartSession {
// Sessions no longer auto-start on foreground (keyboard survival).
// After a manual stop / expiry, Start is the explicit re-entry.
// Foreground activation is automatic. After an explicit stop
// while staying on this screen, Start is the explicit re-entry.
Button {
flowManager.activateOnForeground(
reason: "HomeView.startButton",
@@ -498,18 +586,10 @@ struct HomeView: View {
// MARK: - Flow extras (warnings / hints)
private var showsFlowSessionExtras: Bool {
needsPermissionSetup
|| flowManager.sessionWarning != nil
|| needsAPIKeySetup
|| shouldShowKeyboardHint
|| !flowManager.isActive
}
@ViewBuilder
private var flowSessionExtras: some View {
if needsPermissionSetup {
setupGuidanceCard {
VStack(alignment: .leading, spacing: Spacing.sm) {
Text(AppPermissions.homePermissionGuidanceMessage)
.font(TypeStyle.caption2)
.foregroundStyle(palette.warning)
@@ -526,14 +606,14 @@ struct HomeView: View {
.buttonStyle(.plain)
}
} else if let warning = flowManager.sessionWarning {
setupGuidanceCard {
VStack(alignment: .leading, spacing: Spacing.sm) {
Text(warning)
.font(TypeStyle.caption2)
.foregroundStyle(palette.warning)
.fixedSize(horizontal: false, vertical: true)
}
} else if needsAPIKeySetup {
setupGuidanceCard {
VStack(alignment: .leading, spacing: Spacing.sm) {
Text(config.isLocalEngine
? "home.setup.polishKeyMissing"
: "home.setup.cloudIncomplete")
@@ -542,7 +622,7 @@ struct HomeView: View {
.fixedSize(horizontal: false, vertical: true)
}
} else if shouldShowKeyboardHint {
setupGuidanceCard {
VStack(alignment: .leading, spacing: Spacing.sm) {
Text("home.setup.keyboardHint")
.font(TypeStyle.caption2)
.foregroundStyle(palette.textSecondary)
@@ -584,6 +664,8 @@ struct HomeView: View {
if needsAPIKeySetup { return palette.warning }
if flowManager.isUtteranceRecording { return palette.accent }
if flowManager.isUtteranceProcessing { return palette.accent }
if case .failed = flowManager.pipLifecycleState { return palette.warning }
if case .waitingForForeground = flowManager.pipLifecycleState { return palette.warning }
if flowManager.isActive, FlowSessionBridge.isHostReady() { return palette.accent }
if flowManager.isStarting { return palette.accent }
if needsPermissionSetup { return palette.warning }
@@ -598,23 +680,20 @@ struct HomeView: View {
/// with the OS), so the previous downloading / warming / failed
/// states collapse into the cloud-engine branch.
private var flowCapsuleStatusMessage: String {
if flowManager.isStarting {
return AppL10n.string("home.flow.starting")
let descriptor = FlowHomePiPStatusPolicy.descriptor(
lifecycle: flowManager.pipLifecycleState,
isStarting: flowManager.isStarting,
isRecording: flowManager.isUtteranceRecording,
isProcessing: flowManager.isUtteranceProcessing,
isActive: flowManager.isActive,
isHostReady: FlowSessionBridge.isHostReady()
)
switch descriptor {
case .text(let localizationKey):
return AppL10n.string(localizationKey)
case .progress(let localizationKey, let attempt, let total):
return AppL10n.format(localizationKey, attempt, total)
}
if flowManager.isUtteranceRecording {
return AppL10n.string("home.flow.recording")
}
if flowManager.isUtteranceProcessing {
return AppL10n.string("home.flow.processing")
}
if flowManager.isActive, FlowSessionBridge.isHostReady() {
return AppL10n.string("home.flow.label")
}
if flowManager.isActive {
// Session flag is up but the ready contract is not do not lie.
return AppL10n.string("home.flow.notReady")
}
return AppL10n.string("home.flow.inactive")
}
private var engineStatusLine: some View {
+28 -3
View File
@@ -16,15 +16,30 @@ struct MainAppRoot: View {
// `@ObservedObject` keeps subscriptions correct across Settings replay.
@ObservedObject private var config = ProviderConfig.shared
@ObservedObject private var releaseNotes = ReleaseNotesController.shared
@StateObject private var flowManager = FlowSessionManager()
@ObservedObject private var analytics = AnalyticsHostService.shared
@StateObject private var flowManager: FlowSessionManager
@StateObject private var accountSession: AccountSessionCoordinator
@State private var clmWarmupTask: Task<Void, Never>?
@State private var rimeStartupTask: Task<Void, Never>?
@State private var firstOpenAcquisitionChannel: AnalyticsAcquisitionChannel = .unknown
init(accountDependencies: AccountDependencies? = nil) {
let resolvedDependencies = accountDependencies ?? LiveAccountDependencyFactory.make()
let analytics = AnalyticsHostService.shared
_flowManager = StateObject(
wrappedValue: FlowSessionManager(analyticsClient: analytics.client)
)
_accountSession = StateObject(
wrappedValue: AccountSessionCoordinator(dependencies: resolvedDependencies)
wrappedValue: AccountSessionCoordinator(
dependencies: resolvedDependencies,
analyticsClient: analytics.client,
onAccountAuthenticated: { accountID in
await analytics.observeAuthenticatedAccount(accountID)
},
onAccountDeleted: {
await analytics.handleAccountDeletion()
}
)
)
}
@@ -35,6 +50,7 @@ struct MainAppRoot: View {
.environment(\.locale, config.uiLanguage.swiftUILocale)
.environmentObject(flowManager)
.environmentObject(accountSession)
.environmentObject(analytics)
.background {
FlowPiPHostView { view in
flowManager.attachPiPHostView(view)
@@ -65,6 +81,9 @@ struct MainAppRoot: View {
AppOpenURLRouter.shared.register { url in
handleIncomingURL(url)
}
analytics.prepare(
firstOpenAcquisitionChannel: firstOpenAcquisitionChannel
)
OSGDiag.log(
"MainAppRoot.onAppear scene=\(String(describing: scenePhase)) "
+ "onboarding=\(config.hasCompletedOnboarding) \(OSGDiag.memoryTag())",
@@ -131,6 +150,11 @@ struct MainAppRoot: View {
}
.onChange(of: scenePhase) { _, phase in
flowManager.handleScenePhase(phase)
if phase == .active {
analytics.appDidBecomeActive()
} else if phase == .background {
analytics.appDidEnterBackground()
}
guard phase == .active else {
clmWarmupTask?.cancel()
clmWarmupTask = nil
@@ -200,7 +224,7 @@ struct MainAppRoot: View {
MainTabView()
.id("main")
} else {
OnboardingView(config: config)
OnboardingExperienceView(config: config)
.id("onboarding")
}
}
@@ -246,6 +270,7 @@ struct MainAppRoot: View {
private func handleIncomingURL(_ url: URL) {
if accountSession.handleIncomingURL(url) {
firstOpenAcquisitionChannel = .referral
NotificationCenter.default.post(name: .osgOpenAccountDeepLink, object: nil)
return
}
File diff suppressed because it is too large Load Diff
@@ -565,11 +565,35 @@ struct ClipboardSettingsView: View {
struct AboutSettingsView: View {
@Environment(\.themePalette) private var palette: ThemePalette
@Environment(\.openURL) private var openURL
@EnvironmentObject private var analytics: AnalyticsHostService
@ObservedObject var config: ProviderConfig
var body: some View {
ScrollView {
CardPageContent {
CardSection("settings.analytics.section") {
VStack(alignment: .leading, spacing: 0) {
Toggle(
"settings.analytics.title",
isOn: Binding(
get: { analytics.isEnabled },
set: { analytics.setEnabled($0) }
)
)
.tint(palette.accent)
.settingsListRow()
Divider().background(palette.divider)
Text("settings.analytics.description")
.font(TypeStyle.caption2)
.foregroundStyle(palette.textSecondary)
.fixedSize(horizontal: false, vertical: true)
.settingsListRow()
}
.surfaceCard()
}
CardSection("settings.about.title") {
VStack(spacing: 0) {
Button {
@@ -637,5 +661,8 @@ struct AboutSettingsView: View {
.navigationTitle("settings.about.title")
.navigationBarTitleDisplayMode(.inline)
.hidesTabBarWhenPushed()
.onAppear {
analytics.refreshEnabledState()
}
}
}
+60
View File
@@ -36,6 +36,58 @@
"onboarding.polish.title" = "Text polish (LLM)";
"onboarding.polish.subtitle" = "Cleans up the transcript after recognition. Can differ from the ASR provider.";
"onboarding.polish.localHint" = "Fill in an API key to enable AI polish. Without a key, raw ASR text is inserted.";
"onboarding.experience.intro.eyebrow" = "AI AGENT VOICE KEYBOARD";
"onboarding.experience.intro.title" = "Speak. Its written.";
"onboarding.experience.intro.subtitle" = "More than dictation. OSGKeyboard understands, writes, translates, and edits—then its AI Agent helps with what comes next.";
"onboarding.experience.intro.action" = "Get Started";
"onboarding.experience.privacy.local.title" = "On-device transcription available";
"onboarding.experience.privacy.local.body" = "With the local engine, speech recognition stays on your device.";
"onboarding.experience.privacy.cloud.title" = "Clear cloud boundaries";
"onboarding.experience.privacy.cloud.body" = "Audio or text is sent only when you choose cloud recognition or an AI feature, and only to the configured service.";
"onboarding.experience.privacy.anywhere.title" = "Your AI Agent, in any app";
"onboarding.experience.privacy.anywhere.body" = "Dictate, translate, edit, and act from chat, email, notes, or code.";
"onboarding.experience.permissions.title" = "Let OSGKeyboard hear you";
"onboarding.experience.permissions.subtitle" = "Voice input needs microphone and speech recognition access. Two system prompts will appear next.";
"onboarding.experience.permissions.systemHint" = "These permissions are used only when you start voice input.";
"onboarding.experience.permissions.later" = "Set Up Later";
"onboarding.experience.keyboard.title" = "Add OSGKeyboard";
"onboarding.experience.keyboard.subtitle" = "Add the keyboard in iOS Settings and turn on Allow Full Access.";
"onboarding.experience.keyboard.ready" = "OSGKeyboard and Full Access are ready";
"onboarding.experience.keyboard.fullAccess" = "Full Access connects the keyboard to the host app microphone and optional AI services";
"onboarding.experience.keyboard.openAgain" = "Open Settings Again";
"onboarding.experience.keyboard.practiceAction" = "Try It Now";
"onboarding.experience.keyboard.previewTitle" = "Keyboards";
"onboarding.experience.practice.skip" = "Try Later";
"onboarding.experience.practice.sample" = "“Ill arrive tomorrow afternoon and message you when Im there.”";
"onboarding.experience.practice.editorTitle" = "New Message";
"onboarding.experience.practice.permissionsTitle" = "Voice access is still needed";
"onboarding.experience.practice.permissionsSubtitle" = "Go back to finish permission setup, then try your first voice input.";
"onboarding.experience.practice.switchTitle" = "Switch to OSGKeyboard";
"onboarding.experience.practice.switchSubtitle" = "Tap the field below, hold the globe key, then choose OSGKeyboard.";
"onboarding.experience.practice.fullAccessTitle" = "One more setting";
"onboarding.experience.practice.fullAccessSubtitle" = "OSGKeyboard is enabled. Turn on Allow Full Access to use voice input.";
"onboarding.experience.practice.fullAccessAction" = "Turn On Full Access";
"onboarding.experience.practice.signInTitle" = "Sign in for a free AI writing trial";
"onboarding.experience.practice.signInSubtitle" = "The lesson uses OSG Credits by default. This OOBE request will not deduct any credits.";
"onboarding.experience.practice.signInBody" = "Sign in with Apple to claim one free OOBE AI polish. Speech is still transcribed on your device.";
"onboarding.experience.practice.cloudTitle" = "Enable the free OOBE experience";
"onboarding.experience.practice.cloudSubtitle" = "Only this transcript is sent to OSG managed AI and explicitly tagged as OOBE.";
"onboarding.experience.practice.cloudBody" = "The server verifies this lesson as OOBE, with no credit reservation or charge. OSG Credits becomes your default AI service afterward.";
"onboarding.experience.practice.cloudAction" = "Agree & Try for Free";
"onboarding.experience.practice.creditsTitle" = "Preparing OSG Credits";
"onboarding.experience.practice.creditsSubtitle" = "This OOBE lesson is completely free.";
"onboarding.experience.practice.preparingCredits" = "Preparing your free OOBE request…";
"onboarding.experience.practice.creditsFailed" = "OSG Credits is temporarily unavailable. Try again—no credits were deducted.";
"onboarding.experience.practice.speakTitle" = "Now, say something";
"onboarding.experience.practice.speakSubtitle" = "Tap the microphone on the keyboard, speak, then tap it again. Your text will appear here.";
"onboarding.experience.complete.verifiedTitle" = "Thats it.";
"onboarding.experience.complete.title" = "Youre ready";
"onboarding.experience.complete.subtitle" = "Switch to OSGKeyboard in any text field to speak, write, or hand off the next step to your AI Agent.";
"onboarding.experience.complete.dictate" = "Voice to Text";
"onboarding.experience.complete.translate" = "Translate";
"onboarding.experience.complete.agent" = "AI Agent";
"onboarding.experience.complete.edit" = "Voice Edit";
"onboarding.experience.complete.action" = "Start Using";
"settings.onboarding.replay" = "Restart permission setup";
/* Common navigation */
@@ -419,6 +471,11 @@
"home.setup.keyboardHint" = "Don't see OSGKeyboard? Add it in iOS Settings.";
"home.setup.keyboardHint.dismiss" = "Got it";
"home.flow.starting" = "Starting voice session…";
"home.flow.preparingProgress" = "Preparing background session (%d/%d)…";
"home.flow.recoveringProgress" = "Restoring background session (%d/%d)…";
"home.flow.waitingForForeground" = "Background session paused — reopen this app to restore it";
"home.flow.recoveryFailed" = "Background session needs attention";
"home.flow.retry" = "Retry";
"home.flow.openSettings" = "Open Settings to grant permissions";
"home.flow.start" = "Start voice session";
"home.flow.end" = "End voice session";
@@ -805,3 +862,6 @@
"account.managed.accountCredits" = "Account credits";
"account.managed.explanation" = "This choice is independent from local or cloud speech recognition. Account sessions remain private to the host app; the keyboard receives only a short-lived, limited grant.";
"account.managed.preparing" = "Preparing managed access…";
"settings.analytics.section" = "Privacy";
"settings.analytics.title" = "Product Analytics";
"settings.analytics.description" = "Helps improve reliability with limited first-party events. Keyboard input, audio, transcripts, prompts, model output, credentials, and personal identifiers are never collected. Turning this off deletes queued events from this device.";
@@ -36,6 +36,58 @@
"onboarding.polish.title" = "文本润色(LLM";
"onboarding.polish.subtitle" = "识别完成后整理文字,可与 ASR 服务商不同。";
"onboarding.polish.localHint" = "填写 API Key 才能开启 AI 润色;未填写时将直接插入原始识别结果。";
"onboarding.experience.intro.eyebrow" = "AI AGENT 语音输入法";
"onboarding.experience.intro.title" = "开口,即成文";
"onboarding.experience.intro.subtitle" = "不只是语音输入。OSGKeyboard 能听懂、成文、翻译和编辑,也能调用 AI Agent 帮你处理下一步。";
"onboarding.experience.intro.action" = "开始设置";
"onboarding.experience.privacy.local.title" = "支持设备端转写";
"onboarding.experience.privacy.local.body" = "使用本地引擎时,语音识别在你的设备上完成。";
"onboarding.experience.privacy.cloud.title" = "明确的云端边界";
"onboarding.experience.privacy.cloud.body" = "只有选择云端识别或 AI 功能时,相应的语音或文字才会发送给已配置的服务。";
"onboarding.experience.privacy.anywhere.title" = "AI Agent,随处可用";
"onboarding.experience.privacy.anywhere.body" = "在聊天、邮件、笔记或代码编辑中,随时口述、翻译、编辑和处理任务。";
"onboarding.experience.permissions.title" = "让 OSGKeyboard 听见你";
"onboarding.experience.permissions.subtitle" = "语音输入需要麦克风和语音识别。接下来会依次出现两个系统提示。";
"onboarding.experience.permissions.systemHint" = "权限只在你主动使用语音输入时生效。";
"onboarding.experience.permissions.later" = "暂时跳过";
"onboarding.experience.keyboard.title" = "添加 OSGKeyboard";
"onboarding.experience.keyboard.subtitle" = "在 iOS 设置中添加键盘,并打开「允许完全访问」。";
"onboarding.experience.keyboard.ready" = "OSGKeyboard 与完全访问均已就绪";
"onboarding.experience.keyboard.fullAccess" = "完全访问用于连接主 App 的麦克风和可选 AI 服务";
"onboarding.experience.keyboard.openAgain" = "再次打开设置";
"onboarding.experience.keyboard.practiceAction" = "开始体验";
"onboarding.experience.keyboard.previewTitle" = "键盘";
"onboarding.experience.practice.skip" = "稍后体验";
"onboarding.experience.practice.sample" = "“我明天下午到,到了以后给你消息。”";
"onboarding.experience.practice.editorTitle" = "新消息";
"onboarding.experience.practice.permissionsTitle" = "还需要语音权限";
"onboarding.experience.practice.permissionsSubtitle" = "返回上一步完成权限设置,然后开始第一次语音输入。";
"onboarding.experience.practice.switchTitle" = "切换到 OSGKeyboard";
"onboarding.experience.practice.switchSubtitle" = "点一下下面的文本框,长按地球键,然后选择 OSGKeyboard。";
"onboarding.experience.practice.fullAccessTitle" = "再完成一个设置";
"onboarding.experience.practice.fullAccessSubtitle" = "OSGKeyboard 已启用。打开「允许完全访问」后就能使用语音。";
"onboarding.experience.practice.fullAccessAction" = "去设置打开完全访问";
"onboarding.experience.practice.signInTitle" = "登录,免费体验 AI 成文";
"onboarding.experience.practice.signInSubtitle" = "教学体验默认使用 OSG 积分服务,本次 OOBE 请求不会扣除任何积分。";
"onboarding.experience.practice.signInBody" = "使用 Apple 登录以领取一次免费的 OOBE AI 润色。语音仍在设备端转写。";
"onboarding.experience.practice.cloudTitle" = "启用免费 OOBE 体验";
"onboarding.experience.practice.cloudSubtitle" = "仅将本次转写文字发送到 OSG 托管 AI,并明确标记为 OOBE。";
"onboarding.experience.practice.cloudBody" = "本次教学请求由服务端验证为 OOBE,不预扣、不结算积分。完成后,OSG 积分将作为默认 AI 服务。";
"onboarding.experience.practice.cloudAction" = "同意并免费体验";
"onboarding.experience.practice.creditsTitle" = "正在准备 OSG 积分服务";
"onboarding.experience.practice.creditsSubtitle" = "本次 OOBE 教学完全免费。";
"onboarding.experience.practice.preparingCredits" = "正在准备免费 OOBE 请求…";
"onboarding.experience.practice.creditsFailed" = "暂时无法连接 OSG 积分服务,请重试。不会扣除积分。";
"onboarding.experience.practice.speakTitle" = "现在,说一句话";
"onboarding.experience.practice.speakSubtitle" = "点按键盘上的麦克风,说完后再点一次。文字会直接出现在这里。";
"onboarding.experience.complete.verifiedTitle" = "就是这样。";
"onboarding.experience.complete.title" = "已经准备好了";
"onboarding.experience.complete.subtitle" = "以后在任何输入框切换到 OSGKeyboard,开口成文,也可以继续交给 AI Agent。";
"onboarding.experience.complete.dictate" = "语音成文";
"onboarding.experience.complete.translate" = "翻译";
"onboarding.experience.complete.agent" = "AI Agent";
"onboarding.experience.complete.edit" = "语音编辑";
"onboarding.experience.complete.action" = "开始使用";
"settings.onboarding.replay" = "重新开始权限引导";
/* Common navigation */
@@ -418,6 +470,11 @@
"home.setup.keyboardHint" = "列表里没有?去系统设置里添加键盘。";
"home.setup.keyboardHint.dismiss" = "知道了";
"home.flow.starting" = "正在启动语音会话…";
"home.flow.preparingProgress" = "正在准备后台会话(%d/%d)…";
"home.flow.recoveringProgress" = "正在恢复后台会话(%d/%d)…";
"home.flow.waitingForForeground" = "后台会话已暂停,重新打开本 App 后自动恢复";
"home.flow.recoveryFailed" = "后台会话需要恢复";
"home.flow.retry" = "重试";
"home.flow.openSettings" = "去设置打开权限";
"home.flow.start" = "启动语音会话";
"home.flow.end" = "结束语音会话";
@@ -804,3 +861,6 @@
"account.managed.accountCredits" = "账号积分";
"account.managed.explanation" = "该选择与本地或云端语音识别相互独立。账号会话仅保存在主 App,键盘只能获得短时、受限的网关授权。";
"account.managed.preparing" = "正在准备托管服务…";
"settings.analytics.section" = "隐私";
"settings.analytics.title" = "产品分析";
"settings.analytics.description" = "通过有限的第一方事件帮助改进可靠性。绝不收集键盘输入、音频、转录、Prompt、模型输出、凭证或个人标识。关闭后会删除本机尚未上传的事件。";