merge: bring automatic low-profile PiP release into main
This commit is contained in:
@@ -7,10 +7,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||||||
|
|
||||||
## [Unreleased]
|
## [Unreleased]
|
||||||
|
|
||||||
|
## [1.6.5] - 2026-08-06
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
- **Volcengine ASR API Key auth**: settings can switch to the new-console single `X-Api-Key` mode while keeping legacy APP ID + Access Token as the default; SAUC resource is fixed to Doubao streaming 2.0 (`volc.seedasr.sauc.duration`). / **火山 ASR API Key 鉴权**:设置可切换到新控制台单字段 `X-Api-Key`,默认仍为旧版 APP ID + Access Token;SAUC 资源固定为豆包流式 2.0(`volc.seedasr.sauc.duration`)。
|
- **Volcengine ASR API Key auth**: settings can switch to the new-console single `X-Api-Key` mode while keeping legacy APP ID + Access Token as the default; SAUC resource is fixed to Doubao streaming 2.0 (`volc.seedasr.sauc.duration`). / **火山 ASR API Key 鉴权**:设置可切换到新控制台单字段 `X-Api-Key`,默认仍为旧版 APP ID + Access Token;SAUC 资源固定为豆包流式 2.0(`volc.seedasr.sauc.duration`)。
|
||||||
- **Custom style mood emoji**: custom polish styles can opt in (default off) to allow emotion-matched emoji; the prompt overrides R5 and post-processing keeps them on screen. Paste-only prompts that declare emoji opt-in are detected automatically. / **自定义风格情绪 emoji**:自定义润色风格可单独开启(默认关)按情绪点缀 emoji;提示词覆盖 R5,后处理保留上屏。仅粘贴声明允许 emoji 的 prompt 也会自动识别。
|
- **Custom style mood emoji**: custom polish styles can opt in (default off) to allow emotion-matched emoji; the prompt overrides R5 and post-processing keeps them on screen. Paste-only prompts that declare emoji opt-in are detected automatically. / **自定义风格情绪 emoji**:自定义润色风格可单独开启(默认关)按情绪点缀 emoji;提示词覆盖 R5,后处理保留上屏。仅粘贴声明允许 emoji 的 prompt 也会自动识别。
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
- **Automatic low-profile PiP**: replace the 18 FPS sample-buffer teaching video with a transparent 0.1 pt VideoCall PiP; every host-app open now arms it automatically, keyboard appearance performs one silent handoff when it is absent (including Pinyin/English typing mode), and idle mic UI stays green without PiP startup copy or host overlays. Once active, PiP stops frame rendering, releases the idle audio session, allows screen sleep, and refreshes audio levels only while recording. / **自动低感知 PiP**:用透明且高度仅 0.1pt 的 VideoCall PiP 替换 18 FPS SampleBuffer 教学视频;每次打开主 App 都自动武装,键盘出现且 PiP 缺失时静默跳转一次(包括默认拼音/英文输入模式),空闲麦克风始终保持绿色,不再显示 PiP 启动文案或主 App 浮层。PiP 激活后停止帧渲染、释放空闲音频会话、允许屏幕休眠,并仅在录音时刷新音量。
|
||||||
|
|
||||||
### Fixed
|
### Fixed
|
||||||
- **Shared LevelDB privacy manifest**: ship `PrivacyInfo.xcprivacy` inside `OSGKeyboardShared.framework` so App Store Connect no longer rejects uploads for ITMS-91061 (leveldb via librime). / **Shared LevelDB 隐私清单**:在 `OSGKeyboardShared.framework` 内打包 `PrivacyInfo.xcprivacy`,避免 App Store Connect 因 ITMS-91061(librime 内嵌 leveldb)拒收。
|
- **Shared LevelDB privacy manifest**: ship `PrivacyInfo.xcprivacy` inside `OSGKeyboardShared.framework` so App Store Connect no longer rejects uploads for ITMS-91061 (leveldb via librime). / **Shared LevelDB 隐私清单**:在 `OSGKeyboardShared.framework` 内打包 `PrivacyInfo.xcprivacy`,避免 App Store Connect 因 ITMS-91061(librime 内嵌 leveldb)拒收。
|
||||||
- **PiP post-utterance yellow flash**: after a voice turn, the mic no longer briefly shows yellow「正在启动画中画」— hold ready once the session proved live, refresh host ready on ack, and avoid labeling an already-active PiP as `.starting`. / **PiP 句末黄色闪烁**:语音说完后麦克风不再短暂变黄并提示「正在启动画中画」——会话曾就绪后保持绿灯、ack 后立即刷新 host ready,且已激活的 PiP 不再标成 `.starting`。
|
- **PiP post-utterance yellow flash**: after a voice turn, the mic no longer briefly shows yellow「正在启动画中画」— hold ready once the session proved live, refresh host ready on ack, and avoid labeling an already-active PiP as `.starting`. / **PiP 句末黄色闪烁**:语音说完后麦克风不再短暂变黄并提示「正在启动画中画」——会话曾就绪后保持绿灯、ack 后立即刷新 host ready,且已激活的 PiP 不再标成 `.starting`。
|
||||||
|
|||||||
@@ -1,9 +1,10 @@
|
|||||||
// FlowPictureInPictureController.swift
|
// FlowPictureInPictureController.swift
|
||||||
// OSGKeyboard · Main App
|
// OSGKeyboard · Main App
|
||||||
//
|
//
|
||||||
// PiP keep-alive for Flow sessions: enqueues a looping “tuck to edge”
|
// Low-profile PiP keep-alive for Flow sessions. Production uses a transparent,
|
||||||
// teaching animation (OSG logo card) so the host stays eligible for
|
// static AVPictureInPictureVideoCallViewController with a 0.1 pt content
|
||||||
// multitasking while the mic is off between utterances.
|
// height, no frame pump, and no idle audio session. The legacy sample-buffer
|
||||||
|
// teaching animation remains as a code-level fallback.
|
||||||
|
|
||||||
import AVFoundation
|
import AVFoundation
|
||||||
import AVKit
|
import AVKit
|
||||||
@@ -47,6 +48,8 @@ final class FlowPictureInPictureController: NSObject {
|
|||||||
let displayLayer = AVSampleBufferDisplayLayer()
|
let displayLayer = AVSampleBufferDisplayLayer()
|
||||||
|
|
||||||
private var pipController: AVPictureInPictureController?
|
private var pipController: AVPictureInPictureController?
|
||||||
|
private var videoCallContentController: AVPictureInPictureVideoCallViewController?
|
||||||
|
private var transparentContentView: UIView?
|
||||||
private var displayLink: CADisplayLink?
|
private var displayLink: CADisplayLink?
|
||||||
private weak var hostView: UIView?
|
private weak var hostView: UIView?
|
||||||
private var isStoppingProgrammatically = false
|
private var isStoppingProgrammatically = false
|
||||||
@@ -56,6 +59,17 @@ final class FlowPictureInPictureController: NSObject {
|
|||||||
/// Last system failure reported by the PiP delegate (cleared on each start).
|
/// Last system failure reported by the PiP delegate (cleared on each start).
|
||||||
private var lastSystemStartFailure: Error?
|
private var lastSystemStartFailure: Error?
|
||||||
|
|
||||||
|
/// Production route: a static, transparent VideoCall PiP. Unlike the
|
||||||
|
/// sample-buffer teaching animation, this needs no video frame pump and can
|
||||||
|
/// release AVAudioSession after PiP becomes active.
|
||||||
|
private let usesLowPowerVideoCallPiP = true
|
||||||
|
|
||||||
|
/// Community implementations confirm that VideoCall PiP accepts an extreme
|
||||||
|
/// aspect ratio. The system may clamp its final width, but a 0.1 pt content
|
||||||
|
/// height makes the surface visually negligible without private positioning
|
||||||
|
/// APIs.
|
||||||
|
private static let lowProfileContentSize = CGSize(width: 300, height: 0.1)
|
||||||
|
|
||||||
private enum Canvas {
|
private enum Canvas {
|
||||||
static let width = 480
|
static let width = 480
|
||||||
static let height = 270
|
static let height = 270
|
||||||
@@ -69,6 +83,10 @@ final class FlowPictureInPictureController: NSObject {
|
|||||||
func attachHostView(_ view: UIView) {
|
func attachHostView(_ view: UIView) {
|
||||||
hostView = view
|
hostView = view
|
||||||
hasHostView = true
|
hasHostView = true
|
||||||
|
if usesLowPowerVideoCallPiP {
|
||||||
|
displayLayer.removeFromSuperlayer()
|
||||||
|
return
|
||||||
|
}
|
||||||
let bounds = view.bounds
|
let bounds = view.bounds
|
||||||
displayLayer.frame = (bounds.width >= 1 && bounds.height >= 1)
|
displayLayer.frame = (bounds.width >= 1 && bounds.height >= 1)
|
||||||
? bounds
|
? bounds
|
||||||
@@ -114,20 +132,25 @@ final class FlowPictureInPictureController: NSObject {
|
|||||||
pipController = nil
|
pipController = nil
|
||||||
}
|
}
|
||||||
configureControllerIfNeeded()
|
configureControllerIfNeeded()
|
||||||
|
if !usesLowPowerVideoCallPiP {
|
||||||
warmLogoCacheIfNeeded()
|
warmLogoCacheIfNeeded()
|
||||||
animationStartedAt = CACurrentMediaTime()
|
animationStartedAt = CACurrentMediaTime()
|
||||||
startFramePump()
|
startFramePump()
|
||||||
|
}
|
||||||
guard pipController != nil else { return false }
|
guard pipController != nil else { return false }
|
||||||
|
|
||||||
if pipController?.isPictureInPictureActive == true {
|
if pipController?.isPictureInPictureActive == true {
|
||||||
isPictureInPictureActive = true
|
isPictureInPictureActive = true
|
||||||
|
releaseAudioSessionForLowPowerPiP()
|
||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if !usesLowPowerVideoCallPiP {
|
||||||
// Prime a few frames before asking the system to start PiP.
|
// Prime a few frames before asking the system to start PiP.
|
||||||
enqueueGuideFrame()
|
enqueueGuideFrame()
|
||||||
enqueueGuideFrame()
|
enqueueGuideFrame()
|
||||||
pipController?.invalidatePlaybackState()
|
pipController?.invalidatePlaybackState()
|
||||||
|
}
|
||||||
pipController?.startPictureInPicture()
|
pipController?.startPictureInPicture()
|
||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
@@ -200,10 +223,12 @@ final class FlowPictureInPictureController: NSObject {
|
|||||||
isStoppingProgrammatically = true
|
isStoppingProgrammatically = true
|
||||||
stopFramePump()
|
stopFramePump()
|
||||||
pipController?.stopPictureInPicture()
|
pipController?.stopPictureInPicture()
|
||||||
|
if !usesLowPowerVideoCallPiP {
|
||||||
displayLayer.sampleBufferRenderer.flush(
|
displayLayer.sampleBufferRenderer.flush(
|
||||||
removingDisplayedImage: true,
|
removingDisplayedImage: true,
|
||||||
completionHandler: nil
|
completionHandler: nil
|
||||||
)
|
)
|
||||||
|
}
|
||||||
isPictureInPictureActive = false
|
isPictureInPictureActive = false
|
||||||
animationStartedAt = nil
|
animationStartedAt = nil
|
||||||
lastSystemStartFailure = nil
|
lastSystemStartFailure = nil
|
||||||
@@ -214,13 +239,20 @@ final class FlowPictureInPictureController: NSObject {
|
|||||||
/// `canStartPictureInPictureAutomaticallyFromInline` can take over.
|
/// `canStartPictureInPictureAutomaticallyFromInline` can take over.
|
||||||
func prepareForBackgroundAutoStart() async {
|
func prepareForBackgroundAutoStart() async {
|
||||||
guard isPictureInPictureActive || pipController != nil else { return }
|
guard isPictureInPictureActive || pipController != nil else { return }
|
||||||
|
if isPictureInPictureActive {
|
||||||
|
releaseAudioSessionForLowPowerPiP()
|
||||||
|
return
|
||||||
|
}
|
||||||
_ = await activateAudioSessionForPiP()
|
_ = await activateAudioSessionForPiP()
|
||||||
|
if !usesLowPowerVideoCallPiP {
|
||||||
startFramePump()
|
startFramePump()
|
||||||
enqueueGuideFrame()
|
enqueueGuideFrame()
|
||||||
pipController?.invalidatePlaybackState()
|
pipController?.invalidatePlaybackState()
|
||||||
if !isPictureInPictureActive {
|
|
||||||
pipController?.startPictureInPicture()
|
|
||||||
}
|
}
|
||||||
|
// `canStartPictureInPictureAutomaticallyFromInline` is the primary
|
||||||
|
// transition when the app backgrounds. This explicit request is a
|
||||||
|
// foreground/inactive fallback and remains idempotent.
|
||||||
|
pipController?.startPictureInPicture()
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Keep the shared audio session eligible for PiP after capture stops its
|
/// Keep the shared audio session eligible for PiP after capture stops its
|
||||||
@@ -228,7 +260,11 @@ final class FlowPictureInPictureController: NSObject {
|
|||||||
/// the stable playAndRecord category instead of flipping back to playback.
|
/// the stable playAndRecord category instead of flipping back to playback.
|
||||||
@discardableResult
|
@discardableResult
|
||||||
func reassertKeepAliveAudioSession() async -> Bool {
|
func reassertKeepAliveAudioSession() async -> Bool {
|
||||||
await activateAudioSessionForPiP()
|
if usesLowPowerVideoCallPiP, isPictureInPictureActive {
|
||||||
|
releaseAudioSessionForLowPowerPiP()
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
return await activateAudioSessionForPiP()
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Kept for FlowSessionManager call sites; guide animation ignores live levels.
|
/// Kept for FlowSessionManager call sites; guide animation ignores live levels.
|
||||||
@@ -267,10 +303,39 @@ final class FlowPictureInPictureController: NSObject {
|
|||||||
guard pipController == nil else { return }
|
guard pipController == nil else { return }
|
||||||
guard AVPictureInPictureController.isPictureInPictureSupported() else { return }
|
guard AVPictureInPictureController.isPictureInPictureSupported() else { return }
|
||||||
|
|
||||||
let contentSource = AVPictureInPictureController.ContentSource(
|
let contentSource: AVPictureInPictureController.ContentSource
|
||||||
|
if usesLowPowerVideoCallPiP, #available(iOS 15.0, *),
|
||||||
|
let hostView {
|
||||||
|
let contentController = AVPictureInPictureVideoCallViewController()
|
||||||
|
contentController.preferredContentSize = Self.lowProfileContentSize
|
||||||
|
contentController.view.backgroundColor = .clear
|
||||||
|
contentController.view.isOpaque = false
|
||||||
|
contentController.view.layer.backgroundColor = UIColor.clear.cgColor
|
||||||
|
contentController.view.layer.isOpaque = false
|
||||||
|
contentController.view.clipsToBounds = true
|
||||||
|
|
||||||
|
let transparentView = UIView(frame: contentController.view.bounds)
|
||||||
|
transparentView.autoresizingMask = [.flexibleWidth, .flexibleHeight]
|
||||||
|
transparentView.backgroundColor = .clear
|
||||||
|
transparentView.isOpaque = false
|
||||||
|
transparentView.isUserInteractionEnabled = false
|
||||||
|
transparentView.layer.backgroundColor = UIColor.clear.cgColor
|
||||||
|
transparentView.layer.isOpaque = false
|
||||||
|
transparentView.layer.opacity = 0
|
||||||
|
contentController.view.addSubview(transparentView)
|
||||||
|
|
||||||
|
videoCallContentController = contentController
|
||||||
|
transparentContentView = transparentView
|
||||||
|
contentSource = AVPictureInPictureController.ContentSource(
|
||||||
|
activeVideoCallSourceView: hostView,
|
||||||
|
contentViewController: contentController
|
||||||
|
)
|
||||||
|
} else {
|
||||||
|
contentSource = AVPictureInPictureController.ContentSource(
|
||||||
sampleBufferDisplayLayer: displayLayer,
|
sampleBufferDisplayLayer: displayLayer,
|
||||||
playbackDelegate: self
|
playbackDelegate: self
|
||||||
)
|
)
|
||||||
|
}
|
||||||
let controller = AVPictureInPictureController(contentSource: contentSource)
|
let controller = AVPictureInPictureController(contentSource: contentSource)
|
||||||
controller.delegate = self
|
controller.delegate = self
|
||||||
controller.canStartPictureInPictureAutomaticallyFromInline = true
|
controller.canStartPictureInPictureAutomaticallyFromInline = true
|
||||||
@@ -279,6 +344,13 @@ final class FlowPictureInPictureController: NSObject {
|
|||||||
didActivateAudioSessionBeforeController = true
|
didActivateAudioSessionBeforeController = true
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private func releaseAudioSessionForLowPowerPiP() {
|
||||||
|
guard usesLowPowerVideoCallPiP, isPictureInPictureActive else { return }
|
||||||
|
stopFramePump()
|
||||||
|
FlowAudioSessionCoordinator.shared.deactivate()
|
||||||
|
FlowDiagnostics.log("low-profile PiP active — released audio session and frame pump")
|
||||||
|
}
|
||||||
|
|
||||||
private func startFramePump() {
|
private func startFramePump() {
|
||||||
guard displayLink == nil else { return }
|
guard displayLink == nil else { return }
|
||||||
let link = CADisplayLink(target: self, selector: #selector(handleDisplayLink(_:)))
|
let link = CADisplayLink(target: self, selector: #selector(handleDisplayLink(_:)))
|
||||||
@@ -572,6 +644,7 @@ extension FlowPictureInPictureController: @preconcurrency AVPictureInPictureCont
|
|||||||
) {
|
) {
|
||||||
isPictureInPictureActive = true
|
isPictureInPictureActive = true
|
||||||
lastSystemStartFailure = nil
|
lastSystemStartFailure = nil
|
||||||
|
releaseAudioSessionForLowPowerPiP()
|
||||||
}
|
}
|
||||||
|
|
||||||
func pictureInPictureControllerDidStopPictureInPicture(
|
func pictureInPictureControllerDidStopPictureInPicture(
|
||||||
@@ -587,8 +660,8 @@ extension FlowPictureInPictureController: @preconcurrency AVPictureInPictureCont
|
|||||||
_ pictureInPictureController: AVPictureInPictureController,
|
_ pictureInPictureController: AVPictureInPictureController,
|
||||||
failedToStartPictureInPictureWithError error: Error
|
failedToStartPictureInPictureWithError error: Error
|
||||||
) {
|
) {
|
||||||
// First attempts often fail while the sample-buffer source is still
|
// Sample-buffer fallback may need warm-up retries. VideoCall PiP uses
|
||||||
// warming; keep retrying via the display link / auto-inline path.
|
// the automatic-inline path plus the bounded startAndWait fallback.
|
||||||
lastSystemStartFailure = error
|
lastSystemStartFailure = error
|
||||||
FlowDiagnostics.log("PiP start attempt failed (will retry): \(error.localizedDescription)")
|
FlowDiagnostics.log("PiP start attempt failed (will retry): \(error.localizedDescription)")
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -309,10 +309,10 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
|
|
||||||
/// Foreground entry for Flow.
|
/// Foreground entry for Flow.
|
||||||
///
|
///
|
||||||
/// Default is **light**: clear orphaned Live Activities / permission UI, but
|
/// Default is **light** for audio work: clear orphaned Live Activities /
|
||||||
/// do **not** start continuous capture. Auto-capture on every foreground was
|
/// permission UI and automatically arm the low-profile PiP, but do not
|
||||||
/// holding the host at ~170–220 MB RSS and jetsamming the keyboard extension
|
/// start capture or ASR. The transparent PiP is intentionally cheap; the
|
||||||
/// before `KVC.init` could run.
|
/// 170–220 MB capture/model path still starts only on explicit speech.
|
||||||
///
|
///
|
||||||
/// Pass `startCapture: true` for explicit user intent (Home “Start”, etc.).
|
/// Pass `startCapture: true` for explicit user intent (Home “Start”, etc.).
|
||||||
/// Keyboard mic cold-start uses `osgkeyboard://startflow` → `startSession`.
|
/// Keyboard mic cold-start uses `osgkeyboard://startflow` → `startSession`.
|
||||||
@@ -349,14 +349,19 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
guard startCapture else {
|
let shouldAutoArmPiP = keepAliveMode == .pictureInPicture
|
||||||
|
guard startCapture || shouldAutoArmPiP else {
|
||||||
OSGDiag.log(
|
OSGDiag.log(
|
||||||
"activateOnForeground light — skip capture (keyboard survival) \(OSGDiag.memoryTag())",
|
"activateOnForeground light — skip capture (keyboard survival) \(OSGDiag.memoryTag())",
|
||||||
category: "flow"
|
category: "flow"
|
||||||
)
|
)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
startSession(reason: "activateOnForeground:\(reason)")
|
startSession(
|
||||||
|
reason: shouldAutoArmPiP && !startCapture
|
||||||
|
? "activateOnForeground.autoPiP:\(reason)"
|
||||||
|
: "activateOnForeground:\(reason)"
|
||||||
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
func dismissColdStartOverlay() {
|
func dismissColdStartOverlay() {
|
||||||
@@ -976,7 +981,9 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
FlowSessionBridge.markSessionActive(duration: duration, sessionId: sessionId)
|
FlowSessionBridge.markSessionActive(duration: duration, sessionId: sessionId)
|
||||||
FlowSessionDarwin.postSessionChanged()
|
FlowSessionDarwin.postSessionChanged()
|
||||||
isActive = true
|
isActive = true
|
||||||
ScreenWakeLock.acquire()
|
// Low-profile PiP is a system-owned keep-alive surface. Keeping the
|
||||||
|
// display awake wastes far more power than the static PiP itself.
|
||||||
|
ScreenWakeLock.release()
|
||||||
sessionExpiresAt = nil
|
sessionExpiresAt = nil
|
||||||
|
|
||||||
startHeartbeat()
|
startHeartbeat()
|
||||||
@@ -1097,11 +1104,9 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
|
|
||||||
private func presentColdStartReadyOverlay() {
|
private func presentColdStartReadyOverlay() {
|
||||||
let hostEntry = HostReturnService.pendingHostEntry()
|
let hostEntry = HostReturnService.pendingHostEntry()
|
||||||
coldStartContext = FlowColdStartContext(
|
// Handoff remains fully automatic; no preparing/ready overlay is
|
||||||
hostEntry: hostEntry,
|
// mounted in the host UI.
|
||||||
state: .ready,
|
coldStartContext = nil
|
||||||
keepAliveMode: keepAliveMode
|
|
||||||
)
|
|
||||||
scheduleAutoReturnToHostIfNeeded(hostEntry: hostEntry)
|
scheduleAutoReturnToHostIfNeeded(hostEntry: hostEntry)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1110,7 +1115,8 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
guard skipSwitch, hostEntry != nil else { return }
|
guard skipSwitch, hostEntry != nil else { return }
|
||||||
Task { @MainActor [weak self] in
|
Task { @MainActor [weak self] in
|
||||||
try? await Task.sleep(nanoseconds: 450_000_000)
|
try? await Task.sleep(nanoseconds: 450_000_000)
|
||||||
guard let self, self.coldStartContext?.state == .ready else { return }
|
guard let self, self.isColdStartHandoff, self.isActive,
|
||||||
|
FlowSessionBridge.isHostReady() else { return }
|
||||||
if HostReturnService.openPendingHostIfPossible() {
|
if HostReturnService.openPendingHostIfPossible() {
|
||||||
self.dismissColdStartOverlay()
|
self.dismissColdStartOverlay()
|
||||||
}
|
}
|
||||||
@@ -1197,38 +1203,24 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
}
|
}
|
||||||
|
|
||||||
private func showColdStartPreparing() {
|
private func showColdStartPreparing() {
|
||||||
coldStartContext = FlowColdStartContext(
|
coldStartContext = nil
|
||||||
hostEntry: HostReturnService.pendingHostEntry(),
|
|
||||||
state: .preparing,
|
|
||||||
keepAliveMode: keepAliveMode
|
|
||||||
)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
private func showColdStartPermissionFailure() {
|
private func showColdStartPermissionFailure() {
|
||||||
FlowSessionBridge.setHostReady(false)
|
FlowSessionBridge.setHostReady(false)
|
||||||
coldStartContext = FlowColdStartContext(
|
coldStartContext = nil
|
||||||
hostEntry: HostReturnService.pendingHostEntry(),
|
|
||||||
state: .failed(.permission(message: permissionWarningMessage())),
|
|
||||||
keepAliveMode: keepAliveMode
|
|
||||||
)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
private func showColdStartAudioFailure(message: String) {
|
private func showColdStartAudioFailure(message: String) {
|
||||||
FlowSessionBridge.setHostReady(false)
|
FlowSessionBridge.setHostReady(false)
|
||||||
coldStartContext = FlowColdStartContext(
|
_ = message
|
||||||
hostEntry: HostReturnService.pendingHostEntry(),
|
coldStartContext = nil
|
||||||
state: .failed(.audio(message: message)),
|
|
||||||
keepAliveMode: keepAliveMode
|
|
||||||
)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
private func showColdStartPipFailure(message: String) {
|
private func showColdStartPipFailure(message: String) {
|
||||||
FlowSessionBridge.setHostReady(false)
|
FlowSessionBridge.setHostReady(false)
|
||||||
coldStartContext = FlowColdStartContext(
|
_ = message
|
||||||
hostEntry: HostReturnService.pendingHostEntry(),
|
coldStartContext = nil
|
||||||
state: .failed(.pip(message: message)),
|
|
||||||
keepAliveMode: .pictureInPicture
|
|
||||||
)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
private func bindSessionASRIfNeeded(force: Bool = false) {
|
private func bindSessionASRIfNeeded(force: Bool = false) {
|
||||||
@@ -2515,6 +2507,10 @@ final class FlowSessionManager: ObservableObject {
|
|||||||
levelTask = Task { @MainActor [weak self] in
|
levelTask = Task { @MainActor [weak self] in
|
||||||
while !Task.isCancelled {
|
while !Task.isCancelled {
|
||||||
guard let self, self.isActive else { break }
|
guard let self, self.isActive else { break }
|
||||||
|
guard self.isUtteranceRecording || self.capture.engineIsLive else {
|
||||||
|
try? await Task.sleep(nanoseconds: 1_000_000_000)
|
||||||
|
continue
|
||||||
|
}
|
||||||
let levels = self.capture.currentAudioLevels()
|
let levels = self.capture.currentAudioLevels()
|
||||||
if self.usesPiPKeepAlive {
|
if self.usesPiPKeepAlive {
|
||||||
self.pipController.updateWaveformLevels(levels)
|
self.pipController.updateWaveformLevels(levels)
|
||||||
|
|||||||
@@ -30,19 +30,6 @@ struct MainAppRoot: View {
|
|||||||
}
|
}
|
||||||
.environment(\.locale, config.uiLanguage.swiftUILocale)
|
.environment(\.locale, config.uiLanguage.swiftUILocale)
|
||||||
.environmentObject(flowManager)
|
.environmentObject(flowManager)
|
||||||
.overlay {
|
|
||||||
if let context = flowManager.coldStartContext {
|
|
||||||
FlowColdStartOverlay(
|
|
||||||
context: context,
|
|
||||||
onReturnToHost: { flowManager.returnToPendingHostFromColdStart() },
|
|
||||||
onDismiss: { flowManager.dismissColdStartOverlay() },
|
|
||||||
onRetry: { flowManager.retryColdStartReadiness() },
|
|
||||||
onOpenSettings: { flowManager.openColdStartPermissionSettings() }
|
|
||||||
)
|
|
||||||
.transition(.opacity)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
.animation(.easeInOut(duration: 0.2), value: flowManager.coldStartContext != nil)
|
|
||||||
.background {
|
.background {
|
||||||
FlowPiPHostView { view in
|
FlowPiPHostView { view in
|
||||||
flowManager.attachPiPHostView(view)
|
flowManager.attachPiPHostView(view)
|
||||||
@@ -75,8 +62,8 @@ struct MainAppRoot: View {
|
|||||||
// Heavy work (Flow / CLM / Rime) only after onboarding. Doing it
|
// Heavy work (Flow / CLM / Rime) only after onboarding. Doing it
|
||||||
// earlier jetsams the host (~150 MB+) and the keyboard dies with it.
|
// earlier jetsams the host (~150 MB+) and the keyboard dies with it.
|
||||||
if config.hasCompletedOnboarding {
|
if config.hasCompletedOnboarding {
|
||||||
// Light path only: never auto-start continuous capture here.
|
// Automatically arm the low-profile PiP on every host open.
|
||||||
// Capture starts on Home Start / keyboard startflow / mic press.
|
// Capture/ASR remain lazy and start only on an actual mic press.
|
||||||
flowManager.activateOnForeground(reason: "MainAppRoot.onAppear")
|
flowManager.activateOnForeground(reason: "MainAppRoot.onAppear")
|
||||||
schedulePostOnboardingWarmup(reason: "MainAppRoot.onAppear")
|
schedulePostOnboardingWarmup(reason: "MainAppRoot.onAppear")
|
||||||
} else {
|
} else {
|
||||||
|
|||||||
@@ -197,6 +197,11 @@ public final class KeyboardViewController: UIInputViewController {
|
|||||||
disableSystemGestureDelays()
|
disableSystemGestureDelays()
|
||||||
keyboardHeightConstraint?.constant = targetKeyboardHeight
|
keyboardHeightConstraint?.constant = targetKeyboardHeight
|
||||||
refreshReturnKeyRole()
|
refreshReturnKeyRole()
|
||||||
|
// Run after the extension is fully presented so UIKit accepts the
|
||||||
|
// containing-app handoff even when typing mode is the default surface.
|
||||||
|
DispatchQueue.main.async { [weak self] in
|
||||||
|
self?.flowCoordinator.ensurePiPReadyOnKeyboardOpen()
|
||||||
|
}
|
||||||
OSGDiag.log(
|
OSGDiag.log(
|
||||||
"KVC.viewDidAppear done height=\(targetKeyboardHeight) \(OSGDiag.memoryTag())",
|
"KVC.viewDidAppear done height=\(targetKeyboardHeight) \(OSGDiag.memoryTag())",
|
||||||
category: "boot"
|
category: "boot"
|
||||||
|
|||||||
@@ -116,6 +116,33 @@ final class KeyboardFlowCoordinator {
|
|||||||
stopHostReadyWait()
|
stopHostReadyWait()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Ensure the containing app has armed its low-profile PiP even when the
|
||||||
|
/// keyboard opens directly into Pinyin/English typing mode. The mic stays
|
||||||
|
/// visually ready; if the host contract is missing, one automatic handoff
|
||||||
|
/// prepares PiP so the next press does not need another app switch.
|
||||||
|
func ensurePiPReadyOnKeyboardOpen() {
|
||||||
|
guard FlowHandoffPolicy.allowsProactiveHostAutoLaunch,
|
||||||
|
FlowSessionPolicy.keepAliveMode() == .pictureInPicture,
|
||||||
|
state.hasCompletedOnboarding,
|
||||||
|
hasFullAccess(),
|
||||||
|
AppGroup.isAvailable,
|
||||||
|
!isPendingFlowStart,
|
||||||
|
!isFlowRecording,
|
||||||
|
!isAwaitingFlowResult else { return }
|
||||||
|
|
||||||
|
FlowSessionBridge.reloadFromDisk()
|
||||||
|
if FlowSessionBridge.isHostReady() { return }
|
||||||
|
|
||||||
|
if let reason = FlowSessionBridge.readySnapshot()?.reason,
|
||||||
|
reason == .recording || reason == .processing || reason == .awaitingDelivery {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
detectAndStoreAppContext()
|
||||||
|
beginFlowStart(recordAfterHandoff: false)
|
||||||
|
traceState("keyboardOpen.autoArmPiP")
|
||||||
|
}
|
||||||
|
|
||||||
func refreshSessionState() {
|
func refreshSessionState() {
|
||||||
FlowSessionBridge.reloadFromDisk()
|
FlowSessionBridge.reloadFromDisk()
|
||||||
refreshConfigFromAppGroup()
|
refreshConfigFromAppGroup()
|
||||||
@@ -527,11 +554,7 @@ final class KeyboardFlowCoordinator {
|
|||||||
isPendingFlowStart = true
|
isPendingFlowStart = true
|
||||||
isFlowRecording = false
|
isFlowRecording = false
|
||||||
flowStartDeadline = Date().timeIntervalSince1970 + FlowWatchdog.startTimeout
|
flowStartDeadline = Date().timeIntervalSince1970 + FlowWatchdog.startTimeout
|
||||||
state.lastTranscript = ExtL10n.string(
|
state.lastTranscript = ""
|
||||||
FlowSessionPolicy.keepAliveMode() == .pictureInPicture
|
|
||||||
? "keyboard.flow.startingSession.pip"
|
|
||||||
: "keyboard.flow.startingSession"
|
|
||||||
)
|
|
||||||
recomputeMicVoiceAvailability()
|
recomputeMicVoiceAvailability()
|
||||||
OSGDiag.log(
|
OSGDiag.log(
|
||||||
"beginFlowStart → openHostApp(startflow) recordAfterHandoff=\(recordAfterHandoff) "
|
"beginFlowStart → openHostApp(startflow) recordAfterHandoff=\(recordAfterHandoff) "
|
||||||
|
|||||||
@@ -280,17 +280,15 @@ public struct KeyboardRootView: View {
|
|||||||
}
|
}
|
||||||
|
|
||||||
private var buttonPhase: RecordButton.Phase {
|
private var buttonPhase: RecordButton.Phase {
|
||||||
if case .error = state.phase { return .error }
|
|
||||||
if case .denied = state.phase { return .error }
|
|
||||||
switch state.micVoiceAvailability {
|
switch state.micVoiceAvailability {
|
||||||
case .ready:
|
|
||||||
return .idleReady
|
|
||||||
case .unavailable:
|
|
||||||
return .idleUnavailable
|
|
||||||
case .recording:
|
case .recording:
|
||||||
return .recording
|
return .recording
|
||||||
case .processing:
|
case .processing:
|
||||||
return .processing
|
return .processing
|
||||||
|
case .ready, .unavailable:
|
||||||
|
// Host/PiP readiness is handled by an automatic app handoff. Keep
|
||||||
|
// the idle mic visually green instead of exposing startup state.
|
||||||
|
return .idleReady
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -409,7 +407,16 @@ private struct TranscriptLine: View {
|
|||||||
|
|
||||||
@ViewBuilder
|
@ViewBuilder
|
||||||
private var idleHint: some View {
|
private var idleHint: some View {
|
||||||
let isWarning = micVoiceAvailability.isUnavailable
|
let isWarning: Bool = {
|
||||||
|
switch micVoiceAvailability {
|
||||||
|
case .unavailable(.hostNotReady), .unavailable(.preparingSession):
|
||||||
|
return false
|
||||||
|
case .unavailable:
|
||||||
|
return true
|
||||||
|
case .ready, .recording, .processing:
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}()
|
||||||
Group {
|
Group {
|
||||||
switch micVoiceAvailability {
|
switch micVoiceAvailability {
|
||||||
case .ready:
|
case .ready:
|
||||||
@@ -417,17 +424,9 @@ private struct TranscriptLine: View {
|
|||||||
case .unavailable(.missingAPIKey):
|
case .unavailable(.missingAPIKey):
|
||||||
Text(micDisabledHint)
|
Text(micDisabledHint)
|
||||||
case .unavailable(.hostNotReady):
|
case .unavailable(.hostNotReady):
|
||||||
if FlowSessionPolicy.keepAliveMode() == .pictureInPicture {
|
ExtL10n.text("keyboard.placeholder.idle")
|
||||||
ExtL10n.text("keyboard.flow.sessionInactive.pip")
|
|
||||||
} else {
|
|
||||||
ExtL10n.text("keyboard.flow.sessionInactive")
|
|
||||||
}
|
|
||||||
case .unavailable(.preparingSession):
|
case .unavailable(.preparingSession):
|
||||||
if FlowSessionPolicy.keepAliveMode() == .pictureInPicture {
|
ExtL10n.text("keyboard.placeholder.idle")
|
||||||
ExtL10n.text("keyboard.flow.startingSession.pip")
|
|
||||||
} else {
|
|
||||||
ExtL10n.text("keyboard.flow.startingSession")
|
|
||||||
}
|
|
||||||
case .unavailable(.noFullAccess):
|
case .unavailable(.noFullAccess):
|
||||||
ExtL10n.text("keyboard.error.fullAccessRequired")
|
ExtL10n.text("keyboard.error.fullAccessRequired")
|
||||||
case .unavailable(.appGroupUnavailable):
|
case .unavailable(.appGroupUnavailable):
|
||||||
|
|||||||
@@ -25,15 +25,10 @@ public enum FlowColdStartOverlayDecision: Equatable, Sendable {
|
|||||||
}
|
}
|
||||||
|
|
||||||
public enum FlowHandoffPolicy {
|
public enum FlowHandoffPolicy {
|
||||||
/// Proactive keyboard auto-launch of the host is intentionally disabled.
|
/// Low-profile PiP is armed proactively whenever the keyboard appears.
|
||||||
/// Opening the host must be driven by an explicit mic press (or a Live
|
/// This opens the lightweight host only when its ready contract is absent;
|
||||||
/// Activity tap when that keep-alive mode is selected). PiP sessions
|
/// capture, ASR and model warm-up remain lazy until an actual mic press.
|
||||||
/// never auto-jump once `hostReady` is published.
|
public static let allowsProactiveHostAutoLaunch = true
|
||||||
///
|
|
||||||
/// Ready-wait polls may observe a dead host, but must still gate
|
|
||||||
/// `startflow` on mic intent (`recordWhenHostReady`) — otherwise an idle
|
|
||||||
/// keyboard open relaunches ASR and jetsams the extension.
|
|
||||||
public static let allowsProactiveHostAutoLaunch = false
|
|
||||||
|
|
||||||
/// Samples of "host truly dead" required before a cold-start jump is allowed
|
/// Samples of "host truly dead" required before a cold-start jump is allowed
|
||||||
/// from a mic-driven ready-wait recovery. Idle opens must never cold-start.
|
/// from a mic-driven ready-wait recovery. Idle opens must never cold-start.
|
||||||
|
|||||||
@@ -259,8 +259,8 @@ final class FlowHandoffPolicyTests: XCTestCase {
|
|||||||
|
|
||||||
// MARK: - debouncer + proactive launch flag
|
// MARK: - debouncer + proactive launch flag
|
||||||
|
|
||||||
func testProactiveAutoLaunchDisabled() {
|
func testProactiveAutoLaunchEnabledForLowProfilePiP() {
|
||||||
XCTAssertFalse(FlowHandoffPolicy.allowsProactiveHostAutoLaunch)
|
XCTAssertTrue(FlowHandoffPolicy.allowsProactiveHostAutoLaunch)
|
||||||
}
|
}
|
||||||
|
|
||||||
func testDebouncerIgnoresSingleDeadSample() {
|
func testDebouncerIgnoresSingleDeadSample() {
|
||||||
|
|||||||
+2
-2
@@ -51,8 +51,8 @@ settings:
|
|||||||
ASSETCATALOG_COMPILER_GENERATE_SWIFT_ASSET_SYMBOL_EXTENSIONS: YES
|
ASSETCATALOG_COMPILER_GENERATE_SWIFT_ASSET_SYMBOL_EXTENSIONS: YES
|
||||||
STRING_CATALOG_GENERATE_SYMBOLS: YES
|
STRING_CATALOG_GENERATE_SYMBOLS: YES
|
||||||
CLANG_CXX_LANGUAGE_STANDARD: c++17
|
CLANG_CXX_LANGUAGE_STANDARD: c++17
|
||||||
MARKETING_VERSION: "1.6.2"
|
MARKETING_VERSION: "1.6.5"
|
||||||
CURRENT_PROJECT_VERSION: "50"
|
CURRENT_PROJECT_VERSION: "51"
|
||||||
# 签名配置来自 Signing.local.xcconfig(gitignored,不会被覆盖)
|
# 签名配置来自 Signing.local.xcconfig(gitignored,不会被覆盖)
|
||||||
|
|
||||||
# 项目级签名 xcconfig,适用于所有 target
|
# 项目级签名 xcconfig,适用于所有 target
|
||||||
|
|||||||
Reference in New Issue
Block a user