rocky/OSGKeyboard
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: comprehensive rewrite — push-to-talk pipeline, Typeless UI, Chinese

Browse Source 
This is a major rewrite of OpenLessKeyboard, renamed to OSGKeyboard
and rebuilt end-to-end. 59 files changed (+3205/-1550).

Architecture
------------
- Rename project, targets, directories from OpenLess* to OSGKeyboard*
  (OpenLess / OpenLessKeyboard / OpenLessShared / OpenLessTests).
- AudioCaptureService rewritten as @unchecked Sendable class with
  OSAllocatedUnfairLock instead of an actor, so it survives Swift 6
  strict-concurrency checks while still serialising engine + converter
  state correctly.
- Single design system (Palette / Spacing / Radius / TypeStyle /
  Motion) lifted into OSGKeyboardShared so the host app and the
  keyboard extension stay in lock-step.

Push-to-talk — first-principles fix
-----------------------------------
- App Group + audio-input entitlements were stripped by Xcode's
  Automatic Signing. They are now declared in project.yml so
  'xcodegen generate' re-emits them every time. iOS Developer
  Account is untouched; only the App Group capability was added.
- State machine uses a real stored `phase` (was a derived shim
  that locked out every press after the first because
  recordStream was never nilled after the pipeline finished).
- Microphone permission is requested inside pressBegan (async
  Task) so the press flow optimistically enters .recording;
  permission denial surfaces a short error and returns to idle.
- Replaced LongPressGesture(0.15s) with a DragGesture +
  TapGesture pair separated by pressArmed, so a single tap no
  longer fires both onPressBegan and onTap simultaneously.
- Real RMS / peak level meter from the AVAudioEngine tap (was a
  pseudo-random walk); the visible waveform is now driven by
  actual audio.
- SFSpeechRecognizer(locale:) with selectable ASR locales
  (auto / zh-Hans / zh-Hant / en-US / ja-JP / ko-KR) for
  first-class Chinese / English / Japanese / Korean dictation,
  with on-device recognition when supported.
- AVAudioSession now deactivates on stop so other apps' audio
  routing is restored.

Keyboard UI — Typeless-inspired layout
---------------------------------------
- Hero area is 280 pt with a 96 pt record disc, breathing outer
  ring, and a 12-bar waveform driven by the real RMS.
- inputView.allowsSelfSizing + a heightAnchor constraint so iOS
  no longer crops the keyboard under the Spotlight bar / home
  indicator.
- Top bar: mode chip (Off / 转写 / 润色) + locale chip
  (Auto / 简体 / 繁體 / EN / 日 / 한) + status badge + ⚙.
- Bottom bar: globe / delete / 空格 / return — all 40 pt and
  balanced.
- RecordButton onPressEnded is now safe to fire from a quick
  press; pressArmed prevents double-firing.

LLM / Polishing
---------------
- LLMClient: stopped leaking the server response body in errors
  (server body is now logged at debug, never surfaced to UI);
  added a dedicated .rateLimited case for 429.
- PolishingService timeout 8s → 12s to accommodate slower
  domestic LLM providers.
- AppGroupStore.defaultSystemPrompt is now provider-aware
  (Chinese for zhipu/moonshot/qwen/deepseek, English otherwise).

Onboarding & Settings
---------------------
- Re-themed OnboardingView / HomeView / SettingsView on the
  new design system.
- ProviderPickerSection now shows 6 providers (OpenAI, DeepSeek,
  Qwen DashScope, 智谱 GLM, 月之暗面 Moonshot, Custom) with
  blurb + selected accent.
- PickerRow for Mode and ASR locale; System Prompt editor with
  reset-to-default.
- API settings page "Get an API key" used SwiftUI Link, which
  has a hit-test bug on iOS 18 that ate gestures from adjacent
  TextFields (manifested as "typing jumps to a website"). It is
  now an explicit Button + contentShape + .submitLabel(.done) on
  the fields.

Polish & tests
--------------
- LLMClientTests: 4 unit tests passing (ProviderConfig
  persistence + OpenAI request/response + HTTP error + missing
  key); test App Group renamed to the correct identifier.
- ProviderConfig.apply now captures the previous provider id
  *before* mutating, so switching providers actually resets the
  system prompt to the new default.

Build
-----
- Swift 6 strict concurrency, iOS 18.0 deployment target.
- Tested on Xcode 26 + iPhone 17 Pro simulator. A real device on
  iOS 27 beta aborts with __abort_with_payload (dispatch
  library ABI mismatch); use an iOS 18 real device or the
  iOS 26 simulator for now.

🤖 Generated with Claude Code
This commit is contained in:
Rocky
2026-06-18 01:22:12 +08:00
parent 07067ca6c5
commit bec36befa2
59 changed files with 3205 additions and 1550 deletions
Download Patch File Download Diff File Expand all files Collapse all files
OSGKeyboardExt/KeyboardViewController.swift
+392
View File
@@ -0,0 +1,392 @@
// KeyboardViewController.swift
// OSGKeyboard · Keyboard Extension
//
// Principal class for the Custom Keyboard Extension. Hosts a single
// SwiftUI tree (`KeyboardRootView`) and drives the recording pipeline:
//
// AudioCaptureService ──► ASRService ──► PolishingService ──► insertText
//
// Design notes:
// • The class is `@MainActor` — every UI mutation and `textDocumentProxy`
// call must happen on main, and Swift 6 strict concurrency forces this.
// • State is a single `State` ObservableObject; SwiftUI observes it via
// `@ObservedObject` so we never re-create the hosting root on each tick.
// • `phase` is a real stored property (no derivation) — the previous
// "derive from recordStream" shim locked out every press after the first.
// • Microphone permission is requested *inside* pressBegan, but we still
// start the rest of the press flow optimistically; if permission is
// denied we surface a short error and drop back to idle cleanly.
import UIKit
import SwiftUI
import AVFoundation
import OSGKeyboardShared
@objc(KeyboardViewController)
@MainActor
public final class KeyboardViewController: UIInputViewController {
// MARK: - View model
@MainActor
public final class State: ObservableObject {
public init() {}
public enum Phase: Equatable {
case idle
case recording
case processing
case error(String)
}
public enum InputMode: String, CaseIterable, Identifiable {
case off
case transcribe
case polish
public var id: String { rawValue }
public var labelKey: String {
switch self {
case .off: return "mode.off"
case .transcribe: return "mode.transcribe"
case .polish: return "mode.polish"
}
}
}
@Published public var phase: Phase = .idle
@Published public var level: Double = 0
@Published public var mode: InputMode = .polish
@Published public var localeId: String = "auto"
@Published public var lastTranscript: String = ""
// Action hooks — injected by the view controller at install time.
var beginRecording: () -> Void = {}
var endRecording: () -> Void = {}
var tapMic: () -> Void = {} // tap on the mic area (advances keyboard)
var openSettings: () -> Void = {}
var setMode: (InputMode) -> Void = { _ in }
var setLocale: (String) -> Void = { _ in }
var insertNewline: () -> Void = {}
var insertSpace: () -> Void = {}
var deleteBackward: () -> Void = {}
// MARK: - Preview helpers
#if DEBUG
static var previewIdle: State {
let s = State()
s.phase = .idle
s.level = 0
s.mode = .polish
s.localeId = "zh-Hans"
s.lastTranscript = ""
return s
}
static var previewRecording: State {
let s = State()
s.phase = .recording
s.level = 0.65
s.mode = .polish
s.localeId = "zh-Hans"
s.lastTranscript = "你好,我想说一段测试"
return s
}
static var previewProcessing: State {
let s = State()
s.phase = .processing
s.level = 0
s.mode = .polish
s.localeId = "zh-Hans"
s.lastTranscript = ""
return s
}
#endif
}
// MARK: - State
private let state = State()
private let audio = AudioCaptureService()
private let asr: ASRService = ASRServiceFactory.make()
private let polisher = PolishingService()
private var session: AudioCaptureService.Session?
private var asrTask: Task<Void, Never>?
private var levelTask: Task<Void, Never>?
private var didRequestMicOnce: Bool = false
private var hosting: UIHostingController<KeyboardRootView>!
// MARK: - Lifecycle
public override func viewDidLoad() {
super.viewDidLoad()
// iOS 18 keyboard extension MUST opt in to self-sizing, otherwise
// our SwiftUI `frame(height:)` is ignored and the keyboard is
// cropped by the system chrome (Spotlight bar, home indicator).
inputView?.allowsSelfSizing = true
installStateActions()
installSwiftUI()
loadPersistedLocale()
}
public override func viewWillDisappear(_ animated: Bool) {
super.viewWillDisappear(animated)
cancelPipeline()
}
public override func didReceiveMemoryWarning() {
super.didReceiveMemoryWarning()
cancelPipeline()
}
public override func textDidChange(_ textInput: (any UITextInput)?) {
super.textDidChange(textInput)
// Hook for future per-app mode switching (e.g. password field → .off).
}
// MARK: - Wiring
private func installStateActions() {
state.beginRecording = { [weak self] in self?.pressBegan() }
state.endRecording = { [weak self] in self?.pressEnded() }
state.tapMic = { [weak self] in self?.advanceToNextInputMode() }
state.openSettings = { [weak self] in self?.openHostApp() }
state.setMode = { [weak self] m in self?.persistMode(m) }
state.setLocale = { [weak self] l in self?.persistLocale(l) }
state.insertNewline = { [weak self] in self?.textDocumentProxy.insertText("\n") }
state.insertSpace = { [weak self] in self?.textDocumentProxy.insertText(" ") }
state.deleteBackward = { [weak self] in self?.textDocumentProxy.deleteBackward() }
}
private func installSwiftUI() {
let root = KeyboardRootView(state: state)
let host = UIHostingController(rootView: root)
host.view.backgroundColor = .clear
host.view.translatesAutoresizingMaskIntoConstraints = false
addChild(host)
view.addSubview(host.view)
NSLayoutConstraint.activate([
host.view.leadingAnchor.constraint(equalTo: view.leadingAnchor),
host.view.trailingAnchor.constraint(equalTo: view.trailingAnchor),
host.view.topAnchor.constraint(equalTo: view.topAnchor),
host.view.bottomAnchor.constraint(equalTo: view.bottomAnchor),
// Pin the host view to a fixed height matching KeyboardRootView.totalHeight.
// Without this, iOS lets the system chrome (Spotlight, home
// indicator) bleed into our content. With it, our content area
// is fully reserved and the keyboard feels intentional.
host.view.heightAnchor.constraint(equalToConstant: KeyboardRootView.totalHeight)
])
host.didMove(toParent: self)
self.hosting = host
}
private func loadPersistedLocale() {
let store = AppGroupStore()
let id = store.localeId
state.localeId = id
state.mode = State.InputMode(rawValue: store.modeId) ?? .polish
}
// MARK: - Press handlers
private func pressBegan() {
guard state.phase == .idle else { return }
guard state.mode != .off else { return }
// We optimistically enter `.recording`; the capture session will yield
// frames on its own queue, so even if mic permission takes a beat the
// user already feels the press registered.
Task { @MainActor [weak self] in
guard let self else { return }
let granted = await self.requestMicPermission()
guard granted else {
self.state.phase = .error("麦克风被拒绝,请到「设置」中允许")
self.scheduleAutoClearError()
return
}
self.startPipeline()
}
}
private func pressEnded() {
guard state.phase == .recording else { return }
stopPipeline()
}
// MARK: - Pipeline
private func startPipeline() {
let session = audio.start()
self.session = session
state.phase = .recording
state.level = 0
state.lastTranscript = ""
let locale = resolveLocale(state.localeId)
let events = asr.transcribe(stream: session.audio, locale: locale)
asrTask = Task { @MainActor [weak self] in
guard let self else { return }
var lastPartial: String = ""
for await event in events {
switch event {
case .partial(let s):
lastPartial = s
self.state.lastTranscript = s
case .final(let s):
let transcript = s.isEmpty ? lastPartial : s
self.handleFinalTranscript(transcript)
case .error(let m):
self.state.phase = .error("ASR: \(m)")
self.scheduleAutoClearError()
}
}
}
levelTask = Task { @MainActor [weak self] in
for await level in session.levels {
guard let self else { return }
// Smooth a little extra to feel natural.
self.state.level = Double(self.state.level) * 0.6 + Double(level.meter) * 0.4
}
}
}
private func stopPipeline() {
session?.stop()
session = nil
asrTask?.cancel(); asrTask = nil
levelTask?.cancel(); levelTask = nil
}
private func cancelPipeline() {
stopPipeline()
asr.cancel()
if state.phase == .recording || state.phase == .processing {
state.phase = .idle
}
state.level = 0
}
private func handleFinalTranscript(_ transcript: String) {
let trimmed = transcript.trimmingCharacters(in: .whitespacesAndNewlines)
guard !trimmed.isEmpty else {
state.phase = .idle
return
}
// In `.transcribe` mode, skip the LLM and insert raw.
if state.mode == .transcribe {
textDocumentProxy.insertText(trimmed)
state.lastTranscript = ""
state.phase = .idle
return
}
// `.polish` (default): call the LLM.
state.phase = .processing
Task { @MainActor [weak self] in
guard let self else { return }
do {
let polished = try await self.polisher.polish(trimmed)
self.textDocumentProxy.insertText(polished)
self.state.lastTranscript = ""
self.state.phase = .idle
} catch {
// Fall back to raw transcript on any failure.
self.textDocumentProxy.insertText(trimmed)
self.state.lastTranscript = ""
let msg = (error as? LocalizedError)?.errorDescription
?? "Polishing failed — inserted raw."
self.state.phase = .error(msg)
self.scheduleAutoClearError()
}
}
}
// MARK: - Persistence
private func persistMode(_ m: State.InputMode) {
state.mode = m
AppGroupStore().setModeId(m.rawValue)
}
private func persistLocale(_ id: String) {
state.localeId = id
AppGroupStore().setLocaleId(id)
}
// MARK: - Permissions
private func requestMicPermission() async -> Bool {
if #available(iOS 17.0, *) {
switch AVAudioApplication.shared.recordPermission {
case .granted: return true
case .denied: return false
case .undetermined:
if !didRequestMicOnce {
didRequestMicOnce = true
return await AVAudioApplication.requestRecordPermission()
}
return false
@unknown default: return false
}
} else {
let session = AVAudioSession.sharedInstance()
switch session.recordPermission {
case .granted: return true
case .denied: return false
case .undetermined:
if !didRequestMicOnce {
didRequestMicOnce = true
return await withCheckedContinuation { cont in
session.requestRecordPermission { cont.resume(returning: $0) }
}
}
return false
@unknown default: return false
}
}
}
// MARK: - Open host app
private func openHostApp() {
let urlString = "osgkeyboard://settings"
if let url = URL(string: urlString) {
var responder: UIResponder? = self
while let r = responder {
if let app = r as? UIApplication {
app.open(url)
return
}
responder = r.next
}
}
if let url = URL(string: UIApplication.openSettingsURLString) {
var responder: UIResponder? = self
while let r = responder {
if let app = r as? UIApplication {
app.open(url); return
}
responder = r.next
}
}
}
// MARK: - Helpers
private func resolveLocale(_ id: String) -> Locale {
if id == "auto" { return .current }
return Locale(identifier: id)
}
private func scheduleAutoClearError() {
Task { @MainActor [weak self] in
try? await Task.sleep(nanoseconds: 2_400_000_000)
guard let self else { return }
if case .error = self.state.phase {
self.state.phase = .idle
}
}
}
}