OSGAccountServer/README.md

OSG Account Server

Kotlin 2.4.10 / Ktor 3.5.2 managed AI gateway and invitation page.

Included

• Controlled polish, ai, and agent requests through DeepSeek's OpenAI-compatible API.
• Volcengine SAUC v3 ASR over WebSocket with a strict binary frame codec.
• Credit reservation, settlement, release, identity, grant, and metadata persistence ports.
• Apple DeviceCheck trial enforcement and App Attest attestation/assertion validation.
• Bilingual invitation page at GET /i/{code} with no third-party analytics, tracking scripts, or fingerprinting.
• Privacy-minimized, idempotent product event ingestion and an internal growth, retention, AI usage, monetization, and referral dashboard.
• Flyway migrations for provider metadata, metered usage, gateway grants, and integrity state.

Prompts, audio, transcripts, and provider response bodies are never sent to the usage persistence port. Application logging must also keep request/response body logging disabled.

HTTP API

• GET /v1/gateway/catalog
• POST /v1/gateway/grants, POST /v1/gateway/grants/refresh, DELETE /v1/gateway/grants/{grantId}
• POST /v1/gateway/llm/{polish|ai|agent}
• POST /v1/gateway/asr
• POST /v1/gateway/asr/sessions, GET (WebSocket) /v1/gateway/asr/sessions/{sessionId}/stream
• POST /v1/integrity/challenges
• POST /v1/integrity/attest
• POST /v1/integrity/assert
• POST /v1/auth/apple
• POST /v1/auth/refresh, POST /v1/auth/logout
• POST /v1/analytics/events
• GET/DELETE /v1/account
• POST /v1/apple/events
• GET /v1/credits/{balance|ledger|rates}
• GET/POST /v1/referrals...
• GET /health, GET /health/live, GET /health/ready
• GET /i/{code}
• GET /.well-known/apple-app-site-association
• GET /apple-app-site-association
• GET /v1/admin/analytics through the private mTLS-protected admin edge

See docs/openapi.yaml for request limits and response formats.

Local development

JDK 21 is required. Start a local MySQL 8.4 database, copy .env.example to an untracked .env, replace every required placeholder, then export it into the shell before starting Ktor:

set -a
source .env
set +a
./gradlew test
./gradlew run
curl --fail http://127.0.0.1:8080/health/live

Development may set APP_ENV=development, APPLE_INTEGRITY_ENVIRONMENT=development, and both integrity enforcement flags to false. Production refuses to start with those values. Never commit .env, .p8 files, API keys, tokens, database dumps, or generated response data.

Application wiring

Application.kt is the composition root. It installs Ktor security, serialization, authentication, rate limiting, WebSockets, error handling and Koin; initializes Flyway/Exposed; and mounts auth, account, Apple event, credit, referral, gateway, invitation and health routes.

The first-party invitation page uses the registered read-only ReferralLookupPort adapter. It preserves case and checks code existence plus campaign validity; the composition root mounts exactly one /i/{code} route.

Gateway identity uses the signed session JWT and verifies the account plus session family in MySQL on every authenticated request. Logout, refresh-token replay response, Apple events, and account deletion therefore invalidate access tokens immediately. Gateway grants, provider metadata and metered usage are persisted in MySQL. Managed calls reserve, settle or release credits through the same immutable credit ledger exposed read-only by the public credit API. Clients create scope-limited, time-limited gateway grants explicitly. Signup trial credits are granted only after DeviceCheck bit0 is conservatively marked.

Apple integrity

DeviceCheck uses Apple's official query_two_bits and update_two_bits endpoints. ES256 bearer JWTs are created from APPLE_TEAM_ID, APPLE_KEY_ID, and APPLE_PRIVATE_KEY_PEM; select the matching development or production endpoint with APPLE_INTEGRITY_ENVIRONMENT. Bit0 means the device has consumed its signup trial, while bit1 is preserved. The plaintext device token is never persisted or logged. Because DeviceCheck tokens are ephemeral, a MySQL named lock serializes the complete query-update-confirm window across all server instances. Apple is marked and confirmed before the idempotent credit entry; an interruption may conservatively forfeit a trial but cannot duplicate one.

App Attest is bound to the configured Apple team, bundle, and environment. Cryptographic attestation, receipt, assertion, and certificate-chain validation use the official Apple App Attestation Root CA bundled from Apple Certificate Authority. The server stores the validated public key, receipt, and strictly increasing assertion counter.

Production accepts only production App Attest AAGUIDs by default. For a time-bounded physical-device test against the production service, set ALLOW_DEVELOPMENT_APP_ATTEST=true to admit development AAGUIDs from registered development builds. Disable the flag again after testing; TestFlight and App Store builds do not require it.

Request an attestation challenge after generateKey, then call /attest with the resulting CBOR object. For login assertions, request an assertion challenge and generate the assertion over SHA-256 of the canonical UTF-8 payload documented in docs/openapi.yaml. Challenges are single-use and expire after APP_ATTEST_CHALLENGE_TTL_SECONDS.

ENFORCE_DEVICE_CHECK and ENFORCE_APP_ATTEST fail closed for missing or unavailable evidence. MONITOR permits login during rollout or Apple outages, but unavailable DeviceCheck evidence never qualifies for trial credits. Cryptographically rejected evidence is never allowed by either mode. Production refuses to start unless both policies are ENFORCE and the production Apple environment is selected.

Account deletion and anti-abuse retention

Account deletion requires a fresh matching Sign in with Apple identity token, authorization code, and nonce. After reauthentication succeeds, local deletion is committed before the Apple revocation call. Sessions, referral state, credit balances and reservations, gateway grants, and provider metadata are removed through database cascades. The newly issued Apple refresh token moves encrypted into a durable outbox; a successful revoke clears it immediately, while failures retry with bounded exponential backoff.

The immutable credit ledger is retained with only the now-unmapped random account UUID for financial auditability. To prevent deletion/recreation trial and self-referral abuse, the server retains only an HMAC-SHA-256 identity tombstone—never the Apple subject—for IDENTITY_TOMBSTONE_RETENTION_DAYS (365 days by default). IDENTITY_HMAC_KEY is a separate secret and must not reuse JWT or field-encryption keys. Tombstoned identities may sign in but cannot receive another signup trial or participate in referrals during the retention window.

Apple server events accept both the documented account-delete name and the previously observed account-deleted variant.

Gateway execution and settlement rules:

• X-Request-ID is atomically claimed within the authenticated account and can never invoke a provider twice. Replays return 409, including claimed, started, pending, settled, and released calls.
• LLM requests reserve a UTF-8 byte upper bound plus maximum output tokens, then require internally consistent provider input, output, and total token usage.
• Raw PCM/WAV reservations are derived from audio bytes. Compressed audio reserves the full accepted ten-minute boundary because its duration cannot be proven from byte length alone.
• Only failures before provider completion release a reservation.
• Provider success is durably moved to SETTLEMENT_PENDING before settlement. Settlement failures keep credits frozen and the background reconciliation job retries idempotently.
• A successful settlement is returned to the client even if usage metadata needs a later retry.
• Settlement retries must repeat the same actual usage; parameter drift returns a conflict.
• Public idempotency keys are SHA-256 namespaced before storage and cannot collide with internal referral, signup-trial, or gateway ledger keys.

Provider configuration

Existing application.yaml values can be mapped into:

• DeepSeekConfig(endpoint, apiKey, model, reasoningModel)
• VolcengineAsrConfig(endpoint, resourceId, appId, accessToken)
• InviteWebConfig(appStoreUrl, appleAppId, universalLinkBaseUrl)

VolcengineAsrConfig also supports the newer single apiKey credential when the application configuration exposes it. Missing credentials, non-TLS provider URLs, missing resource IDs, malformed frames, provider error frames, missing final frames, and missing metered duration all fail closed. Production additionally requires the exact provider hosts api.deepseek.com and openspeech.bytedance.com on the default TLS port.

Configuration ownership:

• Apple: APPLE_TEAM_ID, APPLE_KEY_ID, APPLE_CLIENT_ID, APPLE_PRIVATE_KEY_PEM, APPLE_INTEGRITY_ENVIRONMENT, plus the two integrity enforcement flags.
• Volcengine: prefer VOLCENGINE_API_KEY; set the SAUC v3 VOLCENGINE_RESOURCE_ID and WSS VOLCENGINE_ASR_ENDPOINT. The legacy app ID/access token pair remains optional.
• DeepSeek: set DEEPSEEK_API_KEY, the provisioned low-latency DEEPSEEK_MODEL, and HTTPS DEEPSEEK_ENDPOINT. DEEPSEEK_REASONING_MODEL is optional and falls back to DEEPSEEK_MODEL.

Gateway text requests may include the optional stable taskKind values documented in docs/openapi.yaml. The server maps capability + taskKind to a deterministic execution policy; it never infers task type from user content. Polish and transform tasks explicitly disable DeepSeek thinking and do not retry an empty buffered result. AI questions and agent planning explicitly use high-effort thinking. Ordinary AI questions allow model-selected DeepSeek Responses web search; current_information_question requires web search. Search failures fall back to Chat Completions before any result is emitted. Other tools remain disabled, and search usage is billed only through the provider-reported LLM input and output Token counts.

Store production values in 1Panel's secret/environment facility. The Compose environment receives them at runtime because this application does not read Docker /run/secrets/* files directly.

SAUC v3 protocol boundary

The endpoint uses the SAUC v3 API, while the documented binary header version nibble remains 1. The codec implements:

• 4-byte base header and big-endian integer fields.
• Full-client JSON request (0x1) with gzip.
• Audio-only request (0x2) with gzip and final-packet flag 0x2.
• Full-server response (0x9) with a required strictly increasing sequence and negative final sequence.
• Error response (0xF) with error-code field.
• Strict declared-size, payload-limit, gzip, message-type, and final-frame validation.

The client supports pcm, wav, ogg, and mp3; 16 kHz, 16-bit, one or two channels; and a maximum duration of ten minutes. Only the final frame can supply billable audio_info.duration. Clients may use the buffered HTTP endpoint or create a one-shot downstream WebSocket session. Both paths durably claim the request ID before contacting Volcengine and share the same reservation and settlement state machine.

Database

The application runs Flyway against the existing MySQL 8.4 instance before opening the runtime pool. V1-V7 remain append-only migration history; the invitation web feature requires no new table and does not modify V4. V5__account_security_hardening.sql adds account foreign keys, identity tombstones, and the encrypted Apple revocation outbox. V6__gateway_execution_state.sql scopes request IDs by account, binds provider requests to credit reservations, and stores only metering metadata needed for settlement reconciliation. No table has a request-body, prompt, audio, transcript, or response-body column.

Container deployment

The production Compose file starts only account-server; it does not create a second MySQL. The application port is bound to 127.0.0.1:18080, and OpenResty is the only public entry point.

Local build:

./gradlew clean build
docker build -t osg-account-server:local .

Deployment:

# Create this only when the existing MySQL is containerized and the network is absent.
docker network create --internal account-backend

# Authenticate once because the GHCR package is private.
echo "$GHCR_TOKEN" | docker login ghcr.io -u hkgood --password-stdin

# Put deployment values in an uncommitted .env or 1Panel secret/environment store.
docker compose config --quiet
docker compose pull
docker compose up -d
docker compose ps
curl --fail http://127.0.0.1:18080/health/ready

Every successful main CI run publishes ghcr.io/hkgood/osg-account-server:main plus an immutable sha-<commit> tag. Production should pin a tested immutable tag in IMAGE_TAG; use main only for initial staging. GHCR_TOKEN needs package-read permission and must not be stored in .env.

If MySQL runs directly on the host or another private server, keep the external network declaration but set DATABASE_URL to a private hostname reachable from that network. The application also joins the separate egress network for Apple and provider HTTPS calls. Never publish MySQL port 3306 to the Internet.

Required environment

• Public URLs: PUBLIC_BASE_URL, INVITE_BASE_URL, APP_STORE_URL.
• Existing MySQL: DATABASE_URL, runtime DATABASE_USER / DATABASE_PASSWORD, migration-only DATABASE_MIGRATION_USER / DATABASE_MIGRATION_PASSWORD, and DATABASE_POOL_SIZE. Keep connectionTimeZone=UTC&forceConnectionTimeZoneToSession=true in the JDBC URL.
• Session and encryption secrets: JWT_SECRET, FIELD_ENCRYPTION_KEY, IDENTITY_HMAC_KEY. The two Base64 keys must be distinct; the field key decodes to exactly 32 bytes.
• Apple: APPLE_TEAM_ID, APPLE_KEY_ID, APPLE_CLIENT_ID, APPLE_PRIVATE_KEY_PEM, APPLE_INTEGRITY_ENVIRONMENT.
• Providers: VOLCENGINE_API_KEY, VOLCENGINE_RESOURCE_ID, DEEPSEEK_API_KEY, DEEPSEEK_MODEL; optionally DEEPSEEK_REASONING_MODEL.
• Production controls: APP_ENV=production, ENFORCE_DEVICE_CHECK=true, ENFORCE_APP_ATTEST=true.
• Optional tuning: token lifetimes, gateway grant days, credit values, binding window and pool size; defaults are visible in compose.yaml.

All values in the Compose file are placeholders or environment substitutions. Keep real credentials out of Git, Compose YAML, 1Panel screenshots and shell history.

The Apple ES256 .p8 material is supplied as APPLE_PRIVATE_KEY_PEM with literal \n line breaks; the DeviceCheck JWS adapter reuses that configured Team ID, Key ID and private key. Keep Volcengine and DeepSeek API keys in separate 1Panel secret fields and rotate each provider independently.

Existing MySQL minimum privileges

The server uses a Flyway-only account for migrations and a separately pooled runtime account. Apply docs/mysql-minimum-privileges.sql as an administrator after replacing its private host pattern and generated passwords. The runtime account can append to ledger and usage history but cannot update/delete that history or alter rate versions or schemas. The migration account is scoped to osg_account.* and has no global privileges.

Replace the example 10.20.% host pattern with the actual container subnet. Do not grant either account FILE, PROCESS, SUPER, CREATE USER, GRANT OPTION, or access from %.

1Panel and firewall

1. Create HTTPS sites for account.osglab.com and osglab.com in 1Panel.
2. Apply deploy/openresty-account.conf, adjusting certificate/root paths when needed.
3. Validate with openresty -t; AASA is proxied to Ktor at both extensionless paths.
4. Point both DNS records to the server and reload OpenResty after certificate issuance.
5. Allow inbound TCP 80/443 only; allow SSH only from an administrator IP. Do not expose 18080, 8080 or 3306. Permit required outbound HTTPS for Apple, DeepSeek and Volcengine.

See docs/DEPLOYMENT.md for rollback, verification and Universal Link checks.

Invitation link architecture

Invitation links use the first-party Universal Link https://osglab.com/i/{code}. The page contains no third-party JavaScript, analytics, tracking pixels or external fonts. Firebase Dynamic Links and Branch are deliberately not used.

Tests

Run:

./gradlew test

Gateway tests cover account-scoped replay exclusion, cross-account isolation, provider failure release, settlement pending behavior, reconciliation, whole-call timeout, forged audio duration, malformed SAUC sequences, and final-frame-only duration.

The MySQL Testcontainers suite validates all migrations, account-deletion cascades (including linked analytics events), stale access-token rejection, real InnoDB concurrent balance locking, and exactly-once referral rewards. Analytics tests additionally cover event validation, idempotent replay, conflict handling, offline timestamps, and privacy-minimized responses. It is automatically skipped when Docker is unavailable; CI requires Docker and runs it on every build.

Staged acceptance

1. Local: tests pass, malformed/unknown/referral-lookup-failure paths fail closed, and the page makes no third-party requests.
2. Container: UID/GID 10001, read-only root filesystem, dropped capabilities, and both health endpoints pass.
3. Pre-production: migrations use only the migrator; runtime uses only its table grants; HTTP and WSS reverse proxy tests pass.
4. Apple: AASA returns JSON without redirects; a physical device opens the Universal Link; an uninstalled device receives the responsive bilingual page.
5. Production: only 80/443 are public, invitation paths are excluded from access logs, provider credentials work, and an encrypted backup has been restored successfully in isolation.

See docs/DEPLOYMENT.md for exact rollout commands and docs/BACKUP.md for backup and recovery.