Add privacy-safe product analytics
Establish an idempotent analytics pipeline and internal decision dashboard while keeping event metadata allowlisted and account deletion enforceable.
This commit is contained in:
@@ -0,0 +1,175 @@
|
||||
# OSGKeyboard product analytics metrics dictionary
|
||||
|
||||
This document is the canonical definition of product metrics. All dates and
|
||||
cohorts use UTC calendar boundaries. Counts are based on distinct accounts when
|
||||
an installation is linked, otherwise on the pseudonymous installation.
|
||||
|
||||
## North-star metric
|
||||
|
||||
### Weekly AI active users (WAIU)
|
||||
|
||||
The number of distinct users that successfully complete at least one AI feature
|
||||
during a UTC calendar week.
|
||||
|
||||
- Managed AI and ASR use server-settled `credit_usage_records`.
|
||||
- Local and BYOK use accepted `AI_FEATURE_SUCCEEDED` client events.
|
||||
- Managed client success events provide feature breakdowns but are not added to
|
||||
the server-settled total, preventing double counting.
|
||||
- Week-over-week growth is `(current WAIU - previous WAIU) / previous WAIU`.
|
||||
A missing previous population is reported without a percentage.
|
||||
|
||||
## Growth and activation
|
||||
|
||||
### New installations
|
||||
|
||||
Distinct installations whose first accepted `FIRST_OPEN` event occurred in the
|
||||
selected period. Acquisition channel is fixed by the first non-`UNKNOWN`
|
||||
channel observed for the installation.
|
||||
|
||||
Allowed channels:
|
||||
|
||||
- `APP_STORE_ORGANIC`
|
||||
- `REFERRAL`
|
||||
- `SOCIAL_CONTENT`
|
||||
- `UNKNOWN`
|
||||
|
||||
### New accounts
|
||||
|
||||
Accounts whose `accounts.created_at` falls in the selected period.
|
||||
|
||||
### 24-hour AI activation rate
|
||||
|
||||
The percentage of new installations that successfully complete any AI feature
|
||||
within 24 hours of their first open. The numerator uses the same value-event
|
||||
rules as WAIU.
|
||||
|
||||
### Time to first value
|
||||
|
||||
Elapsed time from `FIRST_OPEN` to the first successful AI feature. The dashboard
|
||||
reports the median in minutes. Users without a successful AI feature are not
|
||||
included in the median and remain visible in the activation denominator.
|
||||
|
||||
## Activity
|
||||
|
||||
### AI DAU, WAU and MAU
|
||||
|
||||
Distinct value-active users in the last 1, 7 and 30 UTC days ending at the
|
||||
report's `until` timestamp.
|
||||
|
||||
### DAU/MAU stickiness
|
||||
|
||||
`AI DAU / AI MAU`. The value is null when MAU is zero.
|
||||
|
||||
### Successful AI requests
|
||||
|
||||
The sum of settled managed requests and successful local/BYOK client events.
|
||||
Managed client success events are excluded from this total.
|
||||
|
||||
### Successful AI requests per active user
|
||||
|
||||
`successful AI requests / distinct value-active users` for the selected period.
|
||||
|
||||
## Retention
|
||||
|
||||
The cohort date is the UTC date of a user's first successful AI feature.
|
||||
Retention is value retention, not application-open retention.
|
||||
|
||||
- `D1`: active on cohort date + 1 day.
|
||||
- `D7`: active on cohort date + 7 days.
|
||||
- `D30`: active on cohort date + 30 days.
|
||||
|
||||
Each retention rate uses the original cohort size as denominator. A day that has
|
||||
not fully elapsed at the report's `until` timestamp is returned as unavailable,
|
||||
not zero. Channel and first-feature breakdowns are optional dimensions and must
|
||||
not alter the base cohort definition.
|
||||
|
||||
## AI feature usage
|
||||
|
||||
Allowed feature types:
|
||||
|
||||
- `TRANSCRIPTION`
|
||||
- `POLISH`
|
||||
- `AI_ASSISTANT`
|
||||
- `AGENT`
|
||||
- `HOTWORD`
|
||||
- `OTHER`
|
||||
|
||||
Allowed execution modes:
|
||||
|
||||
- `MANAGED`
|
||||
- `LOCAL`
|
||||
- `BYOK`
|
||||
|
||||
Feature distributions use accepted client events because server billing only
|
||||
distinguishes `ASR` and `LLM`. Server-settled aggregates remain authoritative
|
||||
for managed totals, credits, token counts and ASR duration.
|
||||
|
||||
## Credit consumption
|
||||
|
||||
### Daily total credit consumption
|
||||
|
||||
The sum of non-negative `credit_usage_records.charged_credits` by UTC date.
|
||||
|
||||
### Average daily credits per AI active user
|
||||
|
||||
For each UTC date, divide total settled credits by distinct managed AI users,
|
||||
then average those daily values across days containing at least one active user.
|
||||
|
||||
### Median user-day credits
|
||||
|
||||
The median of per-account daily settled credits. This is shown beside the mean
|
||||
to prevent a small number of heavy users from distorting typical consumption.
|
||||
|
||||
### Average credits per managed request
|
||||
|
||||
`settled credits / settled managed requests`. Local and BYOK events consume no
|
||||
server credits and are excluded.
|
||||
|
||||
## Monetization
|
||||
|
||||
### 7-day and 30-day free-to-paid conversion
|
||||
|
||||
The percentage of newly registered accounts with a first credited StoreKit
|
||||
purchase no later than 7 or 30 days after registration. Cohorts whose conversion
|
||||
window has not elapsed are reported separately from mature cohorts.
|
||||
|
||||
### Paying users
|
||||
|
||||
Distinct accounts with at least one credited StoreKit purchase in the period.
|
||||
|
||||
### Repeat purchase rate
|
||||
|
||||
The percentage of paying accounts with at least two credited StoreKit purchases
|
||||
across their lifetime.
|
||||
|
||||
StoreKit transaction count and granted credits are operational proxies. Net
|
||||
revenue, App Store commission and refunds require App Store financial data and
|
||||
are outside this service's first version.
|
||||
|
||||
## Referral funnel
|
||||
|
||||
The ordered growth funnel is:
|
||||
|
||||
1. `REFERRAL_SHARED` distinct sharing installations.
|
||||
2. Invitation opens: accepted `INVITE_OPENED` client events plus anonymous
|
||||
first-party invitation page views. Page views are aggregate requests rather
|
||||
than distinct people and must be interpreted as a directional funnel signal.
|
||||
3. Referral-bound accounts.
|
||||
4. Referral-bound accounts that reach their first value event.
|
||||
5. Rewarded referral bindings.
|
||||
|
||||
Pending and ineligible bindings are parallel status counts, not sequential
|
||||
funnel steps.
|
||||
|
||||
## Experience guardrails
|
||||
|
||||
- AI success rate: successful client AI completions divided by starts with a
|
||||
terminal success or failure event.
|
||||
- Managed request failure rate: terminal non-settled `provider_requests` divided
|
||||
by terminal managed requests.
|
||||
- P50/P95 latency: client duration bucket distribution for all modes; exact
|
||||
server duration percentiles may be added later.
|
||||
- Credit-blocked users: distinct installations reporting
|
||||
`INSUFFICIENT_CREDITS` during the period.
|
||||
|
||||
Guardrails are diagnostic and never count as value-active events.
|
||||
Reference in New Issue
Block a user