Correct product analytics cohorts and reporting
CI / verify (push) Has been cancelled
CI / publish (push) Has been cancelled

This commit is contained in:
Rocky
2026-08-21 17:50:35 +08:00
parent b25f5ae6e9
commit edd0d9feca
31 changed files with 1134 additions and 262 deletions
+46 -15
View File
@@ -4,6 +4,10 @@ This document is the canonical definition of product metrics. All dates and
cohorts use UTC calendar boundaries. Counts are based on distinct accounts when
an installation is linked, otherwise on the pseudonymous installation.
Admin presets cover exactly 7, 30, or 90 UTC calendar dates, starting at 00:00
on the first date and ending at the current instant. The current UTC date is
therefore explicitly partial.
## North-star metric
### Weekly AI active users (WAIU)
@@ -39,9 +43,10 @@ Accounts whose `accounts.created_at` falls in the selected period.
### 24-hour AI activation rate
The percentage of new installations that successfully complete any AI feature
within 24 hours of their first open. The numerator uses the same value-event
rules as WAIU.
The percentage of new installations that have completed their full 24-hour
observation window and successfully complete any AI feature within 24 hours of
their first open. Unmatured installations are excluded from both numerator and
denominator. Managed usage before that installation's first open is ignored.
### Time to first value
@@ -49,6 +54,14 @@ Elapsed time from `FIRST_OPEN` to the first successful AI feature. The dashboard
reports the median in minutes. Users without a successful AI feature are not
included in the median and remain visible in the activation denominator.
### 24-hour growth funnel
A strict cohort of installations with a completed 24-hour observation window:
first open, account registration after first open, first AI value event after
registration, and first server-verified purchase after that value event. Every
downstream step must occur within 24 hours of first open. D7 belongs only to the
retention report and is not mixed into this funnel.
## Activity
### AI DAU, WAU and MAU
@@ -69,6 +82,13 @@ Managed client success events are excluded from this total.
`successful AI requests / distinct value-active users` for the selected period.
### Registered product-active users
The operations overview counts distinct registered accounts with either a
successful AI value event (managed, local, or BYOK) or a finalized manual
keyboard-input summary in the selected period. The displayed rate divides this
population by all registered accounts.
## Keyboard input usage
Keyboard input metrics use finalized UTC-day summaries produced on-device.
@@ -175,8 +195,11 @@ server credits and are excluded.
### 7-day and 30-day free-to-paid conversion
The percentage of newly registered accounts with a first credited StoreKit
purchase no later than 7 or 30 days after registration. Cohorts whose conversion
window has not elapsed are reported separately from mature cohorts.
purchase no later than 7 or 30 days after registration. The selected report
period filters when each observation window matures: a 7-day report cohort uses
registrations shifted exactly 7 days earlier, and the 30-day cohort is shifted
30 days earlier. This keeps every denominator fully observed and makes the rate
available even when the selected preset is no longer than the conversion window.
### Paying users
@@ -187,21 +210,29 @@ Distinct accounts with at least one credited StoreKit purchase in the period.
The percentage of paying accounts with at least two credited StoreKit purchases
across their lifetime.
### Purchase intent funnel
A strict installation cohort: `PURCHASE_VIEWED`, followed by
`PURCHASE_STARTED`, followed by a StoreKit purchase verified by the server for
the linked account. Each event must occur after the previous step and before the
report's `until`. `PURCHASE_CANCELLED` is a separate signal, not a funnel step.
StoreKit transaction count and granted credits are operational proxies. Net
revenue, App Store commission and refunds require App Store financial data and
are outside this service's first version.
## Referral funnel
The ordered growth funnel is:
The ordered cohort contains bindings created in the selected period:
1. `REFERRAL_SHARED` distinct sharing installations.
2. Invitation opens: accepted `INVITE_OPENED` client events plus anonymous
first-party invitation page views. Page views are aggregate requests rather
than distinct people and must be interpreted as a directional funnel signal.
3. Referral-bound accounts.
4. Referral-bound accounts that reach their first value event.
5. Rewarded referral bindings.
1. Referral binding created.
2. The same invitee reaches an AI value event after binding.
3. The same binding is rewarded before the report's `until`.
`REFERRAL_SHARED` distinct installations and invitation opens are independent
directional signals. Invitation opens combine accepted `INVITE_OPENED` events
with anonymous first-party page-view counters, so they are not people and must
never be placed in the ordered conversion funnel.
Pending and ineligible bindings are parallel status counts, not sequential
funnel steps.
@@ -212,8 +243,8 @@ funnel steps.
terminal success or failure event.
- Managed request failure rate: terminal non-settled `provider_requests` divided
by terminal managed requests.
- P50/P95 latency: client duration bucket distribution for all modes; exact
server duration percentiles may be added later.
- Client latency: successful and failed terminal events grouped by declared
duration bucket. Exact P50/P95 values are not inferred from buckets.
- Credit-blocked users: distinct installations reporting
`INSUFFICIENT_CREDITS` during the period.