first pass analytics redo

statistics-plan.md

@@ -0,0 +1,340 @@
+# Bibdle Statistics Dashboard — Plan
+
+**Status:** design agreed with the project owner · **Supersedes:** the public `/global` dashboard
+**Deliverable:** a private, once-daily, point-in-time **Growth Console** at `/admin/stats`, plus a slim public community-stats widget.
+
+> **Implementing this plan?** Use the agent-ready task list in [`statistics-implementation-checklist.md`](./statistics-implementation-checklist.md). It is ordered, dependency-aware, and includes acceptance criteria and tests per task.
+
+---
+
+## 1. Why we're replacing `/global`
+
+`/global` was a good start but it is now a wall of aggregates and second derivatives ("completions velocity", "user acceleration") with **no verdict, no feature data, and no content data**. It answers *"did a number move?"* but not *"why, and what should I change?"*
+
+Three structural problems:
+
+1. **No "so what."** Every chart is descriptive. Nothing tells you what to improve, change, or add.
+2. **No product surface.** There is no read on features (modes, hints, sharing), funnels (started → completed → shared), or content (which books/verses help or hurt retention).
+3. **Wrong delivery.** It queries the DB on every load, shows partial "today" data, and offers no time machine.
+
+**Goal of the new console:** a product review you can skim in 60 seconds, ordered around the questions that matter, that ends every section with a plain-language verdict and recommended action — and that can be scrolled back to any past day.
+
+### Live baseline (pulled from bibdle.com/global, as of 2026-09-15)
+
+| Metric | Value |
+|---|---|
+| All-time completions | 31,775 |
+| Unique players (all-time) | 6,589 |
+| Registered users | 98 |
+| DAU / WAU / MAU | 102 / 363 / 852 |
+| Net player growth (7d) | ~flat (−1: 175 new vs 176 churned) |
+| Overall return rate (ever played 2×) | 22.8% |
+| 7-day new-player return rate | 12.2% |
+| Survival after 2 plays → 3rd | 67.2% |
+| 7-day / 30-day retention | ~61% / ~51% |
+| Difficulty | mean 4.13 guesses, median 3, **33.7% first-try** |
+| Easiest / hardest books | Revelation 1.34 avg / 83% first-try · Micah 7.84 avg / 12% first-try |
+
+**The core story:** acquisition is replacing churn but not growing. ~77% of all-time players played exactly once; the ~23% who return are very sticky. The leak is the **first session** (and possibly verse mix/achievability), not the endgame.
+
+---
+
+## 2. Decisions locked with the owner
+
+| # | Decision | Choice |
+|---|---|---|
+| 1 | Audience | **Private founder dashboard**, replacing `/global` internally. Reuse `ADMIN_EMAIL` gate (as `/scheduled-verses` does). |
+| 2 | Public presence | Keep a **slim public community-stats widget** (no sensitive/business data). |
+| 3 | Freshness | **Once-daily snapshot, "as of yesterday."** Page load never queries the DB. |
+| 4 | History | **Point-in-time by default, with a "Mature this view" toggle.** |
+| 5 | Snapshot storage | **Store one immutable snapshot per day** (fidelity + speed + no-data fallbacks). |
+| 6 | Config | Segment thresholds configurable at **render time** via a settings menu (env defaults). Requires storing compact per-player state in snapshots. |
+| 7 | Data sources | **Existing SQLite only.** No Umami, no Rybbit, self-contained and durable. |
+| 8 | North star | Lead with **DAU + MAU**, with **Core players (4+/7d)** as the durability signal. |
+| 9 | Signals | **Both:** explicit targets for ~6 core metrics + anomaly detection everywhere else. |
+| 10 | Charts | **LayerChart** (Svelte 5 native; verified `layerchart@2.5.0` peers `svelte ^5`). |
+| 11 | Tone | **Plain language.** Short factual sentences. No jargon, no paragraphs next to charts. |
+| 12 | Mobile | Must be **mobile-friendly**. |
+| 13 | Digest | **Scheduled daily digest** of the snapshot (Discord admin webhook default; email optional). |
+| 14 | Instrumentation approved | **A, B, D, E, F** + `user.created_at` (see §7). **C deferred** (roadmap). |
+
+**Design principles (apply to every chart):**
+- Every visual has a **plain-English caption** and, where useful, a **one-line "what to do."**
+- Never show a number without a **comparison** (previous period, same weekday, or baseline).
+- **Weekday-align** daily comparisons so weekends don't masquerade as growth.
+- Prefer **shape** over derivatives: show "players per day" and a trend line, not "acceleration."
+
+---
+
+## 3. Information architecture
+
+```
+/admin/stats
+├─ Global chrome: date navigator · "Mature this view" toggle · Settings · as-of banner
+├─ 1. Pulse                (10-second read + verdict)
+├─ 2. Growth Loop          (new · active · returning · churned · funnel)
+├─ 3. Retention Lab        (cohorts, curves, time-to-return, segments)
+├─ 4. Verse Mix Lab        (coverage, freshness, retention by book & difficulty)
+├─ 5. Habit Lab            (streaks, calendar, timing)
+├─ 6. Signals & Actions    (rule-based verdicts + recommended actions)
+├─ 7. Feature Impact       (stub until C is instrumented)
+└─ 8. Notes & Glossary     (plain-language metric definitions)
+```
+
+The header is a **time machine**: a date picker plus prev/next arrows, always labeled e.g. *"Showing 2026-09-14 (snapshot built 2026-09-15)."* Days before a metric existed show **"—"**.
+
+---
+
+## 4. The labs (what you see, why, and the chart type)
+
+Chart types in **bold**. Every section ends in a one-sentence verdict from the Signals engine.
+
+### 4.1 Pulse
+
+| Element | Plain-English caption | Chart |
+|---|---|---|
+| Players today | "People who finished yesterday's verse." + same-weekday change | **KPI card + sparkline** |
+| Players this week / month | "Distinct players in the last 7 / 30 days." | **KPI card + delta** |
+| Core players | "Players who finished 4+ times this week — your habit-holders." | **KPI card + trend** |
+| Returning share | "Of yesterday's players, how many had played before." | **KPI + mini stacked bar** |
+| D1 return rate | "Of people who played for the first time, how many came back the next day." | **KPI + gauge vs target** |
+| Net player growth (7d) | "New and returning players minus those who stopped." | **KPI + waterfall delta** |
+| Verdict banner | e.g. "Stable. Biggest leak: 88% of new players never return." | **status pill** |
+
+*Decision it informs:* which lab to open; whether the week was a win.
+
+### 4.2 Growth Loop
+
+| Element | Plain-English caption | Chart |
+|---|---|---|
+| New players per day | "First-time players each day." Annotated with HN/Bluesky/Discord posts. | **Annotated area + marker pins** |
+| Acquisition vs churn | "New + resurrected − churned = net." | **Waterfall** |
+| Lifecycle funnel | "Of everyone who ever played: how many returned 2×, 5×, 10×, 25×, 50×." | **Funnel** |
+| Retention curves | D1 / D7 / D30 over time. | **Small-multiple line** |
+| Signup funnel | "Of first-time players, how many created an account." (needs `user.created_at` + starts) | **Funnel / line** |
+
+*Decision it informs:* is the problem acquisition or activation; did a marketing post actually move retention, not just traffic.
+
+### 4.3 Retention Lab
+
+| Element | Plain-English caption | Chart |
+|---|---|---|
+| Cohort retention triangle | "Each row is a starting week; each column is weeks later. Darker = more came back." | **Heatmap** |
+| Time-to-second-play | "How many days until a first-timer plays again." | **Histogram** |
+| Segment mix over time | New · Returning · Resurrected · At-risk · Churned. | **Stacked area** |
+| Retention by segment | "Do account-holders / core players stick longer?" | **Box plot or grouped bars** |
+| Overall return rate | "Share of all players who ever played more than once." | **KPI + target** |
+
+*Decision it informs:* whether onboarding/notification changes bend retention; which segment is healthiest.
+
+### 4.4 Verse Mix Lab
+
+This lab treats **verse mix and appealability** (not raw difficulty) as the retention lever. Hard *and* easy days can both drive churn — one by frustration, one by boredom.
+
+| Element | Plain-English caption | Chart |
+|---|---|---|
+| Mix balance over time | "Are we spreading across all 66 books / 9 sections / 2 testaments?" | **Stacked bars / treemap** |
+| Coverage heatmap | 66 books × times featured. | **Heatmap** |
+| Freshness | "How often the same book or window repeats." (repeat interval + entropy) | **Histogram + line** |
+| Retention by book | For each book, the next-day return rate of players who got it. | **Bubble scatter** (x=plays, y=D1 return, size=first-try%) |
+| Retention by difficulty | "Do easy or hard days make people leave?" | **Line/grouped bars by guess bucket** |
+| Achievability band | First-try % and solve-by-N against a target band. | **Histogram + step line** |
+| Regression alarms | Books/windows whose cohort churns above average. | **Table with inline bars** |
+| Before/after the curated mix system | Same metrics pre- vs post-launch. | **Annotated line** |
+
+*Decision it informs:* whether the new curated verse-mix system works; which books/windows to retire or feature more; submissions policy.
+
+### 4.5 Habit Lab
+
+| Element | Plain-English caption | Chart |
+|---|---|---|
+| Streak survival | "If you've played N days in a row, how likely are you to play day N+1." | **Survival line** |
+| Streak lengths | Distribution of current streaks. | **Ridgeline / histogram** |
+| Calendar heatmap | Daily completions, GitHub-style. | **Calendar heatmap** |
+| Weekday × hour | When people play. (Note: `completedAt` is server time, approximate.) | **2-D heatmap** |
+
+*Decision it informs:* whether to invest in streaks, reminders, and Discord/content timing.
+
+### 4.6 Signals & Actions
+Plain-language cards, each: **metric · current vs target · trend · interpretation · suggested action.**
+> Example: *"D1 return is 12% (target 20%). Most first-timers never see day two. → Test a 'come back tomorrow' prompt on the win screen."*
+> Example: *"Revelation averages 1.3 guesses (83% first-try). Likely self-revealing text. → Exclude or down-weight for the curated mix."*
+
+### 4.7 Feature Impact (stub)
+Since feature-usage events (C) are deferred, this ships as a **"needs data"** panel listing exactly what would be shown once C is added (Imposter entries, hint reveals, progress-page visits, submissions).
+
+---
+
+## 5. Plain-language metric glossary (shown in the page)
+
+Each metric has a friendly name, a one-sentence definition, and a "what to do."
+
+| Metric | Plain definition | What to do if it's off |
+|---|---|---|
+| Players today | Finished the puzzle yesterday | Investigate content or traffic |
+| WAU / MAU | Played at least once in 7 / 30 days | Reach vs habit |
+| Core players | 4+ finishes this week | If falling, retention problem |
+| Returning share | Played before vs first time | Low = acquisition-heavy, leaky |
+| D1 return | First-timers who came back next day | The #1 activation lever |
+| Return rate (any) | Played again within N days | Onboarding quality |
+| Overall return rate | Ever played more than once | Overall stickiness |
+| Net growth | New + resurrected − churned | Is the base expanding? |
+| Completion rate | Started → finished | Friction/confusion (needs starts) |
+| Signup rate | Players who made an account | Account value |
+| First-try rate | Solved with no wrong guesses | Too high = giveaway; too low = brutal |
+| Share rate | Shares ÷ finishes | Virality (needs D) |
+| Streak survival | Chance a streak continues | Streak feature ROI |
+
+---
+
+## 6. Data model & architecture
+
+### 6.1 New tables (all additive)
+
+**`stats_snapshot`** — one immutable row per as-of day.
+| Column | Type | Notes |
+|---|---|---|
+| `date` | text PK | as-of date `YYYY-MM-DD` |
+| `generated_at` | integer | build time |
+| `build_kind` | text | `scheduled` \| `lazy` \| `reconstructed` |
+| `config_version` | text | hash of thresholds used at build |
+| `payload` | text (JSON) | all aggregate series/lab data |
+| `player_state` | text (JSON, gzip) | compact per-player state as of that day, for render-time segmentation |
+| `config` | text (JSON) | copy of thresholds used |
+
+**`analytics_events`** — generic, extensible event log (unlocks B/D/E/F now, C later).
+| Column | Type | Notes |
+|---|---|---|
+| `id` | text PK | |
+| `name` | text | `verse_served`, `share`, `share_copy`, `first_visit`, `donate_click`, `discord_click`, … |
+| `anonymous_id` | text | pseudonymous |
+| `user_id` | text, nullable | |
+| `date` | text, nullable | game date when relevant |
+| `created_at` | integer | server timestamp |
+| `metadata` | text (JSON) | referrer/utm, snippet flag, channel… |
+
+Indexes: `(name, date)`, `(anonymous_id, date)`. **Unique** on `(name, anonymous_id, date)` for `verse_served` (dedupes reloads → "players who started").
+
+**`dashboard_settings`** — key/value (e.g. `core_plays_per_week`, `churn_days`, `resurrect_gap_days`, `d1_target`, `first_try_band`, `share_target`). Env vars provide defaults; the settings menu writes here.
+
+**`user.created_at`** — new timestamp column (see §7).
+
+### 6.2 Snapshot build
+- **Trigger:** daily cron hits a protected `POST /api/cron/build-stats` (bearer `CRON_SECRET`, same pattern as `/api/send-daily-verse`). It builds yesterday's snapshot and sends the digest.
+- **Lazy fallback:** `/admin/stats` load builds a missing snapshot on demand (guarded against races, marked `lazy`).
+- **Backfill:** an admin action/script can reconstruct a date range from raw data (marked `reconstructed`).
+- **Computation cutoff:** only data with `date <= asOfDate` (and `completedAt` before end of that day) — guarantees point-in-time.
+- **Player state (for render-time config):** per player active in the trailing ~180 days, a compact tuple `[anonId, firstDate, lastDate, totalPlays, plays7, plays30, currentStreak]`. Thresholds applied at render time, so changing them re-segments **all history instantly** while the state stays frozen as of that day.
+- **Retention/pruning:** keep all snapshots (small); optionally prune `player_state` older than a configurable horizon later.
+
+### 6.3 "Mature this view"
+A toggle (off by default). When on, the selected date's metrics are recomputed on demand using data *after* that date (so old cohorts have resolved D7/D30), clearly labeled **"Matured — uses data after this date."** This is the only mode allowed to query the DB directly.
+
+### 6.4 Auth & privacy
+- `/admin/stats` gated to `ADMIN_EMAIL` via existing `src/lib/server/admin.ts`.
+- Aggregate-only rendering; never display emails or raw IDs. Anonymous IDs are pseudonymous.
+- `/api/stats` name already exists (per-user stats) — the new snapshot endpoints will use distinct names (`/api/admin/stats/...`) to avoid collision.
+
+### 6.5 Public community widget
+Redesigned slim `/global` (or an embeddable component) showing only: all-time completions, players this week/month, yesterday's completions and average guesses, and a 30-day calendar heatmap. **No** retention, churn, signup, or revenue data. Labeled "as of yesterday."
+
+---
+
+## 7. Instrumentation plan (approved)
+
+| ID | What | Where it wires in | Cost |
+|---|---|---|---|
+| **A** | Store guess sequences for **all** players | `src/routes/api/submit-completion/+server.ts` already persists `guesses` when provided; confirm the client always sends them and remove any conditional nulling. Historical rows stay null (correct point-in-time). | Tiny |
+| **B** | Log "verse served" (game started) | Client generates/reads `bibdle-anonymous-id` in `src/routes/+page.ts` and sends it in the `POST /api/daily-verse` body; endpoint records a `verse_served` event, unique per `(anonymous_id, date)`. Gives start→completion funnel and bounce. | Small |
+| **D** | Share / virality events | In `src/lib/utils/share.ts` / `WinScreen.svelte`, record `share` and `share_copy` events (with `snippet` flag) via a tiny client helper. Enables share rate + viral proxy. | Small |
+| **E** | Referrer / UTM capture | On first visit (client), read `document.referrer` and `?ref=`/`?utm_source=`; store a `first_visit` event keyed to the anonymous ID. Enables acquisition source. | Small |
+| **F** | Monetization clicks | Record `donate_click` and `discord_click` events from the win-screen/footer links. Enables the patron funnel. | Tiny |
+| **G** | `user.created_at` | Add column + set on signup in `src/lib/server/auth.ts` (and OAuth paths). Existing users show "unknown." Enables signup-per-day, time-to-signup, and the visitor→signup funnel. | Small |
+
+**Deferred (roadmap):** **C — feature-usage event log** (Imposter/hints/progress/submissions). The `analytics_events` table makes this a trivial follow-on; only the emit points need adding.
+
+---
+
+## 8. Signals engine (targets + anomaly detection)
+
+**Explicit targets (defaults; editable in Settings):**
+
+| Metric | Target | Red |
+|---|---|---|
+| D1 return rate | ≥ 20% | < 12% |
+| 7-day return (any) | ≥ 25% | < 15% |
+| Completion rate (started→finished) | ≥ 70% | < 55% |
+| First-try rate (achievability band) | 15–35% | > 45% or < 8% |
+| Share rate | ≥ 25% | < 15% |
+| Core players (4+/7d) | non-declining WoW | down ≥ 2 weeks |
+
+*Completion-rate and share-rate targets are placeholders until 2 weeks of new data exist.*
+
+**Anomaly detection (everything else):** compare each metric to its own trailing 28-day median (weekday-aware), flag moves beyond a robust threshold (e.g. > 2× MAD). Output is a plain sentence, not a chart annotation.
+
+Each rule = `{ metric, comparator, target, severity, interpretation, recommended_action }`. Stored as code (not user-editable), with thresholds from settings.
+
+---
+
+## 9. Chart inventory & LayerChart mapping
+
+| Chart | Used in | LayerChart approach |
+|---|---|---|
+| Sparkline KPI | Pulse | minimal `<LineChart>` |
+| Annotated area | Growth Loop | `<AreaChart>` + marker overlay |
+| Waterfall | Growth Loop | floating `<Bar>` segments |
+| Funnel | Growth Loop, signups | stacked `<Rect>`/`<Bar>` |
+| Cohort heatmap | Retention | `<Chart>` + grid of `<Rect>` |
+| Calendar heatmap | Habit, public widget | `<Chart>` + weekday×week `<Rect>` grid |
+| 2-D heatmap | Habit | `<Chart>` + `<Rect>` grid |
+| Bubble scatter | Verse Mix | `<ScatterChart>` with radius scale |
+| Histogram | Retention, Verse Mix, Habit | `<BarChart>` from binned data |
+| Step / survival line | Verse Mix, Habit | `<LineChart>` with step interpolation |
+| Ridgeline | Habit | stacked `<AreaChart>`/density |
+| Box plot | Retention | custom `<Rect>`+`<Line>` |
+| Stacked area / bars | Retention, Verse Mix | `<AreaChart>`/`<BarChart>` stacked |
+| Gauge / status pill | Pulse | custom `<Chart>` arc + badge |
+| Table with inline bars | Verse Mix, Signals | existing `CollapsibleTable` |
+
+**Rendering model:** the page fetches the day's snapshot once (already computed), then renders charts client-side. No live DB reads in the normal path → matches the "once per day" requirement.
+
+---
+
+## 10. Delivery: daily digest
+
+- Runs right after the snapshot build (same cron endpoint), once per day.
+- **Default channel:** a dedicated Discord admin webhook (`DISCORD_ADMIN_WEBHOOK`), reusing the existing webhook pattern. Optionally email later (note: `nodemailer` is **not** currently a dependency; adding email = adding a dependency).
+- **Content (plain, ~6 lines):** as-of date · DAU/WAU/MAU · new vs returning · D1 · the single top signal · link to `/admin/stats`.
+
+---
+
+## 11. Phasing
+
+| Phase | Scope |
+|---|---|
+| **0 — Plumbing** | Add `stats_snapshot`, `analytics_events`, `dashboard_settings`, `user.created_at`; wire instrumentation A/B/D/E/F; client event helper. |
+| **1 — Core** | Snapshot builder + cron/lazy; date navigator; Pulse; Signals skeleton; Settings menu; plain-language glossary. |
+| **2 — Retention & Growth** | Retention Lab (cohort triangle, curves, time-to-return, segments) + Growth Loop (waterfall, funnel, annotated area, signup funnel). |
+| **3 — Content & Habit** | Verse Mix Lab (coverage, freshness, retention-by-book/difficulty, before/after) + Habit Lab (streaks, calendar, timing). |
+| **4 — Delivery** | Daily digest; slim public community widget; "Mature this view". |
+| **5 — Roadmap** | Feature-usage events (C) + Feature Impact panel; manual marketing-event annotations; CSV/JSON export; optional email digest. |
+
+---
+
+## 12. Open questions / to finalize during build
+
+1. **Digest channel:** Discord admin webhook (default) vs. adding nodemailer for email.
+2. **Target values** in §8 — confirm after ~2 weeks of instrumented data.
+3. **Snapshot retention:** keep `player_state` forever vs. prune beyond a horizon.
+4. **Public widget placement:** standalone `/global` vs. a section on `/about` vs. both.
+5. **Manual marketing annotations** (nice-to-have, not in approved list): a tiny table to pin HN/Bluesky/Discord posts on the Growth Loop timeline.
+6. **C (feature-usage events):** decide when to add — the table is ready, only emit points remain.
+
+---
+
+## 13. Success criteria for this dashboard
+
+- You can answer, in under 60 seconds and without jargon: *are we growing, who's returning, and what's the one thing to fix?*
+- The console tells you whether a **verse-mix** change helped retention, not just whether guesses changed.
+- You can scroll to any past day and see what was true then, with honest "—" where data didn't exist.
+- Adding a future metric (e.g. feature events) requires only emitting events — no rebuild of the dashboard architecture.