3 Commits
Author SHA1 Message Date
George Powell 4d5893ce14 chore(docs): note community stats and admin graph todos 2026-09-28 20:21:49 -04:00
George Powell da29ee7196 feat(stats): quantile-scaled, annotated calendar heatmap
Replace the linear count/max alpha with quantile buckets (GitHub-style)
so a single outlier day no longer flattens the rest of the graph, and
render best/median/worst day labels inside their cells.

- Add buildCalendarScale utility with empty-bucket handling.
- Make CalendarHeatmap size, gap, and annotations configurable; align
  weeks to Monday.
- Expand the community calendar to 90 days, add play totals/averages and
  all-time best day, and format large numbers with locale separators.
2026-09-28 20:21:48 -04:00
George Powell a562ee3cc9 fix(stats): finalize snapshots on the verse-day window
A verse date is a player's local calendar day, spanning D-1 10:00Z
(UTC+14) through D+1 12:00Z (UTC-12). The old UTC end-of-day cutoff both
dropped late-night finishes in negative timezones and could persist a
partial day.

- Scope snapshot loads by the verse date column; use completedBefore only
  to pin an in-progress point-in-time build.
- Gate persistence on isDayClosed (D+1 12:00Z); open days are computed
  live and never written.
- Build the cron snapshot for latestClosedDate instead of EST yesterday,
  and read the same date for the community page.
- Add all-time best single day to snapshot totals.
- Bump PAYLOAD_SCHEMA_VERSION to 4 and extend snapshot/window tests.
2026-09-28 20:21:47 -04:00
15 changed files with 456 additions and 78 deletions

No files matched your search

+1 -1
View File
@@ -133,7 +133,7 @@ Authenticated users who have solved today's puzzle can submit a verse for schedu
| --- | --- | | --- | --- |
| `POST /api/daily-verse` | Fetch (and cache) the verse for a given `YYYY-MM-DD` date. Also records a deduped `verse_served` event for the supplied `anonymousId` (started→finished funnel). | | `POST /api/daily-verse` | Fetch (and cache) the verse for a given `YYYY-MM-DD` date. Also records a deduped `verse_served` event for the supplied `anonymousId` (started→finished funnel). |
| `POST /api/events` | Fire-and-forget analytics sink. Allowlisted names: `verse_served`, `share`, `share_copy`, `first_visit`, `donate_click`, `discord_click`; `userId` resolved from the session. Always returns 204. | | `POST /api/events` | Fire-and-forget analytics sink. Allowlisted names: `verse_served`, `share`, `share_copy`, `first_visit`, `donate_click`, `discord_click`; `userId` resolved from the session. Always returns 204. |
| `POST /api/cron/build-stats` | Cron-only (bearer `CRON_SECRET`). Builds yesterday's snapshot (`America/New_York`) and posts the digest to `DISCORD_ADMIN_WEBHOOK`. | | `POST /api/cron/build-stats` | Cron-only (bearer `CRON_SECRET`). Builds the most recent snapshot whose global data window has closed (`D+1 12:00Z`) and posts the digest to `DISCORD_ADMIN_WEBHOOK`. |
| `POST /api/admin/rebuild-stats` | Admin-only. Rebuilds one snapshot: `{ date: 'YYYY-MM-DD' }`. | | `POST /api/admin/rebuild-stats` | Admin-only. Rebuilds one snapshot: `{ date: 'YYYY-MM-DD' }`. |
| `POST /api/submit-completion` | Submit a game result (`anonymousId`, `date`, `guessCount`, `guesses`); returns solve rank, guess rank, total solves, average guesses, ties, and percentile. Unique on `(anonymousId, date)`. | | `POST /api/submit-completion` | Submit a game result (`anonymousId`, `date`, `guessCount`, `guesses`); returns solve rank, guess rank, total solves, average guesses, ties, and percentile. Unique on `(anonymousId, date)`. |
| `GET /api/streak?anonymousId=X&localDate=Y` | Current streak: walks backwards from the client's local date through completed dates, counting consecutive days. Single-day streaks are reported as 0 (minimum displayed streak is 2). | | `GET /api/streak?anonymousId=X&localDate=Y` | Current streak: walks backwards from the client's local date through completed dates, counting consecutive days. Single-day streaks are reported as 0 (minimum displayed streak is 2). |
@@ -1,22 +1,39 @@
<script lang="ts"> <script lang="ts">
import { buildCalendarScale } from '$lib/utils/calendar-scale';
interface Props { interface Props {
data: { date: string; count: number }[]; data: { date: string; count: number }[];
color?: string; color?: string;
/** Pixel size of each day square. */
cellSize?: number;
/** Gap between squares in pixels. */
gap?: number;
/** Optional date → short label rendered inside the square. */
annotations?: Record<string, string>;
} }
let { data, color = '52,211,153' }: Props = $props(); let {
data,
color = '52,211,153',
cellSize = 12,
gap = 3,
annotations = {}
}: Props = $props();
const values = $derived(new Map(data.map((d) => [d.date, d.count]))); const values = $derived(new Map(data.map((d) => [d.date, d.count])));
const max = $derived(Math.max(1, ...data.map((d) => d.count))); const scale = $derived(buildCalendarScale(data.map((d) => d.count)));
const fontSize = $derived(Math.max(8, Math.round(cellSize * 0.42)));
const radius = $derived(Math.max(2, Math.round(cellSize * 0.16)));
// Build Sunday-aligned weeks ending on the last date. // Build Monday-aligned weeks ending on the last date (weekend at the bottom).
const weeks = $derived.by(() => { const weeks = $derived.by(() => {
if (data.length === 0) return [] as { date: string; inRange: boolean }[][]; if (data.length === 0) return [] as { date: string; inRange: boolean }[][];
const mondayOffset = (date: Date) => (date.getUTCDay() + 6) % 7;
const first = new Date(data[0].date + 'T00:00:00Z'); const first = new Date(data[0].date + 'T00:00:00Z');
const start = new Date(first); const start = new Date(first);
start.setUTCDate(start.getUTCDate() - start.getUTCDay()); start.setUTCDate(start.getUTCDate() - mondayOffset(start));
const last = new Date(data[data.length - 1].date + 'T00:00:00Z'); const last = new Date(data[data.length - 1].date + 'T00:00:00Z');
const end = new Date(last); const end = new Date(last);
end.setUTCDate(end.getUTCDate() + (6 - end.getUTCDay())); end.setUTCDate(end.getUTCDate() + (6 - mondayOffset(end)));
const range = new Set(data.map((d) => d.date)); const range = new Set(data.map((d) => d.date));
const out: { date: string; inRange: boolean }[][] = []; const out: { date: string; inRange: boolean }[][] = [];
let cursor = new Date(start); let cursor = new Date(start);
@@ -34,25 +51,55 @@
</script> </script>
<div class="overflow-x-auto"> <div class="overflow-x-auto">
<div class="flex gap-1"> <div class="flex" style={`gap:${Math.max(4, gap)}px`}>
<div class="flex flex-col justify-between py-0.5 text-[9px] text-gray-500"> <div
<span>Sun</span><span>Wed</span><span>Sat</span> class="flex flex-col justify-between py-0.5 text-gray-500"
style={`font-size:${fontSize}px`}
>
<span>Mon</span><span>Thu</span><span>Sun</span>
</div> </div>
<div class="flex gap-[3px]"> <div class="flex" style={`gap:${gap}px`}>
{#each weeks as week, wi (wi)} {#each weeks as week, wi (wi)}
<div class="flex flex-col gap-[3px]"> <div class="flex flex-col" style={`gap:${gap}px`}>
{#each week as cell (cell.date)} {#each week as cell (cell.date)}
{@const count = values.get(cell.date) ?? 0} {@const count = values.get(cell.date) ?? 0}
{@const alpha = scale.alphaFor(count) || 0.06}
{@const label = annotations[cell.date]}
<div <div
class="h-3 w-3 rounded-[2px]" class="cell-label flex items-center justify-center font-semibold tabular-nums"
style={cell.inRange data-pale={label && alpha < 0.4 ? 'true' : undefined}
? `background:rgba(${color},${(0.12 + 0.88 * (count / max)).toFixed(3)})` style={`width:${cellSize}px;height:${cellSize}px;border-radius:${radius}px;font-size:${fontSize}px;line-height:1;${
: 'background:transparent'} cell.inRange ? `background:rgba(${color},${alpha})` : 'background:transparent'
title={cell.inRange ? `${cell.date}: ${count} finishes` : cell.date} }`}
></div> title={cell.inRange
? `${cell.date}: ${count.toLocaleString('en-US')} finishes`
: cell.date}
>{label ?? ''}</div>
{/each} {/each}
</div> </div>
{/each} {/each}
</div> </div>
</div> </div>
</div> </div>
<style>
/*
* Annotated numbers need to read on both light and dark cell backgrounds.
* Intensity is the dominant factor: the bright/high buckets are light enough
* for dark ink in either theme, while the lowest bucket is a dark green on a
* dark page and needs light ink there.
*/
.cell-label {
color: rgb(17 24 39 / 0.95);
}
:global(html.dark) .cell-label[data-pale='true'] {
color: rgb(249 250 251 / 0.95);
}
@media (prefers-color-scheme: dark) {
:global(html:not(.light)) .cell-label[data-pale='true'] {
color: rgb(249 250 251 / 0.95);
}
}
</style>
+21
View File
@@ -32,6 +32,27 @@ export function todayUtc(): string {
return new Date().toISOString().slice(0, 10); return new Date().toISOString().slice(0, 10);
} }
// A verse `date` is the player's *local* calendar date, not a fixed UTC window.
// Globally the same verse date is played from D-1 10:00Z (UTC+14) through
// D+1 12:00Z (UTC-12), so a day's data is only final after D+1 12:00Z.
const HOUR_MS = 60 * 60 * 1000;
const VERSE_WINDOW_MS = 36 * HOUR_MS; // D+1 12:00Z, relative to D 00:00Z
/** Instant after which no further plays for verse `dateStr` are expected. */
export function endOfVerseWindow(dateStr: string): Date {
return new Date(new Date(dateStr + 'T00:00:00Z').getTime() + VERSE_WINDOW_MS);
}
/** True once the global data window for `dateStr` has closed. */
export function isDayClosed(dateStr: string, now: Date = new Date()): boolean {
return now.getTime() >= endOfVerseWindow(dateStr).getTime();
}
/** Most recent verse date whose global data window has closed. */
export function latestClosedDate(now: Date = new Date()): string {
return new Date(now.getTime() - VERSE_WINDOW_MS).toISOString().slice(0, 10);
}
export function isValidDate(dateStr: string): boolean { export function isValidDate(dateStr: string): boolean {
return /^\d{4}-\d{2}-\d{2}$/.test(dateStr) && !Number.isNaN(new Date(dateStr + 'T00:00:00Z').getTime()); return /^\d{4}-\d{2}-\d{2}$/.test(dateStr) && !Number.isNaN(new Date(dateStr + 'T00:00:00Z').getTime());
} }
+89 -33
View File
@@ -45,7 +45,7 @@ import {
timeToSecondPlayHistogram, timeToSecondPlayHistogram,
weekdayHourHeatmap weekdayHourHeatmap
} from './metrics'; } from './metrics';
import { addDays, dateRange, dayDiff, startOfWeek, todayUtc } from './dates'; import { addDays, dateRange, dayDiff, endOfVerseWindow, isDayClosed, startOfWeek, todayUtc } from './dates';
export type BuildKind = 'scheduled' | 'lazy' | 'reconstructed'; export type BuildKind = 'scheduled' | 'lazy' | 'reconstructed';
@@ -53,16 +53,12 @@ export type BuildKind = 'scheduled' | 'lazy' | 'reconstructed';
* Bump whenever the snapshot payload shape changes. Older snapshots are * Bump whenever the snapshot payload shape changes. Older snapshots are
* rebuilt lazily so the UI never reads a stale shape. * rebuilt lazily so the UI never reads a stale shape.
*/ */
export const PAYLOAD_SCHEMA_VERSION = 2; export const PAYLOAD_SCHEMA_VERSION = 4;
const RETENTION_WINDOW_DAYS = 180; const RETENTION_WINDOW_DAYS = 180;
const SERIES_DAYS = 90; const SERIES_DAYS = 90;
const RECENT_KPI_DAYS = 30; const RECENT_KPI_DAYS = 30;
function endOfDay(dateStr: string): Date {
return new Date(dateStr + 'T23:59:59.999Z');
}
function round1(n: number): number { function round1(n: number): number {
return Math.round(n * 10) / 10; return Math.round(n * 10) / 10;
} }
@@ -76,12 +72,28 @@ export interface LoadedData {
signupDates: string[]; signupDates: string[];
} }
/** Load all rows needed to build a snapshot as of `cutoff` (inclusive). */ export interface LoadStatsOptions {
/**
* Point-in-time cutoff for time-stamped rows. When omitted, the full data
* window is loaded and rows are grouped purely by their verse `date`.
*/
completedBefore?: Date;
}
/**
* Load all rows needed to build a snapshot as of `cutoff` (inclusive).
*
* Rows are scoped by the verse `date` column, not by a UTC 24-hour window:
* a verse date is a player's *local* day, so a single date legitimately spans
* D-1 10:00Z through D+1 12:00Z. `completedBefore` is only for point-in-time
* builds of a day whose window is still open.
*/
export async function loadStatsData( export async function loadStatsData(
cutoff: string, cutoff: string,
database: StatsDb = defaultDb database: StatsDb = defaultDb,
opts: LoadStatsOptions = {}
): Promise<LoadedData> { ): Promise<LoadedData> {
const cutoffDate = endOfDay(cutoff); const { completedBefore } = opts;
const completionRows = await database const completionRows = await database
.select({ .select({
@@ -92,13 +104,23 @@ export async function loadStatsData(
completedAt: dailyCompletions.completedAt completedAt: dailyCompletions.completedAt
}) })
.from(dailyCompletions) .from(dailyCompletions)
.where(and(lte(dailyCompletions.date, cutoff), lte(dailyCompletions.completedAt, cutoffDate))); .where(
completedBefore
? and(
lte(dailyCompletions.date, cutoff),
lte(dailyCompletions.completedAt, completedBefore)
)
: lte(dailyCompletions.date, cutoff)
);
const verseRows = await database const verseRows = await database
.select({ date: dailyVerses.date, bookId: dailyVerses.bookId }) .select({ date: dailyVerses.date, bookId: dailyVerses.bookId })
.from(dailyVerses) .from(dailyVerses)
.where(lte(dailyVerses.date, cutoff)); .where(lte(dailyVerses.date, cutoff));
// Events (and signups) are time-stamped rather than date-scoped. For a final
// build the window end is exact; an open-day build pins to `completedBefore`.
const rowBefore = completedBefore ?? endOfVerseWindow(cutoff);
const eventRows = await database const eventRows = await database
.select({ .select({
name: analyticsEvents.name, name: analyticsEvents.name,
@@ -106,18 +128,19 @@ export async function loadStatsData(
createdAt: analyticsEvents.createdAt createdAt: analyticsEvents.createdAt
}) })
.from(analyticsEvents) .from(analyticsEvents)
.where(lte(analyticsEvents.createdAt, cutoffDate)); .where(lte(analyticsEvents.createdAt, rowBefore));
const startedByDate = new Map<string, number>(); const startedByDate = new Map<string, number>();
let shareEvents = 0; let shareEvents = 0;
let firstEventDate: string | null = null; let firstEventDate: string | null = null;
for (const e of eventRows) { for (const e of eventRows) {
const eventDate = e.date ?? new Date(e.createdAt).toISOString().slice(0, 10); const eventDate = e.date ?? new Date(e.createdAt).toISOString().slice(0, 10);
if (eventDate > cutoff) continue;
if (!firstEventDate || eventDate < firstEventDate) firstEventDate = eventDate; if (!firstEventDate || eventDate < firstEventDate) firstEventDate = eventDate;
if (e.name === 'verse_served' && e.date && e.date <= cutoff) { if (e.name === 'verse_served' && e.date && e.date <= cutoff) {
startedByDate.set(e.date, (startedByDate.get(e.date) ?? 0) + 1); startedByDate.set(e.date, (startedByDate.get(e.date) ?? 0) + 1);
} }
if ((e.name === 'share' || e.name === 'share_copy') && eventDate <= cutoff) { if (e.name === 'share' || e.name === 'share_copy') {
shareEvents++; shareEvents++;
} }
} }
@@ -125,7 +148,7 @@ export async function loadStatsData(
const userRows = await database const userRows = await database
.select({ id: user.id, createdAt: user.createdAt }) .select({ id: user.id, createdAt: user.createdAt })
.from(user) .from(user)
.where(lte(user.createdAt, cutoffDate)); .where(lte(user.createdAt, rowBefore));
return { return {
completions: completionRows as CompletionRow[], completions: completionRows as CompletionRow[],
@@ -521,11 +544,25 @@ export function computePayload(
const verseMix = buildVerseMix(pointCompletions, data.verses, asOfDate); const verseMix = buildVerseMix(pointCompletions, data.verses, asOfDate);
const habit = buildHabit(pointCompletions, players, asOfDate); const habit = buildHabit(pointCompletions, players, asOfDate);
const yesterday = pointCompletions.filter((c) => c.date === asOfDate); const yesterday = pointCompletions.filter((c) => c.date === asOfDate);
// All-time best single day (earliest date wins ties).
const completionsByDate = new Map<string, number>();
for (const c of pointCompletions) {
completionsByDate.set(c.date, (completionsByDate.get(c.date) ?? 0) + 1);
}
let bestDay: { date: string; count: number } | null = null;
for (const [date, count] of completionsByDate) {
if (!bestDay || count > bestDay.count || (count === bestDay.count && date < bestDay.date)) {
bestDay = { date, count };
}
}
const totals = { const totals = {
completions: pointCompletions.length, completions: pointCompletions.length,
players: firstPlayDates(pointCompletions).size, players: firstPlayDates(pointCompletions).size,
yesterdayCompletions: yesterday.length, yesterdayCompletions: yesterday.length,
yesterdayAvgGuesses: averageGuesses(yesterday) yesterdayAvgGuesses: averageGuesses(yesterday),
bestDay
}; };
const payload: SnapshotPayload = { const payload: SnapshotPayload = {
@@ -589,8 +626,12 @@ export interface BuildResult {
} }
/** /**
* Build and persist a snapshot. `replace=false` never overwrites an existing * Build a snapshot. `replace=false` never overwrites an existing (immutable)
* (immutable) snapshot; the caller re-reads instead. * snapshot; the caller re-reads instead.
*
* A snapshot is only persisted once its global data window has closed: an
* in-progress day is computed as a point-in-time view and never written, so
* partial early-morning data can't become the permanent record.
*/ */
export async function buildSnapshot( export async function buildSnapshot(
asOfDate: string, asOfDate: string,
@@ -600,7 +641,10 @@ export async function buildSnapshot(
const settings = await getSettings(database); const settings = await getSettings(database);
const version = hashSettings(settings); const version = hashSettings(settings);
const cutoff = opts.matureThrough ?? asOfDate; const cutoff = opts.matureThrough ?? asOfDate;
const data = await loadStatsData(cutoff, database); const closed = isDayClosed(asOfDate);
const data = await loadStatsData(cutoff, database, {
completedBefore: closed ? undefined : new Date()
});
const payload = computePayload(asOfDate, settings, data, { const payload = computePayload(asOfDate, settings, data, {
matureThrough: opts.matureThrough matureThrough: opts.matureThrough
}); });
@@ -620,7 +664,7 @@ export async function buildSnapshot(
config: JSON.stringify(settings) config: JSON.stringify(settings)
}; };
if (opts.persist !== false) { if (opts.persist !== false && closed) {
if (opts.replace) { if (opts.replace) {
await database await database
.insert(statsSnapshot) .insert(statsSnapshot)
@@ -647,28 +691,40 @@ export async function buildSnapshot(
} }
/** /**
* Return the snapshot for `date`, building it lazily if missing. Race-safe via * Return the snapshot for `date`, building it lazily if missing.
* `INSERT OR IGNORE` followed by a re-read. *
* Closed days are persisted and reused (race-safe via `INSERT OR IGNORE`),
* while an in-progress day is always computed on demand and never written.
*/ */
export async function ensureSnapshot( export async function ensureSnapshot(
date: string, date: string,
opts: { matureThrough?: string } = {}, opts: { matureThrough?: string } = {},
database: StatsDb = defaultDb database: StatsDb = defaultDb
): Promise<StoredSnapshot> { ): Promise<StoredSnapshot> {
const existing = await getSnapshot(date, database); if (isDayClosed(date)) {
if (existing && existing.payload.meta?.schemaVersion === PAYLOAD_SCHEMA_VERSION) { const existing = await getSnapshot(date, database);
return existing; if (existing && existing.payload.meta?.schemaVersion === PAYLOAD_SCHEMA_VERSION) {
return existing;
}
// Missing or built with an older payload shape: (re)build the final view.
await buildSnapshot(
date,
{ kind: 'lazy', matureThrough: opts.matureThrough, replace: !!existing },
database
);
const stored = await getSnapshot(date, database);
if (!stored) throw new Error(`Failed to build snapshot for ${date}`);
return stored;
} }
// Missing or built with an older payload shape: (re)build deterministically
// for this point in time. The point-in-time cutoff keeps it faithful. // Open day: compute a live point-in-time view without persisting it.
await buildSnapshot( return (
date, await buildSnapshot(
{ kind: 'lazy', matureThrough: opts.matureThrough, replace: !!existing }, date,
database { kind: 'lazy', matureThrough: opts.matureThrough, persist: false },
); database
const stored = await getSnapshot(date, database); )
if (!stored) throw new Error(`Failed to build snapshot for ${date}`); ).snapshot;
return stored;
} }
/** True when a stored snapshot was built with different thresholds or shape. */ /** True when a stored snapshot was built with different thresholds or shape. */
+2
View File
@@ -222,6 +222,8 @@ export interface SnapshotPayload {
players: number; players: number;
yesterdayCompletions: number; yesterdayCompletions: number;
yesterdayAvgGuesses: number | null; yesterdayAvgGuesses: number | null;
/** Highest single-day finish count across all time (earliest date wins ties). */
bestDay: { date: string; count: number } | null;
}; };
signals: Signal[]; signals: Signal[];
/** Metadata about data availability, so the UI can show honest em-dashes. */ /** Metadata about data availability, so the UI can show honest em-dashes. */
+96
View File
@@ -0,0 +1,96 @@
/**
* Quantile-bucketed color scale for calendar/heatmap charts.
*
* A naive `count / max` alpha collapses most cells into the same pale shade when
* a single day is an outlier. Bucketing by quantiles of the non-zero values
* (like GitHub's contribution graph) keeps typical variation visible and never
* changes when a new all-time-high day appears.
*/
export interface CalendarBucket {
/** Lowest count in this bucket. */
min: number;
/** Highest count in this bucket. */
max: number;
/** Alpha (0..1) applied to the base color. */
alpha: number;
/** How many non-zero days fall in this bucket. */
days: number;
}
export interface CalendarScale {
min: number;
max: number;
/** Non-empty intensity buckets, low → high. Zero is rendered separately. */
buckets: CalendarBucket[];
/** Alpha for a raw count (0 for a zero/empty day). */
alphaFor: (count: number) => number;
}
/** Linear-interpolated quantile over an already-sorted ascending array. */
export function quantile(sorted: number[], q: number): number {
if (sorted.length === 0) return 0;
if (sorted.length === 1) return sorted[0];
const pos = (sorted.length - 1) * q;
const base = Math.floor(pos);
const rest = pos - base;
const next = sorted[base + 1];
return next === undefined ? sorted[base] : sorted[base] + rest * (next - sorted[base]);
}
/**
* Build a quantile scale from a list of daily counts.
*
* The non-zero counts are split into `steps` intensity buckets at evenly spaced
* quantiles; duplicate-heavy data is tolerated because empty buckets are dropped.
*/
export function buildCalendarScale(counts: number[], steps = 4): CalendarScale {
const nonzero = counts.filter((c) => c > 0).sort((a, b) => a - b);
const min = counts.length ? Math.min(...counts) : 0;
const max = counts.length ? Math.max(...counts) : 0;
if (nonzero.length === 0) {
return { min, max, buckets: [], alphaFor: () => 0 };
}
const alphas =
steps <= 1
? [0.9]
: Array.from({ length: steps }, (_, i) => 0.28 + (0.72 * i) / (steps - 1));
const thresholds: number[] = [];
for (let i = 1; i < steps; i++) thresholds.push(quantile(nonzero, i / steps));
const bucketIndex = (count: number): number => {
if (count <= 0) return -1;
let index = 0;
for (const threshold of thresholds) {
if (count > threshold) index++;
}
return index;
};
const buckets: CalendarBucket[] = alphas.map((alpha) => ({
alpha,
min: Number.POSITIVE_INFINITY,
max: Number.NEGATIVE_INFINITY,
days: 0
}));
for (const count of nonzero) {
const bucket = buckets[bucketIndex(count)];
bucket.min = Math.min(bucket.min, count);
bucket.max = Math.max(bucket.max, count);
bucket.days++;
}
return {
min,
max,
buckets: buckets.filter((bucket) => bucket.days > 0),
alphaFor: (count: number) => {
const index = bucketIndex(count);
return index < 0 ? 0 : alphas[index];
}
};
}
+1 -1
View File
@@ -1,7 +1,7 @@
/** Small formatting helpers shared by the stats dashboard components. */ /** Small formatting helpers shared by the stats dashboard components. */
export function numOrDash(n: number | null | undefined): string { export function numOrDash(n: number | null | undefined): string {
return n === null || n === undefined ? '—' : String(n); return n === null || n === undefined ? '—' : n.toLocaleString('en-US');
} }
export function pctOrDash(n: number | null | undefined): string { export function pctOrDash(n: number | null | undefined): string {
+6 -7
View File
@@ -1,13 +1,13 @@
import { json } from '@sveltejs/kit'; import { json } from '@sveltejs/kit';
import type { RequestHandler } from './$types'; import type { RequestHandler } from './$types';
import { addDays } from '$lib/server/stats/dates'; import { latestClosedDate } from '$lib/server/stats/dates';
import { buildSnapshot } from '$lib/server/stats/snapshot'; import { buildSnapshot } from '$lib/server/stats/snapshot';
import { sendDigest } from '$lib/server/stats/digest'; import { sendDigest } from '$lib/server/stats/digest';
/** /**
* Daily snapshot build trigger. Protected by `Authorization: Bearer $CRON_SECRET` * Daily snapshot build trigger. Protected by `Authorization: Bearer $CRON_SECRET`
* (same pattern as `/api/send-daily-verse`). Builds yesterday in `America/New_York` * (same pattern as `/api/send-daily-verse`). Builds the most recent date whose
* and then sends the digest. * global data window has closed, then sends the digest.
*/ */
export const POST: RequestHandler = async ({ request }) => { export const POST: RequestHandler = async ({ request }) => {
const cronSecret = Bun.env.CRON_SECRET; const cronSecret = Bun.env.CRON_SECRET;
@@ -16,10 +16,9 @@ export const POST: RequestHandler = async ({ request }) => {
return json({ error: 'Unauthorized' }, { status: 401 }); return json({ error: 'Unauthorized' }, { status: 401 });
} }
const todayEst = new Date().toLocaleDateString('en-CA', { timeZone: 'America/New_York' }); const date = latestClosedDate();
const yesterday = addDays(todayEst, -1);
const { snapshot } = await buildSnapshot(yesterday, { const { snapshot } = await buildSnapshot(date, {
kind: 'scheduled', kind: 'scheduled',
replace: true replace: true
}); });
@@ -27,5 +26,5 @@ export const POST: RequestHandler = async ({ request }) => {
// Digest failure must never fail the build. // Digest failure must never fail the build.
await sendDigest(snapshot.payload); await sendDigest(snapshot.payload);
return json({ ok: true, date: yesterday, buildKind: 'scheduled' }); return json({ ok: true, date, buildKind: 'scheduled' });
}; };
+6 -7
View File
@@ -1,15 +1,13 @@
import type { PageServerLoad } from './$types'; import type { PageServerLoad } from './$types';
import { ensureSnapshot } from '$lib/server/stats/snapshot'; import { ensureSnapshot } from '$lib/server/stats/snapshot';
import { addDays } from '$lib/server/stats/dates'; import { latestClosedDate } from '$lib/server/stats/dates';
/** /**
* Slim public community widget. Reads the latest daily snapshot and exposes * Slim public community widget. Reads the latest completed daily snapshot and
* only safe aggregates (no retention, churn, signup or revenue data). * exposes only safe aggregates (no retention, churn, signup or revenue data).
*/ */
export const load: PageServerLoad = async () => { export const load: PageServerLoad = async () => {
const todayEst = new Date().toLocaleDateString('en-CA', { timeZone: 'America/New_York' }); const snapshot = await ensureSnapshot(latestClosedDate());
const yesterday = addDays(todayEst, -1);
const snapshot = await ensureSnapshot(yesterday);
const payload = snapshot.payload; const payload = snapshot.payload;
return { return {
@@ -20,6 +18,7 @@ export const load: PageServerLoad = async () => {
monthlyPlayers: payload.activity.mau, monthlyPlayers: payload.activity.mau,
yesterdayCompletions: payload.totals.yesterdayCompletions, yesterdayCompletions: payload.totals.yesterdayCompletions,
yesterdayAvgGuesses: payload.totals.yesterdayAvgGuesses, yesterdayAvgGuesses: payload.totals.yesterdayAvgGuesses,
calendar: payload.habit.calendar.slice(-30) bestDayAllTime: payload.totals.bestDay,
calendar: payload.habit.calendar.slice(-90)
}; };
}; };
+87 -8
View File
@@ -5,11 +5,50 @@
import { formatDateLabel, numOrDash } from '$lib/utils/stats-format'; import { formatDateLabel, numOrDash } from '$lib/utils/stats-format';
let { data }: { data: PageData } = $props(); let { data }: { data: PageData } = $props();
const calendar = $derived(data.calendar);
const active = $derived(calendar.filter((d) => d.count > 0));
const total = $derived(calendar.reduce((sum, d) => sum + d.count, 0));
const avgPerDay = $derived(calendar.length ? total / calendar.length : 0);
const bestDay = $derived(
active.reduce<{ date: string; count: number } | null>(
(best, d) => (best === null || d.count > best.count ? d : best),
null
)
);
const worstDay = $derived(
active.reduce<{ date: string; count: number } | null>(
(worst, d) => (worst === null || d.count < worst.count ? d : worst),
null
)
);
const medianDay = $derived.by(() => {
if (active.length === 0) return null;
const sorted = [...active].sort((a, b) => a.count - b.count);
return sorted[Math.floor((sorted.length - 1) / 2)];
});
/** Compact enough to sit inside a day square. */
function compactCount(n: number): string {
if (n >= 10000) return `${Math.round(n / 1000)}k`;
if (n >= 1000) return `${(n / 1000).toFixed(1)}k`;
return String(n);
}
const annotations = $derived.by(() => {
const out: Record<string, string> = {};
for (const day of [bestDay, medianDay, worstDay]) {
if (day) out[day.date] = compactCount(day.count);
}
return out;
});
</script> </script>
<svelte:head> <svelte:head>
<title>Community Stats | Bibdle</title> <title>Community Stats | Bibdle</title>
<meta name="description" content="Bibdle community statistics, as of yesterday." /> <meta name="description" content="Bibdle community statistics, as of the latest completed day." />
</svelte:head> </svelte:head>
<div class="mx-auto max-w-3xl px-4 py-8"> <div class="mx-auto max-w-3xl px-4 py-8">
@@ -20,7 +59,7 @@
<div class="grid grid-cols-2 gap-3 md:grid-cols-3"> <div class="grid grid-cols-2 gap-3 md:grid-cols-3">
<Container class="p-4 w-full items-start text-left"> <Container class="p-4 w-full items-start text-left">
<p class="text-[11px] font-semibold uppercase tracking-wider text-gray-600 dark:text-gray-400">All-time finishes</p> <p class="text-[11px] font-semibold uppercase tracking-wider text-gray-600 dark:text-gray-400">All-time plays</p>
<p class="text-2xl font-bold text-gray-900 dark:text-gray-100">{numOrDash(data.completions)}</p> <p class="text-2xl font-bold text-gray-900 dark:text-gray-100">{numOrDash(data.completions)}</p>
</Container> </Container>
<Container class="p-4 w-full items-start text-left"> <Container class="p-4 w-full items-start text-left">
@@ -36,21 +75,61 @@
<p class="text-2xl font-bold text-gray-900 dark:text-gray-100">{numOrDash(data.monthlyPlayers)}</p> <p class="text-2xl font-bold text-gray-900 dark:text-gray-100">{numOrDash(data.monthlyPlayers)}</p>
</Container> </Container>
<Container class="p-4 w-full items-start text-left"> <Container class="p-4 w-full items-start text-left">
<p class="text-[11px] font-semibold uppercase tracking-wider text-gray-600 dark:text-gray-400">Yesterday's finishes</p> <p class="text-[11px] font-semibold uppercase tracking-wider text-gray-600 dark:text-gray-400">Latest day's plays</p>
<p class="text-2xl font-bold text-gray-900 dark:text-gray-100">{numOrDash(data.yesterdayCompletions)}</p> <p class="text-2xl font-bold text-gray-900 dark:text-gray-100">{numOrDash(data.yesterdayCompletions)}</p>
</Container> </Container>
<Container class="p-4 w-full items-start text-left"> <Container class="p-4 w-full items-start text-left">
<p class="text-[11px] font-semibold uppercase tracking-wider text-gray-600 dark:text-gray-400">Avg guesses yesterday</p> <p class="text-[11px] font-semibold uppercase tracking-wider text-gray-600 dark:text-gray-400">Avg guesses that day</p>
<p class="text-2xl font-bold text-gray-900 dark:text-gray-100">{numOrDash(data.yesterdayAvgGuesses)}</p> <p class="text-2xl font-bold text-gray-900 dark:text-gray-100">{numOrDash(data.yesterdayAvgGuesses)}</p>
</Container> </Container>
</div> </div>
<Container class="mt-3 p-4 w-full items-stretch text-left gap-2"> <Container class="mt-3 p-4 w-full items-stretch text-left gap-3">
<p class="font-semibold text-gray-900 dark:text-gray-100">Daily finishes (last 30 days)</p> <p class="font-semibold text-gray-900 dark:text-gray-100">Daily plays (last {calendar.length} days)</p>
<CalendarHeatmap data={data.calendar} /> <div class="flex flex-col gap-4 sm:flex-row sm:items-start">
<div class="flex flex-col gap-1">
<CalendarHeatmap data={calendar} cellSize={22} annotations={annotations} />
<p class="text-[10px] text-gray-500 dark:text-gray-400">
Numbers mark the best, median, and least popular days.
</p>
</div>
<div class="flex min-w-0 flex-1 flex-col gap-4">
<dl class="grid grid-cols-2 gap-x-4 gap-y-3 text-sm">
<div>
<dt class="text-[11px] uppercase tracking-wider text-gray-500 dark:text-gray-400">{calendar.length}-day plays</dt>
<dd class="font-semibold tabular-nums text-gray-900 dark:text-gray-100">{numOrDash(total)}</dd>
</div>
<div>
<dt class="text-[11px] uppercase tracking-wider text-gray-500 dark:text-gray-400">Daily average</dt>
<dd class="font-semibold tabular-nums text-gray-900 dark:text-gray-100">
{avgPerDay.toLocaleString('en-US', { maximumFractionDigits: 1 })}
</dd>
</div>
{#if bestDay}
<div>
<dt class="text-[11px] uppercase tracking-wider text-gray-500 dark:text-gray-400">Best day ({calendar.length}d)</dt>
<dd class="font-semibold tabular-nums text-gray-900 dark:text-gray-100">
{numOrDash(bestDay.count)}
<span class="font-normal text-gray-500 dark:text-gray-400">on {formatDateLabel(bestDay.date)}</span>
</dd>
</div>
{/if}
{#if data.bestDayAllTime}
<div>
<dt class="text-[11px] uppercase tracking-wider text-gray-500 dark:text-gray-400">Best day · all-time</dt>
<dd class="font-semibold tabular-nums text-gray-900 dark:text-gray-100">
{numOrDash(data.bestDayAllTime.count)}
<span class="font-normal text-gray-500 dark:text-gray-400">on {formatDateLabel(data.bestDayAllTime.date)}</span>
</dd>
</div>
{/if}
</dl>
</div>
</div>
</Container> </Container>
<p class="mt-4 text-center text-xs text-gray-500 dark:text-gray-500"> <p class="mt-4 text-center text-xs text-gray-500 dark:text-gray-500">
Updated once a day from completed games. <a class="underline hover:text-gray-300" href="/">Play today's verse →</a> Updated once a day from completed games.<br /> <a class="underline hover:text-gray-300" href="/">Play today's verse →</a>
</p> </p>
</div> </div>
+2 -2
View File
@@ -220,10 +220,10 @@ Indexes: `(name, date)`, `(anonymous_id, date)`. **Unique** on `(name, anonymous
**`user.created_at`** — new timestamp column (see §7). **`user.created_at`** — new timestamp column (see §7).
### 6.2 Snapshot build ### 6.2 Snapshot build
- **Trigger:** daily cron hits a protected `POST /api/cron/build-stats` (bearer `CRON_SECRET`, same pattern as `/api/send-daily-verse`). It builds yesterday's snapshot and sends the digest. - **Trigger:** daily cron hits a protected `POST /api/cron/build-stats` (bearer `CRON_SECRET`, same pattern as `/api/send-daily-verse`). It builds the most recent date whose global data window has closed and sends the digest.
- **Lazy fallback:** `/admin/stats` load builds a missing snapshot on demand (guarded against races, marked `lazy`). - **Lazy fallback:** `/admin/stats` load builds a missing snapshot on demand (guarded against races, marked `lazy`).
- **Backfill:** an admin action/script can reconstruct a date range from raw data (marked `reconstructed`). - **Backfill:** an admin action/script can reconstruct a date range from raw data (marked `reconstructed`).
- **Computation cutoff:** only data with `date <= asOfDate` (and `completedAt` before end of that day) — guarantees point-in-time. - **Computation cutoff:** data is grouped by the verse `date` (a player's local day), not a UTC 24-hour window. A date is final after `D+1 12:00Z` (UTC-12 has finished the day); until then a build is a non-persisted point-in-time view. Snapshots are only persisted once the window has closed.
- **Player state (for render-time config):** per player active in the trailing ~180 days, a compact tuple `[anonId, firstDate, lastDate, totalPlays, plays7, plays30, currentStreak]`. Thresholds applied at render time, so changing them re-segments **all history instantly** while the state stays frozen as of that day. - **Player state (for render-time config):** per player active in the trailing ~180 days, a compact tuple `[anonId, firstDate, lastDate, totalPlays, plays7, plays30, currentStreak]`. Thresholds applied at render time, so changing them re-segments **all history instantly** while the state stays frozen as of that day.
- **Retention/pruning:** keep all snapshots (small); optionally prune `player_state` older than a configurable horizon later. - **Retention/pruning:** keep all snapshots (small); optionally prune `player_state` older than a configurable horizon later.
+1 -1
View File
@@ -40,7 +40,7 @@ const payload: SnapshotPayload = {
}, },
habit: { streakSurvival: [], streakLengths: [], calendar: [], weekdayHour: [] }, habit: { streakSurvival: [], streakLengths: [], calendar: [], weekdayHour: [] },
growth: { lifecycleFunnel: [], signups: [] }, growth: { lifecycleFunnel: [], signups: [] },
totals: { completions: 31775, players: 6589, yesterdayCompletions: 102, yesterdayAvgGuesses: 4.1 }, totals: { completions: 31775, players: 6589, yesterdayCompletions: 102, yesterdayAvgGuesses: 4.1, bestDay: null },
signals: [ signals: [
{ {
id: 'd1_return', id: 'd1_return',
+1 -1
View File
@@ -41,7 +41,7 @@ function makePayload(overrides: {
segmentMix: [] segmentMix: []
}, },
growth: { lifecycleFunnel: [], signups: [] }, growth: { lifecycleFunnel: [], signups: [] },
totals: { completions: 0, players: 0, yesterdayCompletions: 0, yesterdayAvgGuesses: null }, totals: { completions: 0, players: 0, yesterdayCompletions: 0, yesterdayAvgGuesses: null, bestDay: null },
verseMix: { verseMix: {
coverage: [], coverage: [],
testamentMix: [], testamentMix: [],
+69
View File
@@ -11,9 +11,11 @@ import {
computePayload, computePayload,
computePlayerState, computePlayerState,
ensureSnapshot, ensureSnapshot,
loadStatsData,
PAYLOAD_SCHEMA_VERSION, PAYLOAD_SCHEMA_VERSION,
type LoadedData type LoadedData
} from '$lib/server/stats/snapshot'; } from '$lib/server/stats/snapshot';
import { endOfVerseWindow, isDayClosed, latestClosedDate } from '$lib/server/stats/dates';
import { SETTINGS_DEFAULTS, type CompletionRow } from '$lib/types/stats'; import { SETTINGS_DEFAULTS, type CompletionRow } from '$lib/types/stats';
import crypto from 'node:crypto'; import crypto from 'node:crypto';
@@ -72,6 +74,20 @@ describe('computePayload point-in-time cutoff', () => {
expect(payload.kpis.completionRate).not.toBeNull(); expect(payload.kpis.completionRate).not.toBeNull();
}); });
test('all-time best day picks the highest count and ignores future days', () => {
const payload = computePayload('2026-09-10', settings, {
...data,
completions: [
c('A', '2026-09-01'),
c('A', '2026-09-02'),
c('B', '2026-09-02'),
c('C', '2026-09-02'),
c('D', '2026-09-11') // future relative to asOfDate
]
});
expect(payload.totals.bestDay).toEqual({ date: '2026-09-02', count: 3 });
});
test('payload has all lab sections', () => { test('payload has all lab sections', () => {
const payload = computePayload('2026-09-10', settings, data); const payload = computePayload('2026-09-10', settings, data);
expect(payload.retention.cohorts.length).toBeGreaterThan(0); expect(payload.retention.cohorts.length).toBeGreaterThan(0);
@@ -138,4 +154,57 @@ describe('snapshot persistence', () => {
const snap = await ensureSnapshot('2026-01-10', {}, testDb); const snap = await ensureSnapshot('2026-01-10', {}, testDb);
expect(snap.payload.meta?.schemaVersion).toBe(PAYLOAD_SCHEMA_VERSION); expect(snap.payload.meta?.schemaVersion).toBe(PAYLOAD_SCHEMA_VERSION);
}); });
test('groups completions by verse date, not a UTC 24h window', async () => {
// A player in a UTC-negative timezone finishing "Jan 10" late at night
// lands on Jan 11 04:00Z. The old UTC end-of-day cutoff dropped them.
await testDb.insert(dailyCompletions).values({
id: crypto.randomUUID(),
anonymousId: 'late',
date: '2026-01-10',
guessCount: 4,
completedAt: new Date('2026-01-11T04:00:00Z')
});
const { snapshot } = await buildSnapshot('2026-01-10', { kind: 'lazy' }, testDb);
expect(snapshot.payload.activity.dau).toBe(1);
});
test('isDayClosed gates persistence for in-progress days', async () => {
await seed();
const open = new Date().toISOString().slice(0, 10);
await buildSnapshot(open, { kind: 'lazy' }, testDb);
await buildSnapshot(open, { kind: 'lazy', replace: true }, testDb);
const live = await ensureSnapshot(open, {}, testDb);
expect(live.date).toBe(open);
// None of the open-day builds may leave a persisted row behind.
expect(await testDb.select().from(statsSnapshot)).toHaveLength(0);
});
});
describe('verse day window', () => {
test('closes at D+1 12:00Z (UTC-12 has finished the local day)', () => {
expect(endOfVerseWindow('2026-01-10').toISOString()).toBe('2026-01-11T12:00:00.000Z');
expect(isDayClosed('2026-01-10', new Date('2026-01-11T11:59:59Z'))).toBe(false);
expect(isDayClosed('2026-01-10', new Date('2026-01-11T12:00:00Z'))).toBe(true);
});
test('latestClosedDate reflects the window lag', () => {
expect(latestClosedDate(new Date('2026-01-11T11:00:00Z'))).toBe('2026-01-09');
expect(latestClosedDate(new Date('2026-01-11T13:00:00Z'))).toBe('2026-01-10');
});
test('completedBefore pins a point-in-time build', async () => {
await testDb.delete(analyticsEvents);
await testDb.delete(dailyCompletions);
await testDb.insert(dailyCompletions).values([
{ id: crypto.randomUUID(), anonymousId: 'early', date: '2026-01-10', guessCount: 1, completedAt: new Date('2026-01-10T05:00:00Z') },
{ id: crypto.randomUUID(), anonymousId: 'late', date: '2026-01-10', guessCount: 2, completedAt: new Date('2026-01-11T04:00:00Z') }
]);
const pinned = await loadStatsData('2026-01-10', testDb, {
completedBefore: new Date('2026-01-10T12:00:00Z')
});
expect(pinned.completions.map((c) => c.anonymousId)).toEqual(['early']);
const full = await loadStatsData('2026-01-10', testDb);
expect(full.completions).toHaveLength(2);
});
}); });
+11 -1
View File
@@ -1,5 +1,16 @@
# in progress # in progress
(admin/stats)
- [ ] WAU/MAU graph
- [ ] new players per day should show 3 months
- [ ] full width graphs need to have mouse tracking correctly
- [ ] re: "segment mix over time": we need a counter for recently returning players
- [ ] (add clarification for past 30 days) maybe just remove "churned"
(community stats)
- [ ] show yesterday's verse in the community stats
- [ ] show human readable numbers in the community stats
- [ ] fix the "last 30 days" github style view to be full-width
# todo # todo
@@ -17,7 +28,6 @@
- How do you balance rewarding knowledge vs incentivising learning? - How do you balance rewarding knowledge vs incentivising learning?
- instructions - instructions
- classic mode: identify what book the verse is from (e.g. Genesis, John, Revelations...) in as few guesses as possible. - classic mode: identify what book the verse is from (e.g. Genesis, John, Revelations...) in as few guesses as possible.