Observability & Telemetry

AI Web Feeds uses a dual-write telemetry model: every API route observation is always appended to local NDJSON files, and—when DATABASE_URL is configured—the same payload is persisted asynchronously to Postgres (api_request_logs). Product analytics events from the browser follow the same pattern via usage_events.

Architecture

Browser (trackEvent)
       │
       ▼
POST /api/telemetry/events ──► usage_events (Postgres, when configured)

App Router handler (withRouteTelemetry)
       │
       ├──► data/telemetry/api-events.ndjson   (always)
       └──► api_request_logs (Postgres, when configured)

Implementation lives in:

• apps/web/lib/telemetry.ts — NDJSON append and summary aggregation
• apps/web/lib/telemetry-route.ts — withRouteTelemetry wrapper
• apps/web/lib/server/telemetry-store.ts — Postgres dual-write
• apps/web/lib/track-event.ts — browser trackEvent() helper
• packages/ai_web_feeds/src/ai_web_feeds/observability.py — optional Logfire

Route telemetry (withRouteTelemetry)

Wrap App Router handlers to record one row per request without blocking the response:

import { withRouteTelemetry } from "@/lib/telemetry-route";

export const GET = withRouteTelemetry("articles.list", GETHandler);

The wrapper:

1. Assigns or forwards an x-request-id header
2. Measures handler duration and captures status, pathname, method, and sorted query keys
3. Hashes client IPs (never stores raw addresses)
4. Redacts error messages before persistence
5. Queues writes via Next.js after() so the client response is not delayed

Dual-write order inside queueTelemetryWrite:

1. NDJSON — recordApiTelemetry() appends to api-events.ndjson (always succeeds when the directory is writable)
2. Postgres — recordApiRequestLog() inserts into api_request_logs (skipped silently when DATABASE_URL is unset; errors are logged otherwise)

Pass backendTarget when a route proxies to the Python service:

export const POST = withRouteTelemetry("search.log", POSTHandler, {
  backendTarget: process.env.BACKEND_URL ?? null,
});

NDJSON event shape (ApiTelemetryEvent)

Written to {AIWF_TELEMETRY_DIR}/api-events.ndjson:

Admin audit events use a separate file: admin-audit.ndjson.

usage_events schema

Product analytics events use contract version usage-event-v1. The TypeScript types are in apps/web/lib/server/usage-events.ts; the SQLModel mirror is UsageEvent in packages/ai_web_feeds/src/ai_web_feeds/models.py. Alembic migration 008_usage_events creates the tables for the Python catalog database; the Next.js app bootstraps equivalent DDL inline on first write when using Neon.

Event surfaces

api_request_logs schema

Mirrors ApiTelemetryEvent with an additional ingested_at timestamptz defaulting to NOW(). Indexed on route_key, timestamp, and request_id.

Client trackEvent API

Import from apps/web/lib/track-event.ts in client components and hooks:

import { trackEvent } from "@/lib/track-event";

await trackEvent("reader.article.open", {
  surface: "reader",
  properties: {
    articleId: article.id,
    openedFrom: "feed",
  },
});

Behavior:

• Browser-only — no-op during SSR
• Session id — auto-generated via crypto.randomUUID() and stored in sessionStorage under aiwf_telemetry_session_id
• User id — defaults to getStoredUserId() when omitted
• Transport — POST /api/telemetry/events with keepalive: true so events survive navigation
• Failure handling — network errors are swallowed; analytics must never break reader or search UX

Ingest endpoint

POST /api/telemetry/events accepts:

{
  "events": [
    {
      "eventName": "reader.filter.apply",
      "surface": "reader",
      "properties": { "filterId": "ml-only" }
    }
  ]
}

• Batch size limit: 25 events per request
• Returns 202 with { "accepted": N } on success
• Returns 503 when DATABASE_URL is not configured (usage events require Postgres)
• Binds userId from the authenticated session when the client omits it

The ingest route itself is wrapped with withRouteTelemetry("telemetry.events.ingest", …) so its own latency is recorded in the dual-write pipeline.

Optional Python Logfire

Python paths (catalog sync, CLI, storage) support optional Pydantic Logfire tracing. When LOGFIRE_TOKEN is unset, all observability calls are no-ops.

from ai_web_feeds.observability import configure_observability, span

configure_observability()

with span("catalog_sync.stage", stage="topics"):
    ...

• configure_observability() reads LOGFIRE_TOKEN and LOGFIRE_SERVICE_NAME (default ai-web-feeds)
• Returns False when the token is missing or the logfire package is not installed
• span() opens a Logfire span only when observability is active; otherwise it yields immediately

Catalog sync calls configure_observability() at startup in catalog_sync/sync.py.

Environment variables

Next.js / NDJSON

Postgres dual-write

When DATABASE_URL is unset:

• NDJSON route telemetry continues to work
• POST /api/telemetry/events returns 503
• recordApiRequestLog is skipped without failing the request

Python Logfire

Example local .env fragment (see root .env.example):

AIWF_TELEMETRY_DIR=../../data/telemetry
# AIWF_TELEMETRY_SALT=replace-with-a-stable-hashing-salt
DATABASE_URL=postgresql://user:pass@host.neon.tech/db?sslmode=require  # pragma: allowlist secret
# LOGFIRE_TOKEN=pylf_...
# LOGFIRE_SERVICE_NAME=ai-web-feeds

Rate limiting on serverless

POST /api/telemetry/events applies a per-IP in-memory limiter (apps/web/lib/server/rate-limit.ts). Buckets live in a process-local Map, so on serverless platforms each instance enforces its own window. This is sufficient for abuse mitigation at the edge of a single replica but is not a globally consistent quota across all deployments.

For strict cross-instance limits, add a shared store (for example Vercel KV or Redis) in front of the ingest route. Until then, treat rate limits as best-effort and monitor anomalous ingest volume via usage_events and api_request_logs.

Admin API authorization is enforced in route handlers via withBetterAuthAdminGuard (role must be admin). Middleware only checks for the presence of a session cookie on /api/admin/* and /admin/* pages as defense in depth.

Retention and operations

For local development, NDJSON files under data/telemetry/ are gitignored. Delete them freely when resetting a dev environment.

Verification

# Web unit tests (telemetry store, trackEvent, route wrapper)
cd apps/web && pnpm vitest run lib/telemetry lib/track-event lib/server/telemetry-store

# Python observability (optional Logfire paths)
uv run pytest tests/tests/packages/ai_web_feeds/unit/test_observability.py -q

With DATABASE_URL set, exercise the ingest path:

curl -s -X POST http://localhost:3000/api/telemetry/events \
  -H 'Content-Type: application/json' \
  -d '{"events":[{"eventName":"reader.filter.apply","surface":"reader","properties":{"test":true}}]}'

Confirm dual-write by checking both data/telemetry/api-events.ndjson (after hitting any instrumented API route) and the usage_events / api_request_logs tables in Neon.

Related documentation

• Admin Observability — OAuth-protected /admin dashboard and security model
• Runtime authority — which stores own catalog vs. operational data