Architecture

Overview

You deploy exactly one Worker. MCP, the dashboard and the scheduled sync are three entrances to it. They share one core library and one D1 database, so there is only one place where numbers are computed.

Request flow

1. MCP requests to /mcp and dashboard requests to / and /api/* hit Cloudflare Access first, which authenticates the person (cookie for the browser, OAuth for MCP clients) and attaches a signed JWT.
2. The Worker, built on Hono, verifies the JWT on every request and computes the readiness state before serving anything but /healthz, /setup and /robots.txt.
3. The MCP entrance is a stateless Streamable HTTP server built with the official TypeScript SDK; the dashboard is a static single-page app served from Worker assets that talks to /api.
4. The cron entrance (scheduled()) is invoked internally by Cloudflare every 15 minutes. It checks the ready flag in D1, re-reads the Access configuration through the API, and then calls core’s runSync(), which does only the work that is due within its budgets.
5. All three call core. Core reads and writes data only through the Store interface and calls external APIs only through Source adapters.

Packages

The repository is a pnpm workspace. apps/ holds deployable entry points; packages/ holds reusable libraries. Dependencies only point from apps to packages, and core depends on no platform.

searchkeep/
├─ apps/
│  ├─ cloudflare/       Worker entry (Hono) + wrangler.jsonc   @searchkeep/cloudflare
│  └─ site/             This website (SvelteKit)               @searchkeep/site
├─ packages/
│  ├─ core/             Domain model, analysis, Store/Auth/Source interfaces
│  ├─ store-d1/         Drizzle schema, D1 migrations, Store on D1
│  ├─ mcp/              MCP tool definitions (call core only)
│  ├─ dashboard/        Dashboard UI (Svelte + Vite single-page app)
│  └─ sources/
│     ├─ gsc/           @searchkeep/source-gsc
│     ├─ bing/          @searchkeep/source-bing
│     └─ cloudflare/    @searchkeep/source-cloudflare
├─ docs/                charter, ADRs, roadmap, deploy runbook
└─ AGENTS.md            instructions for AI agents working on the repo

• core uses only Web-standard APIs (fetch, Request/Response, WebCrypto). No Cloudflare binding types appear in core, mcp, dashboard or the sources; a dependency-cruiser rule in CI enforces it.
• store-d1 is the only place with SQL. Core calls Store methods, never queries.
• mcp and dashboard are separate packages so a future runtime entry can reuse them unchanged.

Data model

Search data from GSC and Bing goes into one fact table keyed by site, source, date, page and query, with pages and queries in their own dimension tables. Country and device breakdowns, AI crawler requests, Web Analytics visits, Web Vitals, imported AI citations, sitemaps and crawl issues have tables of their own. Every page is stored as a normalized path, which is what lets cross-source views such as the GEO funnel join them.

• Dates are UTC calendar dates.
• Positions are stored as impression-weighted sums so they re-aggregate exactly across days, pages and queries.
• Site-wide daily totals are stored separately from query × page rows, so anonymized queries do not distort totals.
• Migrations are generated with drizzle-kit and are forward-only. The Worker applies pending migrations itself on its first request, tracked in the same d1_migrations table Wrangler uses.
• Sync progress (cursors, the daily write count) and the readiness flag live in a small key-value table in D1.

There are no Durable Objects, which are the hardest primitive to port. Short-lived API access tokens are cached in memory per isolate.

Extension points

Version 0.1 ships Cloudflare implementations only, but every platform-specific capability sits behind an interface. Adding a platform means adding an implementation, not changing core.

Store implementations are verified by one shared test suite that runs against a local D1 today; a Postgres implementation is done when it passes the same suite.

Principles

From the project charter, in priority order. When two conflict, the earlier one wins.

1. User data stays in the user’s own account: no telemetry, no callbacks, no dependency on the project’s servers.
2. Only first-party data the site owner has access to: no backlinks, competitors or rank scraping.
3. Self-hosted with zero operations: runs on free tiers, upgrades with “Sync fork”.
4. Read-only by default: any write to an external system must be explicitly enabled.
5. Core is platform-agnostic; platform capabilities live behind interfaces.
6. Docs are written for agents to execute: every step repeatable and verifiable.
7. Few dependencies: if a Web-standard API can do it, no library is added.

Key decisions

Recorded as ADRs in docs/adr/, because these are the choices most likely to be “helpfully” reverted:

• One Worker, three entrances rather than separate services.
• No Durable Objects, to keep the core portable.
• Cloudflare Access for authentication, with no auth code in the app.
• Drizzle behind a Store interface, so databases are swappable.
• Fork + Workers Builds for deploys and upgrades, instead of a deploy button.
• Discrete MCP tools rather than Code Mode. About a dozen tools fit comfortably in context; each can be annotated read-only or write so clients can auto-approve reads and confirm the one write; and it needs no sandbox. Tools aggregate on the server and cap their output, which keeps responses small.
• Agent maintenance only through GitHub issues and pull requests, with humans merging and tagging releases.