Deploy

Overview

You deploy Searchkeep by forking the repository and letting Cloudflare Workers Builds build and deploy your fork. There is no “Deploy to Cloudflare” button on purpose: a fork you own is what lets you upgrade later with one click, and it keeps every piece of configuration out of the repository and in Worker Secrets.

This page is the condensed version of the runbook in the repository, docs/deploy.md, which has every click path, CLI and API alternative and troubleshooting entry. The facts and checks are the same. The whole flow takes about 30–45 minutes the first time. Each step says who does it and how to check it worked.

Before you start

• A GitHub account and a Cloudflare account (the Free plan is enough).
• A hostname on a zone that is active in that Cloudflare account, for example searchkeep.example.com. It must not have a DNS record yet, and it does not have to be on the same zone as the sites you analyze.
• Owner access to the Search Console properties (only owners can add users), the Bing Webmaster Tools sites, and the Cloudflare zones you want to connect. Sites you want AI crawler data for must be proxied through Cloudflare.
• Node.js 22 or newer if you want to set secrets from a terminal with npx wrangler. You can also set every secret in the Cloudflare dashboard instead.

1. Fork the repository

Fork noobnooc/searchkeep to your account or organization and keep only the default branch (main). A public fork is fine: there is nothing secret in the repository, and nothing you configure ever gets committed. Do not commit to your fork’s main; keeping it identical to upstream makes “Sync fork” conflict-free.

gh repo fork noobnooc/searchkeep --clone=false
gh repo view <you>/searchkeep --json parent --jq .parent.nameWithOwner
# noobnooc/searchkeep

Verify: the last command prints noobnooc/searchkeep, or your fork’s page on GitHub shows “forked from noobnooc/searchkeep”.

2. Authorize the Cloudflare GitHub App

Verify: github.com/settings/installations lists Cloudflare Workers and Pages with access to your fork, and the fork appears in Cloudflare’s repository picker.

3. Create the Worker with Workers Builds

Pick your fork in the repository picker and use these settings:

The build command compiles the dashboard into the Worker’s static assets. The watch paths make sure a change to the Worker, its packages or its dependencies triggers a build, while changes to the website in apps/site or to the docs do not.

The deploy command (scripts/deploy.mjs) tries to apply D1 migrations remotely, deploys, then tries the migrations once more, and never fails because of them. The default Workers Builds build token has no D1 permission, so on Workers Builds the remote migration step is normally skipped with a warning. That is expected and fine: the Worker applies any pending migrations itself on its first request, tracking them in the same d1_migrations table that Wrangler uses, so a migration is never applied twice. (If you want the build to migrate too, add D1 Edit to the build token; it is optional.)

Verify: the Worker’s Deployments (or Builds) tab shows the latest build as Success, and the build settings show the root directory, both commands and all three include paths.

4. The first build creates D1

The repository’s wrangler.jsonc declares a D1 binding called DB with database_name: "searchkeep" but deliberately no database_id. On the first deploy, Wrangler provisions the database automatically. Its ID lives only in your Cloudflare account and is never written back to the repository, so syncing upstream never conflicts with your setup.

Verify: Storage & Databases → D1 (or npx wrangler d1 list) lists a database named searchkeep. Its tables appear at the latest after the Worker’s first request.

5. Add a custom domain

In the Worker’s Settings → Domains & Routes (or its Domains tab), add a Custom domain such as searchkeep.example.com. Cloudflare creates the DNS record and the certificate. Then make sure the workers.dev route and Preview URLs are disabled.

• A custom domain is required, not cosmetic: the repository’s wrangler.jsonc turns off workers_dev and preview_urls on every deploy and declares no routes, and the Access application in step 6 attaches to this hostname. Adding the domain in the dashboard keeps it out of the repository, so it survives upgrades.
• With workers.dev and previews off, no hostname reaches the Worker except the one Access covers. The Worker would still reject requests without a valid Access JWT anywhere else, but there is no reason to leave the door visible.

Verify (the certificate can take a few minutes):

curl -s https://searchkeep.example.com/healthz
# {"ok":true}

curl -s -w ' %{http_code}\n' https://searchkeep.example.com/api/status
# {"error":"Searchkeep is locked: …","code":"locked","state":"no_access","setup":"/setup"} 403

curl -s -w ' %{http_code}\n' -X POST https://searchkeep.example.com/mcp
# {"jsonrpc":"2.0","error":{"code":-32001,"message":"Searchkeep is locked: https://searchkeep.example.com
#  is not protected by Cloudflare Access yet. See https://searchkeep.example.com/setup."},"id":null} 403

/healthz returns only {"ok":true}, opening https://searchkeep.example.com/ in a browser shows the public “Searchkeep is locked” setup page, and the Worker’s workers.dev URL no longer answers.

6. Protect it with Cloudflare Access

1. In Zero Trust → Access controls → Applications (older dashboards: Access → Applications), add a Self-hosted application for searchkeep.example.com, covering the whole hostname: no path, no bypass rules, no exceptions.
2. Add one Allow policy whose include rule is Emails with your own address, or a short list of specific addresses. Searchkeep reads the policies through the API and stays locked (policy_too_broad) if one lets in Everyone, a public email domain such as gmail.com, any identity-provider login, every IP or a whole country, or uses Bypass or Service Auth.
3. In the application’s Advanced settings, turn on Managed OAuth with dynamic client registration, and allow localhost and loopback clients (used by Claude Code, Cursor and other local clients). This makes Access the OAuth 2.1 authorization server for MCP clients, with PKCE. Without it, Searchkeep stays in oauth_disabled.
4. Copy the application’s Application Audience (AUD) Tag and your team domain from Zero Trust → Settings, and store both as Worker Secrets:

# Run anywhere with Node.js 22+; wrangler asks you to log in the first time.
npx wrangler secret put ACCESS_TEAM_DOMAIN --name searchkeep   # e.g. myteam.cloudflareaccess.com
npx wrangler secret put ACCESS_AUD --name searchkeep

Values are prompted for interactively and never echoed. For ACCESS_TEAM_DOMAIN enter just the hostname. Both values must come from your Access configuration, never from a JWT the Worker received.

Verify:

curl -s -o /dev/null -w '%{http_code}\n' https://searchkeep.example.com/healthz
# 302 (redirect to <team>.cloudflareaccess.com) or 401, never 200

curl -s -o /dev/null -D - https://searchkeep.example.com/mcp | grep -iE '^(HTTP|www-authenticate)'
# HTTP/2 401 and a www-authenticate header pointing to Access (a 302 means Managed OAuth is off)

Opening https://searchkeep.example.com/ in a private window shows the Access login. After you sign in, the dashboard loads instead of the setup page and no longer reports no_access; it may report another state until the next step is done.

7. Add credentials

The Cloudflare token is required: Searchkeep uses it to check your Access setup, and stays in oauth_disabled without it. Search Console and Bing are optional and independent; connect what you have and add the rest later. Setting a secret deploys a new version of the Worker within seconds; no build is needed.

Cloudflare account ID and API token

The account ID is under Account details on the Workers & Pages overview, or in npx wrangler whoami.

npx wrangler secret put CLOUDFLARE_ACCOUNT_ID --name searchkeep
npx wrangler secret put CLOUDFLARE_API_TOKEN --name searchkeep

# Verify the token (paste it when prompted; it is not echoed):
read -rs CF_TOKEN && curl -s https://api.cloudflare.com/client/v4/user/tokens/verify \
  -H "Authorization: Bearer $CF_TOKEN" | jq -r '.result.status'; unset CF_TOKEN
# active

Sites

Describe the sites to analyze in a JSON array and store it in the SITES secret. Only domain is required; leave out the fields of sources you do not use. The full format, and where to find each identifier, is in Configuration. Keep the file outside your clone.

[
  {
    "id": "acme",
    "domain": "acme.com",
    "gscProperty": "sc-domain:acme.com",
    "bingSiteUrl": "https://acme.com/",
    "cloudflareZoneId": "023e105f4ecef8ad9ca31a8372d0c353",
    "webAnalyticsSiteTag": "abc123",
    "brandRegex": "acme"
  }
]

jq -e 'type=="array" and length>0 and all(.[]; .domain)' sites.json   # prints true
npx wrangler secret put SITES --name searchkeep < sites.json

Google Search Console

Searchkeep reads Search Console with a Google Cloud service account.

gcloud services enable searchconsole.googleapis.com --project my-project
gcloud iam service-accounts create searchkeep --project my-project \
  --display-name "Searchkeep"
gcloud iam service-accounts keys create searchkeep-key.json \
  --iam-account searchkeep@my-project.iam.gserviceaccount.com

# Read both fields straight from the key file, so nothing is pasted by hand.
jq -r .client_email searchkeep-key.json | npx wrangler secret put GSC_CLIENT_EMAIL --name searchkeep
jq -r .private_key searchkeep-key.json | npx wrangler secret put GSC_PRIVATE_KEY --name searchkeep

# Then delete the local key file.
rm searchkeep-key.json

Verify: Users and permissions lists the service account with Restricted permission. After step 8, setup_status reports that the service account can read each property.

Bing Webmaster Tools

npx wrangler secret put BING_API_KEY --name searchkeep

# Verify the key (paste it when prompted; it is not echoed):
read -rs BING_KEY && curl -s "https://ssl.bing.com/webmaster/api.svc/json/GetUserSites?apikey=$BING_KEY" \
  | jq '.d[].Url'; unset BING_KEY

Verify: the output lists your sites, including each bingSiteUrl from SITES.

Optional settings

BRAND_REGEX enables brand vs non-brand splits; ENABLE_WRITES=true registers submit_sitemap (and needs Full permission in Search Console). The sync budgets (GSC_ROWS_PER_DAY, GSC_BACKFILL_MONTHS, SYNC_DAILY_WRITE_BUDGET, SYNC_MAX_CALLS_PER_RUN) have defaults sized for the free plans. All of them are described in Configuration.

8. Verify and run the first sync

Open the dashboard and its status page: it reads GET /api/status, which answers in every state once you are signed in, and names any failing readiness check. When it shows ready, connect an MCP client (see MCP server); it should list 12 tools (13 with ENABLE_WRITES=true). Before ready, /mcp answers every request with a JSON-RPC error that explains the lock. Ask the client to run setup_status. You are done when it reports:

• readiness ready;
• every secret you intended to set as present;
• each configured source passing its probe for each site.

The Access check is cached for up to five minutes; GET /api/status?refresh=1 re-checks immediately. Once an authenticated request (opening the dashboard, or any MCP call) sees ready, the Worker records a ready flag in D1. From then on the cron, which runs every 15 minutes, syncs whatever is due. Each pass re-checks Access through the API first and locks again if Access was turned off or loosened.

To start right away instead of waiting for the next pass, use Sync now in the dashboard’s Settings, or POST /api/sync from an authenticated session:

# From a machine with cloudflared; it opens a browser to sign in through Access.
cloudflared access curl https://searchkeep.example.com/api/sync -X POST

Each pass does only what fits its call, time and write budgets and resumes on the next one, so a result of partial with “continues next run” is normal. Daily increments come first: Search Console’s recent days (it lags two to three days), Bing’s full weekly data, 30 days of AI crawler requests and 90 days of Web Analytics. The Search Console history backfill then runs one day at a time back 16 months; on the free D1 write quota that takes days to weeks.

Verify: setup_status shows a recent sync without errors, and list_sites shows a date range for each connected dataset. Over the following days the Search Console range reaches back 16 months.

Deploy with an agent

The repository includes docs/deploy.md and AGENTS.md, written for AI agents: every step is idempotent, has a verification, and human-only steps are marked so the agent stops and asks. With the GitHub and Cloudflare MCP servers connected, a prompt like this is enough:

Deploy Searchkeep (github.com/noobnooc/searchkeep) to my Cloudflare account,
following docs/deploy.md in the repository. Use searchkeep.example.com as the
custom domain and allow only me@example.com in the Access policy.
Stop and ask me whenever a step needs a human.

The steps that need you:

1. authorizing the Cloudflare GitHub App for the fork;
2. creating the Zero Trust organization, the first time;
3. creating the read-only Cloudflare API token (recommended as a human step);
4. adding the service account in Search Console (and creating it, if the agent has no gcloud);
5. generating the Bing Webmaster Tools API key;
6. signing in through Access when you connect an MCP client or open the dashboard.

Troubleshooting

Start with the dashboard’s status page (/api/status), or setup_status once the deployment is ready: it names the failing check. The runbook’s troubleshooting section covers every case in detail.

The dashboard only shows the setup guide (no_access)

Either the request did not come through Access, or ACCESS_TEAM_DOMAIN / ACCESS_AUD are missing or wrong. Check that the Access application covers the exact hostname with no path, that the AUD tag is this application’s, and that you are not using a workers.dev or preview URL.

oauth_disabled

Managed OAuth is off, or Searchkeep cannot read the Access application. Turn Managed OAuth on, and check that CLOUDFLARE_ACCOUNT_ID is set and CLOUDFLARE_API_TOKEN has Access: Apps and Policies Read.

policy_too_broad

A policy lets in more than specific people. Keep one Allow policy that includes your email address.

A source check fails

For GSC, confirm the service account was added to that exact property and that gscProperty matches it (sc-domain: for domain properties). For Bing, confirm bingSiteUrl matches the site as registered. For AI crawler data, confirm the zone is proxied and the token has Zone Analytics Read.

The build log says the remote migration was skipped

Expected with the default build token, which has no D1 permission. The deploy continues and the Worker applies pending migrations itself. If the deploy itself failed, check the root directory (apps/cloudflare) and the project name (searchkeep).

Sync units are reported as skipped after interrupted attempts

A unit that is interrupted three times in a row, usually by the 10 ms CPU limit of Workers Free, is skipped so the sync never gets stuck. Lower GSC_ROWS_PER_DAY, or move to Workers Paid.