Data sources

Overview

Searchkeep reads three first-party sources through their official APIs. Each is optional and configured per site in SITES. AI citations, which have no API yet, are imported from a CSV export.

How syncing works

A Cron Trigger runs every 15 minutes. Each pass does only the work that is due, within a budget of API calls (SYNC_MAX_CALLS_PER_RUN), time and D1 writes, stores its progress in D1 and stops; the next pass continues where it left off. A pass with nothing due exits immediately. The cron only syncs once the deployment has recorded ready, and every pass re-checks Access first. Analysis and MCP tools then read from D1, which is why answers are fast and do not consume API quotas. Live calls are reserved for operations that must be current, such as inspect_url.

• Daily increments first. Every day each source is brought up to date before any history is backfilled: Search Console re-pulls the last 4 days and fills gaps of up to 30 days, the sitemap snapshot is refreshed, Bing is pulled in full, and Cloudflare adds each missing day of AI crawler requests and Web Analytics (going back 30 and 90 days on the first run).
• Then the history backfill. Search Console history is backfilled one day at a time, newest first, back 16 months (GSC_BACKFILL_MONTHS). The backfill stops for the day when it has written SYNC_DAILY_WRITE_BUDGET rows (80,000 by default, under D1 Free’s 100,000 rows written per day), so on the free plan the full backfill takes days to weeks. list_sites and setup_status show how far back the stored range reaches.
• Small batches. Workers Free allows 10 ms of CPU per invocation, so work is split into small units. A unit that is interrupted three times in a row (for example by the CPU limit) is skipped, so the sync never gets stuck; if that keeps happening, lower GSC_ROWS_PER_DAY or move to Workers Paid.
• Upserts by date. Every row is keyed by its date (or week), so re-running a sync is safe and idempotent.
• Nothing is deleted. Data older than a source’s retention window stays in D1. That is the whole point.

To run a pass right away, use Sync now in the dashboard’s Settings, or POST /api/sync from an authenticated session. The dashboard also lists recent sync runs and what each one did.

Google Search Console

Access

• A Google Cloud service account with the Google Search Console API enabled in its project.
• Its key’s client_email and private_key in GSC_CLIENT_EMAIL and GSC_PRIVATE_KEY.
• The service account’s email added to each property as a Restricted user. It uses the webmasters.readonly scope. Grant Full only if you enable submit_sitemap.
• gscProperty must match the property exactly: sc-domain:example.com for domain properties, https://example.com/ for URL-prefix properties.

Data

• Search Analytics per day: site totals, totals by country and device, and query × page rows with clicks, impressions, CTR and position.
• Sitemaps: submission and download times, errors, warnings, submitted vs indexed counts (snapshot once a day).
• URL Inspection, live, through inspect_url.

Limitations

• 16 months of history. Google keeps only the last 16 months. Searchkeep can backfill exactly that much and no more, so history before your deploy date minus 16 months is unavailable. Deploy early.
• Anonymized queries. Google hides rare queries to protect user privacy. Totals by query therefore add up to less than the site total. Searchkeep stores site-level totals separately so overall numbers stay correct, and labels query-level results accordingly.
• Rows per day. The API returns at most 25,000 rows per request. Searchkeep stores up to GSC_ROWS_PER_DAY query × page rows per day (2,500 by default), paging 25,000 at a time when you raise it. Raising it keeps more of the long tail at the cost of D1 writes and CPU.
• Delay. Data arrives two to three days late, and the most recent days can change.
• URL Inspection quota. About 2,000 inspections per property per day.

Bing Webmaster Tools

Access

• An API key from Settings → API access in Bing Webmaster Tools, in BING_API_KEY.
• Each user has one key that covers all their verified sites. Set bingSiteUrl to the site URL as registered.
• The key can also submit URLs and remove sites. Searchkeep never calls those endpoints, but anyone holding the key could. Store it only as a secret.

Data

• Traffic totals, query statistics and page statistics: clicks, impressions, average position.
• Crawl issues (URL, issue and HTTP code), refreshed once a day.

Limitations

• Weekly granularity, no date filter. The Bing API returns all of its data on every call, aggregated by week. Searchkeep pulls it in full once a day and upserts by week. Queries against Bing data align the date range to whole Monday–Sunday weeks.
• Queries and pages separately. Bing has no query × page data and no country or device breakdown.
• No API for AI Performance. Bing’s AI Performance report (citations in Copilot and Bing AI answers) has been in public preview since February 2026 without an API; Microsoft says an API is on its backlog. Searchkeep imports its CSV export instead, see Importing AI citations.

Cloudflare

Access

One API token with three read permissions:

• Zone · Analytics · Read for AI crawler traffic on each zone.
• Account · Account Analytics · Read for Web Analytics.
• Account · Access: Apps and Policies · Read for the security checks (not a data source, but the same token).

Data

• AI crawler requests from Cloudflare’s HTTP request analytics: requests per day by crawler, path and status code. Searchkeep matches the user agent against a list of known AI crawlers and assistants and classifies them by operator (OpenAI, Anthropic, Perplexity, Google, and many more) and by purpose: training, search, or user (an assistant fetching a page because someone asked).
• Web Analytics: page views and visits by path and referrer host, used for the Clicked from AI step (referrers such as chatgpt.com, perplexity.ai, claude.ai, copilot.microsoft.com, gemini.google.com).
• Core Web Vitals: p75 LCP, INP and CLS by path.

Limitations

• Proxied zone required for AI crawler data. Cloudflare only sees requests that pass through it, so the zone must be proxied (orange cloud). DNS-only records produce no crawler data.
• User agents only. On Free plans, crawlers are identified by their user agent, so a bot that pretends to be a browser is not counted, and one that spoofs a crawler’s user agent is.
• Limited retention. Cloudflare’s GraphQL analytics keep HTTP request data for at least 31 days and Web Analytics for about six months. Searchkeep backfills 30 and 90 days on the first run and then adds every day, so a window that expired before you deployed cannot be recovered.
• Referrers are lossy. Some AI apps strip the referrer, so AI referral visits are a lower bound.

Importing AI citations

The Cited step of the GEO funnel comes from Bing Webmaster Tools’ AI Performance report. Because it has no API, you export it as CSV in Bing Webmaster Tools and import the file into Searchkeep:

• In the dashboard: Settings → Import AI citations.
• Or from a terminal: POST /api/import/ai-citations?site=<id>&date=<YYYY-MM-DD> with the CSV as the request body.

# From a machine with cloudflared; it opens a browser to sign in through Access.
cloudflared access curl "https://searchkeep.example.com/api/import/ai-citations?site=acme&date=2026-11-01" \
  -X POST --data-binary @ai-performance.csv
# {"site":"acme","imported":42,"skipped":0,"rowsWritten":42,"columns":{"page":"URL","citations":"Citations","date":null}}

Columns are detected by name: a URL or page column, a citations column, and an optional date column. Without a date column, every row gets the date parameter (today by default). Rows without a URL or a number are skipped and counted. Re-importing the same file overwrites the same rows, so it is safe. The CSV can be up to 5 MB.

How the GEO funnel is built

For each page, geo_funnel joins these steps on the normalized path:

The interesting rows are the gaps: pages that are crawled but not cited (AI reads them and moves on) and pages that are cited but not clicked (AI answers the question without sending anyone). Sort with sortBy (crawled, cited, aiVisits or clicks); once citation data exists, the result also lists the most-crawled pages that were never cited. When a step has no data, the result’s notes say which setting or import is missing.

Reading the numbers

• Sum of query rows ≠ site total, because of anonymized queries in GSC and the per-day row limit.
• Bing figures are weekly; daily charts show Bing only at weekly resolution.
• Positions are impression-weighted averages, so they can be re-aggregated exactly across days and pages.
• The latest two or three days of GSC data are provisional and will be revised by the next sync.
• Crawler requests count fetches, not unique pages: one page fetched 50 times counts 50.