Methodology

Everything below is what actually runs, not a description of what we intend to run. Every parameter that affects a measurement is recorded per observation and reflected here — if this page and the collector ever disagree, the collector is what shipped and this page has a bug.

Models measured

ProviderModel IDSurfaceUser location signal
OpenAI gpt-5.5-2026-04-23 grounded Yes — city_region
Anthropic claude-sonnet-5 grounded Yes — city_region
Perplexity sonar native Yes — city_region
Google Gemini gemini-3.5-flash grounded No — city/state named in the prompt text only

"Surface" describes how the model reaches the web: grounded means an official web-search tool attached to the API call; native means web access built into the model's own inference path (Perplexity). We never automate a consumer chat interface — only official APIs with active search.

Prompt battery — version 1, frozen

Eight prompts, four question archetypes (direct, purchase, comparative, conversational) across two services. This exact wording is frozen: it is not corrected, improved, or rephrased once measurement starts, because a changed prompt breaks comparability with every prior cycle. A wording change requires a new version number and a documented discontinuity in the series — never a silent edit.

Fixed parameters

Temperature: left at each provider's default — not overridden. This is a limit, not a choice: an unstated default can drift when a provider changes it, and we have no lever to hold it constant today.

Search limits: OpenAI is capped at 3 tool calls per response (max_tool_calls) with reasoning.effort: "low" — "minimal" is not supported together with the web_search tool on this model family. Anthropic is capped at 3 web searches per response (max_uses). Neither cap is a floor: a model can search fewer times, or not at all, and that is recorded as data, not corrected.

User location: OpenAI, Anthropic and Perplexity receive an approximate city/region/timezone location object alongside the prompt. Gemini's grounding tool does not accept one — its only location signal is the city name written into the prompt text itself.

Measurement origin: requests are issued from EU infrastructure (Helsinki), not from within the United States. This is disclosed because it can plausibly influence what a location-aware search surfaces, and we have not measured whether or how much.

Measurement cadence

Cost, not just methodology, decides which provider runs how often — disclosed here because it affects how fresh a given number is. Weekly (every market, every service): Gemini and Perplexity only. Monthly deep pass (tier-1 metro markets only, 3 repetitions for variance): all four providers, including OpenAI and Anthropic. Daily sentinel (tier-1 metros, one prompt, Perplexity only): a low-cost daily pulse, not a substitute for the weekly series.

Declared limits

Data license

All measurement data is published under CC BY 4.0. Reuse it, republish it, build on it — commercially or not — the only requirement is attribution back to The Sierra Index.

Methodology by Valerio Fioretti