Skip to content
§ CRAWLER How Quantiveo fetches
Updated 2026-09

The Quantiveo crawler.

Who we are

Quantiveo is a media-intelligence platform. Our crawler fetches news articles whose URLs appear in public event data (the GDELT Project) and in publishers' own RSS feeds, extracts a short excerpt and metadata, computes derived signals, and links readers back to the original page. We do not republish articles. Since September 2026 the text of a page is deleted within 30 days of being fetched; text collected before then is being purged in scheduled batches.

How we identify ourselves

Every request to a third-party site carries the user agent Quantiveo/1.0 (+https://quantiveo.com/bot; contact@quantiveo.com). The one exception is a page a customer explicitly asks one of our agents to read, which carries Quantiveo-Agent/1.0 (+https://quantiveo.com/bot; document ingest). We never use browser identities, we do not rotate identities, and we do not use proxies to get around blocks.

Earlier releases identified some checks as QuantiveoBot or QuantiveoLivenessCheck. Those names are retired, and a robots.txt rule written against any of them still applies to us. Data we read through a service's own API (GDELT, Wikimedia, the YouTube Data API) identifies as that service's policy requires; we do not crawl those sites' pages — a YouTube channel is checked with the Data API or not at all, including when we verify a candidate source or re-check a listed one.

What we honour

robots.txt. Before every fetch we read your robots.txt and follow the rules for User-agent: Quantiveo (or any of our earlier names), then User-agent: *, including Crawl-delay. A redirect is a new fetch: the target's robots.txt is checked before we follow it. To block us entirely:

User-agent: Quantiveo
Disallow: /

Reservation signals. An X-Robots-Tag header containing noai or noimageai, or a tdm-reservation: 1 header, stops us from using that page and from fetching that host again for a day.

Refusals. A 403 or 451 response is treated as final. We do not retry under a different identity. We pace requests to any one site at no more than one per second, or slower when your Crawl-delay asks for it. If your robots.txt cannot be read we treat that as a refusal for the next hour.

Removal and corrections

Publishers can ask us to stop fetching a site or to remove excerpts at copyright@quantiveo.com. People who appear in our data can ask for access, correction, or removal through our privacy request page.