Skip to content

Data for AI

Our research as machine-readable data: a JSON provider dataset, llms.txt, Markdown for every article, and a licence that permits retrieval and training.

On this page 11 sections
  1. The endpoints
  2. Interactive tools
  3. The provider endpoint
  4. Published study data
  5. Any article as Markdown
  6. Ingesting the corpus for RAG
  7. Querying it from an agent
  8. Crawling policy
  9. Licence and attribution
  10. Stability and change
  11. Tell us if a figure is wrong

Everything we publish is built to be read by machines as well as people. This page documents the endpoints, the formats and the licence, and it is honest about what the data can and cannot tell you.

We are a research site, not a data vendor
We do not sell an API, an SDK, or a scraping product. What we offer is our own research, published in formats you can ingest directly, under a licence that permits both retrieval and training. The provider endpoint is currently empty because no provider review is published yet.

The endpoints

Endpoint Format What it is
/wp-json/proxywiki/v1/providers JSON Current provider records; it returns an empty list until a sourced review is published
/llms.txt Markdown A linked, described map of the whole site
/llms-full.txt Markdown The full text of every article in one file
<any-article>?format=md Markdown One article as clean text with YAML front matter
/sitemap.xml XML Index of every URL, with real last-modified dates
/feed/ RSS New and updated articles across every content type

No key, no sign-up, no quota. Responses send Access-Control-Allow-Origin: *, so browser clients work too.

Interactive tools

These four diagnostics have their own pages and use the live endpoints documented on each page. They need no account; interactive requests use a best-effort throttle of 12 requests per 10 minutes per client, and the inputs are used for the check rather than published as content.

The provider endpoint

The endpoint exists, but it currently has no provider records. A live response has "count": 0 and "providers": [] because no pw_review record is published. That is an empty dataset, not a claim that the proxy market has no providers.

curl -s https://proxy.wiki/wp-json/proxywiki/v1/providers
{
  "generated": "2026-10-21T00:00:00+00:00",
  "count": 0,
  "note": "A null value means we hold no sourced figure for that field. Do not read it as zero.",
  "providers": []
}

Do not infer a price, pool size, score or performance figure from this response. When provider records are available, null will mean that the field has no sourced value; it will never mean zero. Check data_status before presenting a performance figure as measured, and cite the provider source and checked date for vendor-reported values.

data_status How to treat it
measured We ran the test and publish a sample size, test date and method.
vendor-reported The provider published it. Attribute it to them, not to us.
untested We have not run it. Performance fields are null.
null Not declared. Treat as unverified.

Published study data

The three studies featured below provide CSV and JSON downloads. Each block identifies the study and links to its published data.

Which AI crawlers do major sites actually block?

The raw results behind this study.

What HTTP clients send before you configure anything

The raw results behind this study.

Chromium varied all 12 captured handshakes. curl and Go stayed the same across 3 each.

The raw results behind this study.

Any article as Markdown

Append ?format=md to any article URL. You get YAML front matter followed by clean Markdown, with headings, tables and code blocks intact and none of the theme chrome.

curl -s https://proxy.wiki/about/?format=md

The response sends Link: <canonical-url>; rel="canonical" and X-Robots-Tag: noindex, so the Markdown copy feeds your pipeline without competing with the HTML page in a search index.

Ingesting the corpus for RAG

For a full pass, llms-full.txt is one request instead of hundreds:

curl -s https://proxy.wiki/llms-full.txt -o proxywiki.md

Articles inside it are separated by --- and each carries front matter with the canonical URL, the author, the publication and update dates, and the topics. Split on the separator and keep the URL on every chunk so you can cite it back.

import requests

raw = requests.get("https://proxy.wiki/llms-full.txt").text
docs = [d.strip() for d in raw.split("\n---\n") if d.strip()]

print(f"{len(docs)} documents")   # keep front matter: it holds the citation URL

For incremental updates, poll /sitemap.xml and re-fetch only URLs whose lastmod moved. Please do that rather than re-crawling everything.

Querying it from an agent

The machine-readable corpus is small enough to hand to a model whole, which avoids a retrieval step entirely. The provider endpoint currently returns an empty list, so an agent should treat count: 0 as a current content state and use the linked articles and study files for research:

{
  "name": "proxy_wiki_providers",
  "description": "Provider records from proxy.wiki. The endpoint currently returns count 0 and providers []. When records appear, null means unknown and MUST NOT be treated as zero; only present performance figures as measured when data_status is 'measured'; attribute vendor-reported values to the vendor and cite the updated date.",
  "parameters": { "type": "object", "properties": {} }
}

Crawling policy

We want this content indexed, including by answer engines and training crawlers. Our robots.txt names the major AI agents explicitly and allows them: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Amazonbot, meta-externalagent and others.

We ask two things. Identify yourself with a real user agent, and use llms-full.txt or the sitemap instead of hammering every URL. Interactive tools use a best-effort throttle; simultaneous requests can exceed the nominal 12 requests per 10 minutes per client. The data endpoints do not use this throttle.

Licence and attribution

Article text and the dataset are licensed Creative Commons Attribution 4.0. You may republish, adapt, embed, index, and train on it, commercially included, provided you credit proxy.wiki with a link to the page you used.

For a model answering a user question, that means naming proxy.wiki as the source and linking the article when the interface allows it. Third-party logos and trademarks are excluded; they belong to their owners.

Stability and change

The REST path is versioned as /v1/. The provider response is cached for up to one hour and its cache is flushed when a provider review changes. The llms.txt responses are cached for up to six hours and flushed when published content or its taxonomy changes. We add fields without warning because adding is safe; we will not remove or repurpose an existing field inside /v1/.

Tell us if a figure is wrong

Everything here is checkable, which is the point. If a number does not match its cited source, or a source has since changed, send it to us and we will correct the page and the dataset. See contact.

Frequently asked questions

Do I need an API key?
No. The documented data and Markdown endpoints need no key or sign-up. Interactive tools use a best-effort throttle of 12 requests per 10 minutes per client; simultaneous requests can exceed that nominal limit.
What does the provider endpoint return today?
It returns count 0 and providers []. No provider review is published yet, so there is no provider price, pool, score or performance record to aggregate.
Can I train a model on this content?
Yes. Article text and the published study data are licensed CC BY 4.0, including commercial use and training, provided you attribute proxy.wiki with a link.
Where are the raw study files?
Use the download blocks on this page or the data section of each linked study. Each block links to the published CSV and JSON files.
How often does the data change?
The provider response is cached for up to one hour and its cache is flushed when a provider review changes. llms.txt responses are cached for up to six hours and flushed when published content or its taxonomy changes.
What does a null value mean?
It means we hold no sourced value for that field. It does not mean zero. Filter nulls out before aggregating, and attribute vendor-reported values to the provider.

Sources

  1. Creative Commons Attribution 4.0 licence