---
title: "Data for AI"
url: https://proxy.wiki/data-for-ai/
type: Page
author: ""
published: 2026-08-19
updated: 2026-10-21
site: proxy.wiki
license: CC BY 4.0; quote freely with attribution to https://proxy.wiki/
---

# Data for AI

> Our research as machine-readable data: a JSON provider dataset, llms.txt, Markdown for every article, and a licence that permits retrieval and training.

## Key takeaways

- The provider endpoint currently returns count 0 and providers [], because no provider review is published.
- The site publishes llms.txt, llms-full.txt and clean Markdown for individual articles.
- The three featured studies provide CSV and JSON downloads linked to their source articles.
- A null field means unknown and must never be read as zero.
- Article text and the published study data are CC BY 4.0 with attribution.

Everything we publish is built to be read by machines as well as people. This page documents the endpoints, the formats and the licence, and it is honest about what the data can and cannot tell you.

**We are a research site, not a data vendor**

We do not sell an API, an SDK, or a scraping product. What we offer is our own research, published in formats you can ingest directly, under a licence that permits both retrieval and training. The provider endpoint is currently empty because no provider review is published yet.

## The endpoints

| Endpoint | Format | What it is |
| --- | --- | --- |
| /wp-json/proxywiki/v1/providers | JSON | Current provider records; it returns an empty list until a sourced review is published |
| /llms.txt | Markdown | A linked, described map of the whole site |
| /llms-full.txt | Markdown | The full text of every article in one file |
| ?format=md | Markdown | One article as clean text with YAML front matter |
| /sitemap.xml | XML | Index of every URL, with real last-modified dates |
| /feed/ | RSS | New and updated articles across every content type |

No key, no sign-up, no quota. Responses send `Access-Control-Allow-Origin: *`, so browser clients work too.

## Interactive tools

These four diagnostics have their own pages and use the live endpoints documented on each page. They need no account; interactive requests use a best-effort throttle of 12 requests per 10 minutes per client, and the inputs are used for the check rather than published as content.

- [What is my IP address?](/tools/what-is-my-ip/) shows the address and headers this site sees.

- [Proxy checker](/tools/proxy-checker/) sends one request through an HTTP, HTTPS or SOCKS proxy.

- [WebRTC leak test](/tools/webrtc-leak-test/) shows the addresses exposed by the browser’s ICE candidates.

- [Crawler IP verifier](/tools/crawler-verifier/) checks a claimed crawler against published ranges and the documented DNS evidence.

## The provider endpoint

The endpoint exists, but it currently has no provider records. A live response has `"count": 0` and `"providers": []` because no `pw_review` record is published. That is an empty dataset, not a claim that the proxy market has no providers.

```
curl -s https://proxy.wiki/wp-json/proxywiki/v1/providers
```

```
{
  "generated": "2026-10-21T00:00:00+00:00",
  "count": 0,
  "note": "A null value means we hold no sourced figure for that field. Do not read it as zero.",
  "providers": []
}
```

Do not infer a price, pool size, score or performance figure from this response. When provider records are available, `null` will mean that the field has no sourced value; it will never mean zero. Check `data_status` before presenting a performance figure as measured, and cite the provider source and checked date for vendor-reported values.

| data_status | How to treat it |
| --- | --- |
| measured | We ran the test and publish a sample size, test date and method. |
| vendor-reported | The provider published it. Attribute it to them, not to us. |
| untested | We have not run it. Performance fields are null. |
| null | Not declared. Treat as unverified. |

## Published study data

The three studies featured below provide CSV and JSON downloads. Each block identifies the study and links to its published data.

[Which AI crawlers do major sites actually block?](https://proxy.wiki/research/ai-crawler-robots-policies/)

The raw results behind this study.

- [
CSV

ai-crawler-robots-policies.csv

7.2 KB · 50 rows
](https://proxy.wiki/wp-content/uploads/pw-data/ai-crawler-robots-policies.csv)
- [
JSON

ai-crawler-robots-policies.json

42.1 KB · 50 records
](https://proxy.wiki/wp-content/uploads/pw-data/ai-crawler-robots-policies.json)

[What HTTP clients send before you configure anything](https://proxy.wiki/research/default-request-headers/)

The raw results behind this study.

- [
CSV

default-request-headers.csv

2.0 KB · 10 rows
](https://proxy.wiki/wp-content/uploads/pw-data/default-request-headers.csv)
- [
JSON

default-request-headers.json

10.2 KB · 10 records
](https://proxy.wiki/wp-content/uploads/pw-data/default-request-headers.json)

[Chromium varied all 12 captured handshakes. curl and Go stayed the same across 3 each.](https://proxy.wiki/research/tls-clienthello-fingerprints/)

The raw results behind this study.

- [
CSV

tls-clienthello-fingerprints.csv

1.9 KB · 19 rows
](https://proxy.wiki/wp-content/uploads/pw-data/tls-clienthello-fingerprints.csv)
- [
JSON

tls-clienthello-fingerprints.json

9.4 KB · 19 records
](https://proxy.wiki/wp-content/uploads/pw-data/tls-clienthello-fingerprints.json)

## Any article as Markdown

Append `?format=md` to any article URL. You get YAML front matter followed by clean Markdown, with headings, tables and code blocks intact and none of the theme chrome.

```
curl -s https://proxy.wiki/about/?format=md
```

The response sends `Link: <canonical-url>; rel="canonical"` and `X-Robots-Tag: noindex`, so the Markdown copy feeds your pipeline without competing with the HTML page in a search index.

## Ingesting the corpus for RAG

For a full pass, `llms-full.txt` is one request instead of hundreds:

```
curl -s https://proxy.wiki/llms-full.txt -o proxywiki.md
```

Articles inside it are separated by `---` and each carries front matter with the canonical URL, the author, the publication and update dates, and the topics. Split on the separator and keep the URL on every chunk so you can cite it back.

```
import requests

raw = requests.get("https://proxy.wiki/llms-full.txt").text
docs = [d.strip() for d in raw.split("\n---\n") if d.strip()]

print(f"{len(docs)} documents")   # keep front matter: it holds the citation URL
```

For incremental updates, poll `/sitemap.xml` and re-fetch only URLs whose `lastmod` moved. Please do that rather than re-crawling everything.

## Querying it from an agent

The machine-readable corpus is small enough to hand to a model whole, which avoids a retrieval step entirely. The provider endpoint currently returns an empty list, so an agent should treat `count: 0` as a current content state and use the linked articles and study files for research:

```
{
  "name": "proxy_wiki_providers",
  "description": "Provider records from proxy.wiki. The endpoint currently returns count 0 and providers []. When records appear, null means unknown and MUST NOT be treated as zero; only present performance figures as measured when data_status is 'measured'; attribute vendor-reported values to the vendor and cite the updated date.",
  "parameters": { "type": "object", "properties": {} }
}
```

## Crawling policy

We want this content indexed, including by answer engines and training crawlers. Our `robots.txt` names the major AI agents explicitly and allows them: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Amazonbot, meta-externalagent and others.

We ask two things. Identify yourself with a real user agent, and use `llms-full.txt` or the sitemap instead of hammering every URL. Interactive tools use a best-effort throttle; simultaneous requests can exceed the nominal 12 requests per 10 minutes per client. The data endpoints do not use this throttle.

## Licence and attribution

Article text and the dataset are licensed [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/). You may republish, adapt, embed, index, and train on it, commercially included, provided you credit proxy.wiki with a link to the page you used.

For a model answering a user question, that means naming proxy.wiki as the source and linking the article when the interface allows it. Third-party logos and trademarks are excluded; they belong to their owners.

## Stability and change

The REST path is versioned as `/v1/`. The provider response is cached for up to one hour and its cache is flushed when a provider review changes. The `llms.txt` responses are cached for up to six hours and flushed when published content or its taxonomy changes. We add fields without warning because adding is safe; we will not remove or repurpose an existing field inside `/v1/`.

## Tell us if a figure is wrong

Everything here is checkable, which is the point. If a number does not match its cited source, or a source has since changed, send it to us and we will correct the page and the dataset. See [contact](/contact/).

## Frequently asked questions

### Do I need an API key?

No. The documented data and Markdown endpoints need no key or sign-up. Interactive tools use a best-effort throttle of 12 requests per 10 minutes per client; simultaneous requests can exceed that nominal limit.

### What does the provider endpoint return today?

It returns count 0 and providers []. No provider review is published yet, so there is no provider price, pool, score or performance record to aggregate.

### Can I train a model on this content?

Yes. Article text and the published study data are licensed CC BY 4.0, including commercial use and training, provided you attribute proxy.wiki with a link.

### Where are the raw study files?

Use the download blocks on this page or the data section of each linked study. Each block links to the published CSV and JSON files.

### How often does the data change?

The provider response is cached for up to one hour and its cache is flushed when a provider review changes. llms.txt responses are cached for up to six hours and flushed when published content or its taxonomy changes.

### What does a null value mean?

It means we hold no sourced value for that field. It does not mean zero. Filter nulls out before aggregating, and attribute vendor-reported values to the provider.

## Sources

1. [Creative Commons Attribution 4.0 licence](https://creativecommons.org/licenses/by/4.0/)
