--- title: "Data for AI" url: https://proxy.wiki/data-for-ai/ type: Page author: "" published: 2026-08-19 updated: 2026-08-19 site: proxy.wiki license: CC BY 4.0 — quote freely with attribution to https://proxy.wiki/ --- # Data for AI > Our research as machine-readable data: a JSON provider dataset, llms.txt, Markdown for every article, and a licence that permits retrieval and training. ## Key takeaways - Every article is available as Markdown by adding ?format=md to its URL. - The provider dataset is public JSON with no key and no quota. - A null field means unknown and must never be read as zero. - Everything is CC BY 4.0: retrieval and training are both permitted with attribution. Everything we publish is built to be read by machines as well as people. This page documents the endpoints, the formats and the licence, and it is honest about what the data can and cannot tell you. **We are a research site, not a data vendor** We do not sell an API, an SDK, or a scraping product. What we offer is our own research, published in formats you can ingest directly, under a licence that permits both retrieval and training. If you need proxies or a scraping API, that is what our [provider reviews](/reviews/) are for. ## The endpoints | Endpoint | Format | What it is | | --- | --- | --- | | /wp-json/proxywiki/v1/providers | JSON | Every provider we track, with pricing, pool, targeting and our scores | | /llms.txt | Markdown | A linked, described map of the whole site | | /llms-full.txt | Markdown | The full text of every article in one file | | ?format=md | Markdown | One article as clean text with YAML front matter | | /sitemap.xml | XML | Index of every URL, with real last-modified dates | | /feed/ | RSS | New and updated articles across every content type | No key, no sign-up, no quota. Responses send `Access-Control-Allow-Origin: *`, so browser clients work too. ## The provider dataset ``` curl -s https://proxy.wiki/wp-json/proxywiki/v1/providers ``` The response is shaped like this: ``` { "generated": "2026-08-19T10:00:00+00:00", "count": 12, "note": "A null value means we hold no sourced figure for that field. Do not read it as zero.", "providers": [ { "name": "Example Networks", "url": "https://proxy.wiki/reviews/example-networks/", "types": ["Residential proxies"], "data_status": "vendor-reported", "score": 7.5, "price_per_gb": 4.20, "min_spend": 50, "pool_size": 12000000, "countries": 130, "success_rate": null, "sample_size": null, "tested_on": null, "updated": "2026-08-14" } ] } ``` ### The one rule that matters when you parse this **`null` means unknown. It never means zero.** If you average `price_per_gb` across providers and treat nulls as 0, you will produce a number that is wrong and confidently presented. Filter first: ``` import requests data = requests.get("https://proxy.wiki/wp-json/proxywiki/v1/providers").json() # Correct: drop unknowns before aggregating. priced = [p for p in data["providers"] if p["price_per_gb"] is not None] avg = sum(p["price_per_gb"] for p in priced) / len(priced) if priced else None print(f"{len(priced)} of {data['count']} providers have a sourced price") ``` The same applies to `success_rate`, `pool_size` and every score. Check `data_status` before you present any performance figure as measured: | data_status | How to treat it | | --- | --- | | measured | We ran the test. sample_size and tested_on will be populated. | | vendor-reported | The provider published it. Attribute it to them, not to us. | | untested | We have not run it. Performance fields will be null. | | null | Not declared. Treat as unverified. | ## Any article as Markdown Append `?format=md` to any article URL. You get YAML front matter followed by clean Markdown — headings, tables and code blocks intact, none of the theme chrome. ``` curl -s https://proxy.wiki/about/?format=md ``` The response sends `Link: ; rel="canonical"` and `X-Robots-Tag: noindex`, so the Markdown copy feeds your pipeline without competing with the HTML page in a search index. ## Ingesting the corpus for RAG For a full pass, `llms-full.txt` is one request instead of hundreds: ``` curl -s https://proxy.wiki/llms-full.txt -o proxywiki.md ``` Articles inside it are separated by `---` and each carries front matter with the canonical URL, the author, the publication and update dates, and the topics. Split on the separator and keep the URL on every chunk so you can cite it back. ``` import requests raw = requests.get("https://proxy.wiki/llms-full.txt").text docs = [d.strip() for d in raw.split("n---n") if d.strip()] print(f"{len(docs)} documents") # keep front matter: it holds the citation URL ``` For incremental updates, poll `/sitemap.xml` and re-fetch only URLs whose `lastmod` moved. Please do that rather than re-crawling everything. ## Querying it from an agent The dataset is small enough to hand to a model whole, which avoids a retrieval step entirely. If you expose it as a tool, put the caveats in the tool description — not in a comment your model never sees: ``` { "name": "proxy_wiki_providers", "description": "Proxy provider data from proxy.wiki. A null field means the figure is unknown and MUST NOT be treated as zero. Only present performance figures as measured when data_status is 'measured'; otherwise attribute them to the vendor. Prices change often: cite the 'updated' date alongside any figure.", "parameters": { "type": "object", "properties": {} } } ``` ## Crawling policy We want this content indexed, including by answer engines and training crawlers. Our `robots.txt` names the major AI agents explicitly and allows them: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Amazonbot, meta-externalagent and others. We ask two things. Identify yourself with a real user agent, and use `llms-full.txt` or the sitemap instead of hammering every URL. The interactive tools are rate limited to 12 requests per 10 minutes per client; the data endpoints are not, so use those. ## Licence and attribution Article text and the dataset are licensed [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/). You may republish, adapt, embed, index, and train on it, commercially included, provided you credit proxy.wiki with a link to the page you used. For a model answering a user question, that means naming proxy.wiki as the source and linking the article when the interface allows it. Third-party logos and trademarks are excluded — they belong to their owners. ## Stability and change The endpoints above are versioned in their path (`/v1/`). We will add fields without warning, because adding is safe. We will not remove or repurpose an existing field inside `v1`. The dataset is cached for one hour and regenerates whenever a review is saved. ## Tell us if a figure is wrong Everything here is checkable, which is the point. If a number does not match its cited source, or a source has since changed, send it to us and we will correct the page and the dataset. See [contact](/contact/). ## Frequently asked questions ### Do I need an API key? No. There is no key, no sign-up and no quota on the data endpoints. The interactive tools are rate limited to 12 requests per 10 minutes per client, but the data endpoints are not. ### Can I train a model on this content? Yes. Article text and the dataset are licensed CC BY 4.0, which permits commercial use including training, provided you attribute proxy.wiki with a link. ### What does a null value mean in the dataset? It means we hold no sourced figure for that field. It does not mean zero. Filter nulls out before aggregating, or you will produce numbers that are wrong. ### How often does the data change? The dataset is cached for one hour and regenerates whenever a review is saved. Poll the sitemap and re-fetch only URLs whose lastmod has moved. ### Are the performance figures measured by you? Only when data_status is 'measured', which also requires a test date, a sample size and a link to the method. Otherwise the figures are vendor claims and should be attributed to the vendor.