---
name: datafuel
description: |
  DataFuel fetches web pages through anti-bot unlockers and proxies and
  returns clean markdown, HTML, JSON or screenshots. Use it when a plain
  HTTP fetch is blocked, rendered client-side, or when you need many pages
  of one site. Also runs Google searches and returns parsed results.
  Pick the endpoint by the shape of the work: one URL, a URL
  list, a whole site section, or a batch of known URLs.
---

# DataFuel

Base URL: `https://scraping-api.datafuel.ai/api/v1`
Auth: `X-API-Key: df_key_…` on every request. `Authorization: Bearer df_key_…` is accepted too; `X-API-Key` wins if both are set.
MCP: `https://scraping-api.datafuel.ai/mcp` (streamable HTTP, same header).
Full schema: `https://scraping-api.datafuel.ai/docs/openapi.yaml`. This file: `https://scraping-api.datafuel.ai/skill.md` (also `/llms.txt`).
Human docs: https://docs.datafuel.ai (guides and the full reference, same rules, longer explanations).

## Choose the endpoint

| You have | Use | Sync? | Cost (Basic / Premium proxy) |
|---|---|---|---|
| One URL, need its content | `POST /task` type `unlocker` | yes | 1 / 10 credits, 5 / 20 with `js_rendering` |
| One site, need its URL list | `POST /map` | yes | 1 / 10 per call, any number of links |
| One start URL, need many pages | `POST /crawl` | no | per page queued, same as a task |
| A list of known URLs | `POST /job` type `unlocker` | no | per URL, same as a task |
| A question for an AI engine | `POST /task` type `llm_scraping` | yes | 100 |
| A Google search | `POST /task` type `serp` | yes | 10 per result page, proxy fields ignored |
| Many Google searches | `POST /job` type `serp` | no | 10 per query and result page |

Task types and LLM engines can be switched off at runtime. `GET /api/v1/config/capabilities` (no auth) lists what is on, with the reason for anything that is off, and for engines that take `model` the accepted models in `engines[].models` (default first). Check it once before an `llm_scraping` call or a big batch.

Rules of thumb:

- Start with `/task` on the request engine (`js_rendering` off). Cheapest, fastest, keeps the exact URL.
- Turn `js_rendering` on only when the result is empty, `redirected` is true and you wanted the filtered URL, or `/map` came back with `no_links_on_page`. Browser pricing applies.
- Discover before you crawl: `/map` costs one credit and tells you how big a section is. Then `/crawl` with `include_paths` and a `max_pages` you can afford.
- Use `/job` when you already know the URLs (at least two, at most 10,000 per job; `queries` for serp, `prompts` for llm_scraping). It is cheaper and more predictable than a crawl.
- Never loop `/task` over a list you got from `/map`. That is a `/job`.
- Send an `Idempotency-Key` on every write. A retry with the same key attaches to the running task instead of charging twice.

## Request shapes

Every write has the same envelope: `type`, proxy fields (`proxy_type` Basic or Premium, `proxy_country`, ...), and `attributes`. LLM scraping uses only `proxy_country`, the country the engine answers from (any country when omitted); SERP ignores every proxy field.

```json
POST /task
{"type": "unlocker", "proxy_type": "Basic",
 "attributes": {"url": "https://example.com/p/1", "result_format": "markdown"}}
```

```json
POST /map
{"attributes": {"url": "https://example.com/docs", "search": "guide", "limit": 500}}
```

```json
POST /crawl   → 202 {"job_id": "…"}
{"proxy_type": "Basic",
 "attributes": {"url": "https://example.com/docs", "max_pages": 100, "max_depth": 3,
                "exclude_paths": ["\\.pdf$"], "result_format": "markdown"}}
```

```json
POST /job     → 200 {"id": "…"}
{"type": "unlocker", "multithreaded": true,
 "attributes": {"urls": ["https://example.com/p/1", "https://example.com/p/2"],
                "result_format": "markdown"}}
```

```json
POST /task
{"type": "serp",
 "attributes": {"query": "best crm for startups", "country": "us", "language": "en",
                "page": 1, "result_format": "json"}}
```

```json
POST /task
{"type": "llm_scraping", "proxy_country": "us",
 "attributes": {"prompt": "best crm for startups", "engine": "duck_ai", "model": "claude-haiku-4-5"}}
```

Engine `duck_ai` (duck.ai) also takes `model` (`gpt-6-luna` default, `gpt-5.4-mini`, `claude-haiku-4-5`, `mistral-small-2603`, `tinfoil/gpt-oss-120b`, `tinfoil/gemma4-31b`; the current list is `engines[].models` in `GET /api/v1/config/capabilities`). `websearch` switches search for the `tinfoil/*` models only; the other duck.ai models decide on their own. `result.data` is `{text, citations}`; `[n]` markers in `text` point at `citations[].n`. Image-generation prompts are not supported and fail (refunded). `follow_up_prompt` works on `openai` and `perplexity` only; other engines fail the task (refunded).

SERP attributes: `query` (`queries` on a job), `engine` (`google_search`, the default, or `google_shopping`, see below), `country` (`gl`), `language` (`hl`), `page` (how many result pages, 1-10, merged into one `organic` list and one `product_grids` list; 10 credits each, unserved pages refunded), `google_domain`, `device` (`desktop` default, or `mobile` for the results a phone gets, same sections; no `total_results`), one of `location` / `uule` / `lat`+`lon` (must lie in `country` when both are set, or Google ignores it; set `country` with `location`, since a named location alone does not change the results' country, and use `lat`+`lon` for a precise origin), and Google filters (`tbs`, `safe`, `cr`, `lr`, `uds`, ...). `ai_overview: true` adds Google's AI Overview (`text` + `references`) when the query gets one (a few seconds more when Google defers it). `result_format` is `json` (default: `organic`, `product_grids`, `places`, `questions`, `related_searches`, `filters`, ...), `markdown` or `html`. With `html`, the AI Overview is a `<section data-df="ai_overview">` (`text`, `products`, `references`) in Google's overview container. `resolve_url: true` resolves Google's `/goto` links (a few seconds more): `url` in `json`/`markdown`, a `data-df-url` attribute next to each `/goto` `href` in `html`; without it, `/goto` rows carry only `redirect` (except `site:` queries, always resolved to check the operator). Proxy fields are ignored: searches exit through DataFuel's own pool. `engine: google_shopping` reads one Google Shopping product page instead: pass `url` (`urls` on a job), a `/search` product link with `ibp=oshop` and `prds`, e.g. a `product_grids` link, plus `country`/`language` for the market; it returns `offers` (merchant, price, availability, delivery, link), `rating`, `reviews`, `specifications`, `variants`, reviews, pros/cons and Q&A where shown, 10 credits; web search options are rejected, and a product with no page in that country is refunded as `SERP_EMPTY_RESULTS`.

Useful unlocker attributes: `result_format` (`html`, `markdown`, `json`, `png`, `jpeg`), `main_content_only` and `include_images: false` for smaller markdown, `wait_for_selector` with `js_rendering`.

`extract_selector` and `extract_regex` are **strings**: a bare selector/pattern, or a JSON-encoded object of name → selector. An object is rejected with `400 INVALID_ATTRIBUTES`.

```json
"extract_selector": "{\"price\": \".price\", \"links\": \"a @href\"}"
```

`js_instructions` is an **array** of single-action objects, run in order, actions may repeat:

```json
"js_instructions": [{"fill": ["input[name=q]", "laptops"]}, {"click": "button[type=submit]"}, {"wait_ms": 1000}]
```

The older object form keyed by action is still accepted, but its order is not guaranteed and an action cannot repeat. `GET /api/v1/config/js-instructions` lists the actions; unknown actions answer `400 INVALID_ATTRIBUTES`. `block_resource` takes one resource type or an array (`["Image", "Font", "Media"]`). `body` and `content_type` are sent with `method` `POST`, `PUT` or `PATCH` on both engines. Sticky proxy sessions go in `attributes.proxy_session_id` / `attributes.proxy_ttl`; the top-level fields of the same name are ignored.

Crawl defaults: `max_pages` 100 (cap 10 000), `max_depth` 3 (cap 10), `concurrency` 5. Only same-site links under the start path are followed unless `include_subdomains` or `allow_backward_links` is set. `result_use_ai` is rejected on crawl; sitemaps and robots.txt are not used yet.

## Read the envelope before `result`

Every task, job task, map and crawl page carries metadata next to `result`:

| Field | Meaning |
|---|---|
| `status` | `completed`, `failed`, `pending`, `processing` |
| `status_code` | HTTP status the target answered. A 200 can still be an error page, a 404 still completes and bills. |
| `final_url`, `redirected` | Where the page really came from. Filtered listing URLs often redirect to a canonical page. |
| `blocked`, `protection` | Target answered 403/429/503 or an anti-bot wall. The task is `failed` and refunded. Retry with `js_rendering: true` or a Premium proxy. |
| `credits_used` | Charged for this task, 0 when it failed |
| `error` | Engine detail when `status` is `failed` |
| `engine` | `llm_scraping` only: the LLM engine the task asked for |

`result.data` is a string for `html` and `markdown`, an object for `json`, `extract_selector` and AI output.

Result bodies are kept for 30 days after the task finishes, then deleted: store what you need. After that `GET /task/{id}` answers `410 RESULT_EXPIRED`, and job and crawl result pages keep the entry's metadata with `result: {"status": "expired", "error_detail": "…"}`. A `POST /task` or `/map` retried with the old `Idempotency-Key` answers `410` too, every time: the key stays bound to the expired task.

`POST /task`, `POST /map` and `GET /task/{id}` can answer `202 {"id": "…", "code": "TASK_STILL_PROCESSING"}`: not a result, not an error. Poll `GET /task/{id}` with that id, or resend the POST with the same `Idempotency-Key`.

## Async recipe

```
POST /crawl or /job                   → id
loop: GET /crawl/{id} or /job/{id}    → status, counters
      sleep 2s while status is pending or processing
GET /crawl/{id}/results?limit=100     → pages + next_cursor, follow until absent
GET /job/{id}/results?limit=100       → task results + next_cursor, follow until absent
POST /crawl/{id}/cancel or /job/{id}/cancel → stop; tasks not started are refunded
```

Instead of the polling loop, add `"webhook": {"url": "https://…", "metadata": {…}}` to `POST /job` or `POST /crawl`: one signed `POST` arrives when it finishes (`job.completed|failed|cancelled`, `crawl.*`), with counts but no results, so fetch those as above. Verify it with any Standard Webhooks library and the secret from `GET /webhooks/secret` (rotate: `POST /webhooks/secret/rotate`, the old secret keeps signing 24 h). Deduplicate on the `webhook-id` header; failed deliveries are retried for about 46 hours. To receive every job or crawl event without passing `webhook` each time, create an account endpoint: `POST /webhooks/endpoints {"url", "event_types": ["crawl.*", "job.failed"]}` returns its own `secret` once; `POST /webhooks/endpoints/{id}/test` sends a `webhook.test` event. `GET /webhooks/deliveries?job_id=…` shows what was sent and why it failed, and `POST /webhooks/deliveries/{id}/retry` resends one. An endpoint that fails for 72 h is disabled; re-enable it with `PATCH {"enabled": true}`.

Crawl and job results can be read while the work runs; unfinished tasks appear as status stubs. Job results come in task order, at most 100 per call (default 50); `tasks_count`, `tasks_complete` and `tasks_failed` cover the whole job on every page. `stop_reason` says why a crawl stopped: `max_pages`, `max_depth_exhausted`, `insufficient_credits`, `cancelled`.

## Billing

Charged per task when queued, refunded when the task fails. A blocked page is a failed page on both engines. Crawl `total_cost` counts charges at queue time and does not subtract refunds; sum `credits_used` over the results for the net figure. Balance: `GET /users/@me/balance` returns `balance` (spendable total), `plan_balance` (spent first, expires if the plan is not renewed) and `payg_balance` (never expires).

## Find earlier work and usage

| Need | Call |
|---|---|
| Jobs and crawls you started | `GET /job?type=crawl&status=completed&limit=50` → `{jobs, next_cursor}` |
| Tasks, e.g. the failed ones of a job | `GET /task?job_id=…&status=failed` → `{tasks, next_cursor}`, no result: `GET /task/{id}` for that |
| Credit movements | `GET /users/@me/transactions?operation=refund&page=1&limit=50` → `{transactions, total_count, sums}` |
| Usage totals and breakdowns | `GET /task/analytics/dashboard?start_date=…&end_date=…&interval=daily` |

Lists run newest first. `start_date`/`end_date` are `YYYY-MM-DD` (UTC, end inclusive). `/job` and `/task` page with `cursor` (pass `next_cursor` back until it is absent) and `limit` (default 50, max 200); transactions page with `page` and `limit` (default 10, max 200). Each transaction has `plan_amount`, the part of `amount` that moved plan credits; `purchase` is a credit pack purchase (pay-as-you-go credits). In the analytics, `status_code` 0 means the target never answered (timeout, DNS).

## Errors

| HTTP | `code` | Do |
|---|---|---|
| 202 | `TASK_STILL_PROCESSING` | Not ready. Poll, do not treat as failure. |
| 400 | `INVALID_ATTRIBUTES`, `MISSING_TARGET`, `JOB_REQUIRES_MULTIPLE_TARGETS`, `JOB_TOO_MANY_TARGETS`, `UNSUPPORTED_TASK_TYPE`, `INVALID_REQUEST_BODY`, `INVALID_WEBHOOK_URL` (public https only), `INVALID_WEBHOOK_METADATA` (JSON object, at most 4096 bytes), `INVALID_WEBHOOK_ENDPOINT` (1 to 20 known `event_types`, description at most 256 characters); on list calls `INVALID_QUERY_PARAM`, `INVALID_CURSOR`, `INVALID_DATE_FORMAT`, `INVALID_DATE_RANGE` | Fix the request. Do not retry unchanged. |
| 401 | `UNAUTHORIZED` | Bad or missing key. |
| 402 | `INSUFFICIENT_CREDITS` | Top up or lower `max_pages`. |
| 403 | `FORBIDDEN` | The account behind the key is deactivated. Do not retry. |
| 404 | `TASK_NOT_FOUND`, `JOB_NOT_FOUND`, `CRAWL_NOT_FOUND`, `WEBHOOK_ENDPOINT_NOT_FOUND`, `WEBHOOK_DELIVERY_NOT_FOUND` | Wrong id or another account (deliveries are kept 30 days). |
| 409 | `JOB_NOT_CANCELLABLE` | The job already finished. |
| 410 | `RESULT_EXPIRED` | The result body is past the 30-day retention and was deleted. Retrying with the same `Idempotency-Key` answers 410 forever; to run the task again, use a new key (or none). |
| 409 | `WEBHOOK_ENDPOINT_LIMIT_REACHED` | 20 endpoints already. Delete one or widen `event_types`. |
| 409 | `WEBHOOK_ENDPOINT_EXISTS` | The account already has an endpoint with this URL (after a retried create, it is the one you made: rotate its secret if you lost it). |
| 409 | `WEBHOOK_TEST_IN_PROGRESS` | The previous test event for the endpoint is still open; wait a few seconds. |
| 409 | `WEBHOOK_DELIVERY_NOT_RETRYABLE` | An attempt is running (retry in a few seconds) or the endpoint is disabled or deleted (re-enable it first). |
| 409 | `TASK_ALREADY_EXISTS`, `JOB_ALREADY_EXISTS` | The create collided with an existing row. Nothing charged. Retry. |
| 422 | `IDEMPOTENCY_KEY_REUSED` | Same key, different request. Use a new key. |
| 429 | `CONCURRENCY_LIMIT_REACHED` | Wait `Retry-After` (1 s, sent on every route) and retry. |
| 503 | `MODULE_UNAVAILABLE`, `ENGINE_UNAVAILABLE` | Switched off by an operator, reason in `message`. Nothing charged. Do not retry: pick another engine or wait, and check `GET /api/v1/config/capabilities`. |
| 503 | `SERVICE_UNAVAILABLE` | The API could not check your key because a backing store is down. The key is fine and nothing ran. Retry with backoff. |
| 503 | `WEBHOOKS_UNAVAILABLE` | Only `/webhooks/*`: webhook settings are unreachable. Jobs and deliveries are unaffected. Retry. |

A `failed` task with `blocked: false` is usually a timeout or proxy error. Retry once with a new `Idempotency-Key` (the old key replays the stored failure); if it fails again, switch engine or proxy plan.

Repeated 5xx or timeouts from the API itself (not from a target site): check https://status.datafuel.ai before retrying in a loop. `GET /api/v1/healthz?deep=1` answers 503 while a dependency is down. No auth for either.

## MCP

Point any MCP client at `https://scraping-api.datafuel.ai/mcp` with the `X-API-Key` header (or `Authorization: Bearer`). Tools mirror the REST routes and bill the same: `scrape`, `search`, `get_task`, `map`, `create_job`, `get_job`, `get_job_results`, `cancel_job`, `crawl`, `get_crawl`, `get_crawl_results`, `get_balance`, `list_jobs`, `list_tasks`, `get_analytics`. `scrape` and `create_job` cover `unlocker` and `llm_scraping`; `search` runs one Google search (SERP batches are REST-only). The tools default `include_images` to false for markdown and take `extract_selector` / `extract_regex` as plain objects.

`npx -y @datafuel/mcp init` sets this up in Claude Code, Cursor, VS Code, Windsurf, Claude Desktop, Codex and Gemini CLI.
