Fetching sources that refuse

Source retrieval

Fetching sources that refuse

Every endpoint that reads a URL goes through one fetch path. A large share of publishers, repositories, and government hosts refuse an ordinary server-side request while serving the same file to a browser. This is what the service recognises, what it does about it, and — the part that costs the most time to learn — what does not work.

01A refusal is not a missing file

The failure that matters is the one that looks like success. A shielded host answers 200 or 202 with Content-Type: text/html and either no body or a challenge page. Trusting the status line or the content type reports a live document as a paywall, a landing page, or an unparseable file. Body evidence outranks headers, and these signals escalate rather than resolve:

Signals treated as a refusal
Signal	What it actually means	Seen on
202, empty body	Request accepted, file withheld pending a challenge the client did not run	AWS WAF (bepress / Digital Commons repositories)
401 · 403 · 429 · 451	Refused outright — often on agent string alone, not identity	Publishers, journals, some `.gov` hosts
200, `Content-Length: 0`	A refusal wearing a success code	CDN edge rules
Challenge markers in the first bytes	An interstitial served in place of the file: `awsWafCookieDomainList`, `gokuProps`, `challenge-platform`, `_Incapsula_Resource`, and similar	AWS WAF, Cloudflare, Imperva, DataDome, PerimeterX
`%PDF-` in the first bytes	Not a refusal — this wins over any status or content type	Hosts that serve real files under a 403

02Three rungs, cheapest first

An ordinary response never escalates; only a refusal does. The rung that produced the bytes is reported as fetch_path (and X-Source-Fetch-Path), because a document that could only be retrieved through a proxy or a browser is not ordinarily reachable, and a caller should be able to say so.

Each rung exits on success. Only a refusal moves right.

direct — the header set a real browser sends

This is the rung that matters most, and the cheapest. A bare User-Agent is the giveaway: no Chrome tab has ever made a document request without Sec-Fetch-Dest, Sec-Fetch-Mode, client hints, and a same-origin Referer. Sending the complete set turns a blank 403 into the actual file on a large share of hosts — including ones that returned zero bytes to a perfect agent string alone.
residential — Oxylabs exit, pinned per host

Clears blocks keyed on datacentre IP reputation. The session is sticky per host, because a challenge cleared from one exit IP is worthless if the next request leaves from another.
browser — Browser Driver executes the challenge

The only rung that passes a JavaScript challenge. It loads the origin, lets the challenge run, then re-fetches the file from inside that page so the cookies it set are attached. Capped at 25 MiB, since the body returns as base64.

03What does not work

These are the dead ends, kept here because each one costs a day to rediscover.

Approaches that fail, and why
Approach	Outcome	Why
Residential IP against a JavaScript challenge	Still refused	AWS WAF and Cloudflare challenges are proof-of-browser, not proof-of-IP. A residential exit returns the same `202`, only from a nicer address.
Navigating a browser straight at the file	No document	Chromium treats a PDF URL as a download, not a page, so there is nothing to read. The challenge is bound to the origin anyway — load that first, then fetch in-page.
A commercial rendering proxy	Works, priced out	It does return the file, at ~25 credits per request against a small monthly pool. Self-hosted Playwright returns byte-identical results for free.
Peeking at a streamed body to detect a challenge	Corrupts the download	Reading the first bytes consumes them. An ordinary stream is therefore never peeked; when suspect headers force a peek and the body turns out to be the real file, the bytes are put back before the caller sees it.
Escalating without a wall-clock budget	Gateway timeout	Three rungs in sequence outlive nginx's 60s read timeout, and the caller gets an HTML 504 instead of a reason. The ladder shares a 50s budget and names any rung it had to skip.
Escalating every result on a list	Unaffordable	Proxy traffic is metered and the browser rung costs ~20s. Escalation is per document, on request — a page of ten links uses ten cheap direct probes.

04When every rung is refused

The origin's own status and message are returned. A block is never smoothed into an empty document, a partial extraction, or a generic error — a caller that cannot tell "this host refuses us" from "this file is not there" will present one as the other.