Why your scraper looks like a bot. And how to stop.

Most people picture bot detection as a single yes/no check — “is this a bot?”. That’s a 2018 mental model. The 2026 model is a continuous risk score, computed per-request, from many independent signals: the TLS handshake, the IP’s reputation, header order, the JS environment, mouse behaviour, account history, time of day. Each signal nudges the score up or down.

What you experience as “blocked” is what happens when the score crosses a threshold. The work isn’t to defeat individual checks — it’s to keep the score low enough that no threshold ever fires. A scraper that looks identical to a normal browser at every layer simply gets the page, the same way a normal user would.

Every layer of the request stack leaks information. The table below is the complete list of layers we’ve seen sampled by Cloudflare, Akamai, DataDome, PerimeterX, and Imperva. The middle column is what a real Chrome on a real Mac actually sends; the right column is what a typical scraper sends instead.

The exact numbers vary per vendor, but the structure is universal. Every off-signal pushes the score up; every plausible signal keeps it flat. Once the score crosses a threshold (commonly ~50), you get a challenge. Past ~75, you get a hard block. Below the table is what each factor typically contributes — your own measurements will vary, but the shape is reliable.

The collection happens in three layers, in this order — each one is its own bouncer at the door.

Three flags do most of the work. The rest of the dial is for edge cases. We rotate fingerprints per session and keep them internally consistent — Chrome on Win11/Intel stays Chrome on Win11/Intel for the entire JS environment, not Chrome’s UA with a Mac canvas hash.

The flags below are the entire surface area. Combine based on the target’s protection level.

The default — works on most sites

For 80% of public-page targets, this is enough. No CAPTCHAs trigger because the score never gets close to threshold.

curl -X GET 'https://api.example.com/scrape' \
  -H 'ApiKey: YOUR_API_KEY' \
  -G \
  --data-urlencode 'url=https://example.com' \
  --data-urlencode 'js=true' \
  --data-urlencode 'stealth=true'

The protected-site default

Add residential IPs and geo-match for sites with stricter scoring (Amazon, Google, LinkedIn, classifieds, travel).

import requests

response = requests.get(
    'https://api.example.com/scrape',
    params={
        'url': 'https://heavily-protected-site.com',
        'js': 'true',
        'stealth': 'true',
        'premium_proxy': 'true',
        'proxy_country': 'US',
    },
    headers={'ApiKey': 'YOUR_API_KEY'},
)
print(response.text)

Mobile profile

Some sites serve different HTML to mobile and only protect the desktop tier — flipping device can be enough to drop the risk score below threshold.

const params = new URLSearchParams({
  url: 'https://target.com',
  js: 'true',
  stealth: 'true',
  device: 'mobile',
});

const res = await fetch('https://api.example.com/scrape?' + params, {
  headers: { ApiKey: 'YOUR_API_KEY' },
});
console.log(await res.text());