A scraper that works on page one and gets blocked on page one hundred is a common, frustrating story. Usually the proxies aren't the problem — the technique is. Combine quality IPv4 proxies with the habits below and your success rate will climb dramatically.
Match the proxy type to the target
Residential or mobile for strict sites; datacenter or dedicated IPv4 for tolerant ones. This single choice decides more than any other. See residential vs datacenter.
Rotate IPs intelligently
Spread requests across many addresses so no single IP looks abnormal. Use per-request rotation for broad crawls and sticky sessions only where state (login, cart) requires it.
Respect rate limits
Throttle your requests to a human-plausible pace and add randomised delays. Hammering a site is the fastest way to get every IP in your pool flagged.
Send realistic headers
Use a complete, consistent set of browser-like headers — User-Agent, Accept, Accept-Language — and rotate them sensibly. Mismatched or missing headers scream "bot".
Handle cookies and sessions properly
Persist cookies within a session and keep the same IP for that session. Sudden IP changes mid-session break the illusion of a single user.
Back off and retry gracefully
On a 429 or 503, pause and retry with exponential backoff rather than pounding the endpoint. Treat blocks as a signal to slow down, not push harder.
Cache and deduplicate
Don't re-fetch what you already have. Caching responses and skipping duplicate URLs saves bandwidth (and money) and reduces load on the target.
Scrape during off-peak hours
Where timing allows, run heavy jobs when the target is least busy. Your traffic blends in better and you're a lighter burden on their servers.
Target APIs and sitemaps first
Many sites expose public JSON endpoints or sitemaps that are cheaper and more stable to pull than rendering full HTML pages. Always check before brute-forcing pages.
Monitor success rate, not just volume
Track the share of requests returning valid data. A falling success rate is your early warning to adjust rotation, pace or proxy type before you waste bandwidth.
Stay legal and ethical
Collect only publicly available data, honour each site's terms of service and robots guidance, and never overload small sites. Responsible scraping is sustainable scraping.
Use a reliable provider
Cheap, dirty IPs tank your success rate no matter how good your code is. Start from a clean, well-sourced pool — our 2026 ranking shows which networks deliver.