Rotating proxy setup is usually treated as a checkbox: switch rotation on, assume blocks disappear. In practice rotation is a policy — a deliberate decision about how long one IP identity should live before the next request inherits a new one. Get that policy right and a crawler keeps running for weeks; get it wrong and you spend your whole bandwidth budget proving that rotation cannot repair a behavioral block.
This guide covers the three rotation models most teams actually need — per-request, per-minute or per-N requests, and sticky sessions — plus the failure handling that keeps rotation honest and the audit trail that proves which exit IP served which response. Every rule below can be applied to an existing scraper without rewriting it.
Match the rotation window to the task
- Per-request rotation: a new exit IP on every request. Correct for stateless targets — listing pages, SERP sweeps, public JSON endpoints, price feeds. Nothing carries over, so there is no session to maintain and no cookie to lose.
- Per-minute or per-N rotation: one identity for a fixed window or a fixed number of requests. Correct for multi-step crawls that need partial continuity — pagination, faceted search, product variants — without a full session.
- Sticky sessions: one IP for the whole flow. Mandatory for anything authenticated: logins, carts, checkouts, dashboards, support portals. An IP change mid-flow re-triggers verification and can void the cart.
- Hybrid: sticky per hostname plus per-request for CDN assets. The HTML stays consistent while images and scripts are prefetched through a wider IP spread.
Six-point checklist before you rotate anything
- Classify every URL in the crawl by statefulness: does it set or read a session, cart, or auth token? Store the answer in the queue record so the worker picks the model automatically.
- Define a session key strategy — one key per account, per cart, or per target market. Reusing a single key across accounts is how cross-contamination starts.
- Set a session length in minutes and a maximum request count per key, whichever comes first. Most providers support both limits on the same sticky key.
- Choose the geo pool per target, not globally. A US price crawl should never land on an EU exit just because that pool is larger.
- Instrument the client to log exit IP, session key, status code, and latency for every response. Without that field you cannot tell rotation from luck.
- Define a retry budget per URL — typically two retries on different IPs — plus a hard stop, so a failing endpoint cannot loop forever.
Comparison
| Rotation window | Best for | Watch out for |
|---|---|---|
| Per request | Stateless listings, SERP, JSON APIs, price feeds | Breaks flows that rely on cookies or tokens; identical fingerprints jumping IPs every call look bot-like |
| Per minute / per N | Paginated crawls, faceted search, variant collection | Rotation can split one logical session across two identities; keep N below the token lifetime |
| Sticky 10–30 min | Logins, carts, checkout QA, dashboards, geo-testing | One IP absorbs the whole request budget; a single block takes the session down with it |
| Hybrid sticky + rotate | Sites with heavy assets or per-page personalization | More moving parts — log the model per request or audits become guesswork |
How to set up rotation step by step
- Pick two entry points: a single rotating endpoint for stateless work and a sticky-capable username format for sessions. Keep both credentials in one place so workers never mix them up.
- Wire the model into the queue. The worker reads the statefulness flag and requests either a fresh IP or a session key derived from the account or cart ID.
- Add localization headers that match the exit country — Accept-Language, currency, and a timezone where the client supports it. A German IP paired with an en-US header is a mismatch worth avoiding.
- Implement backoff before rotation. On 429, wait with exponential jitter; rotating harder into a rate limit only burns gigabytes. On 403, rotate and put that target on a cool-down.
- Validate the exit IP before trusting the payload. One echo request per session confirms the IP, country, and ASN you think you bought. Our guide on how to test a proxy lists the checks worth automating.
- Emit metrics per window: success rate, block rate, bytes per successful page, distinct IPs used. Review weekly, because rotation settings drift as targets change.
Common mistakes
- Rotating on every failure. If the block is behavioral, a new IP hits the same wall with a fresh identity and a worse reputation. Back off first, rotate second.
- Rotating mid-checkout. Any IP change inside an authenticated flow re-triggers verification, and some sites empty the cart. Sticky is not optional there.
- Ignoring the geo pool boundary. A global rotation pool silently mixes markets, so your price comparison measures two countries and calls it noise.
- No exit IP in logs. Without that column you cannot separate a rotating success from a cache hit, or prove which IP fetched a page.
FAQ
Should I rotate per request or use sticky sessions?
Rotate per request when the target is stateless: listings, search results, JSON feeds. Use sticky when state matters — logins, carts, checkout, dashboards — because an IP change mid-flow re-triggers verification and can break the session outright. If a target is unclear, start sticky for one session and inspect whether cookies or tokens appear in the response; if they do, keep it sticky. Our rotating vs sticky guide walks the decision tree with concrete examples.
When does rotation not help at all?
Rotation only fixes blocks scoped to the IP. If you are rate-limited per account, tripping a WAF rule because of your fingerprint or request pattern, or blocked by a login wall, a new IP changes nothing — you are simply blocked again faster. Diagnose first: check the status code and response body, then try the same request from a different account on the same IP.
How fast should I rotate?
As slowly as the task allows. Use per-request only for stateless endpoints, and set sticky sessions to the shortest window that keeps a multi-step flow coherent — often 5 to 15 minutes. Faster rotation costs more IPs, triggers more verification, and rarely improves success once pacing is correct.
How do I keep an audit trail?
Log one record per response: timestamp, target URL, session key, exit IP, exit country, status code, latency, and bytes. Store the same fields on screenshots and data snapshots. That record lets you replay a failure, prove which IP fetched a page, and show that geo-targeting was honored — useful for compliance reviews and for debugging “the price changed” complaints.
Conclusion
Rotation is a policy you design per task, not a switch you flip once. Classify each URL by statefulness, choose the window, keep geo discipline, back off before you rotate, and log the exit IP on every response. Then rotation becomes boring infrastructure instead of a source of mystery failures. Start with FTProxy residential proxies — per-request and sticky windows on the same credentials — and tune the window per target.