E-commerce scraping is the hardest routine job in the data-collection business because major marketplaces deploy the most mature anti-bot stacks on the internet: request rate limiting, IP reputation scoring, JavaScript challenges and behavioral analytics, usually stacked on top of each other. A collector that only solves one layer — say, it rotates IPs but ships an obvious headless fingerprint — survives days, not months. None of those layers is optional: the marketplace scores the combination, so a perfect IP with a broken fingerprint still loses.

The teams that collect pricing, assortment and availability data for years all share one trait: they treat the proxy as layer one of a four-layer architecture, not as a magic bypass. The good news is that the defenders’ stack is knowable — every layer they deploy answers a specific signal, and you can answer each one deliberately rather than hoping your rotation hides the rest. This playbook walks through those four layers, the rotation strategy that matches each page type, and the operational discipline — especially around 429 responses — that keeps a scraping operation alive at scale. If your end goal is a price intelligence feed, the same architecture powers a dedicated price monitoring guide on this blog.

The four-layer architecture for e-commerce scraping

Each layer answers a different detector. Miss one and the marketplace can still classify you as automation, whatever your IP looks like.

  • Layer 1 — Identity: consumer-ISP residential IPs. Datacenter addresses die at the reputation check before any content loads; residential IPs with a real ASN history pass the door and see actual product pages. Rotate per request for listing sweeps, pin sticky sessions for flows that must hold identity.
  • Layer 2 — Fingerprint hygiene: User-Agent, Accept-Language, timezone and screen geometry must agree with the exit IP’s geography. Use a real browser engine for JS-challenge pages and randomize viewport and platform slightly — identical fingerprints cluster and get culled together.
  • Layer 3 — Human pacing: randomized delays instead of fixed intervals, believable navigation sequences, and honest retry behavior. Behavioral analytics profile request cadence; metronomic timing is the cheapest signal for them to catch.
  • Layer 4 — Attribution logging: every request tagged with session ID, exit IP, country and retry count, every response status recorded. When a block wave starts, you can tell which subnet decayed instead of guessing.

Checklist: before your first production run

Run this six-point pass before pointing a collector at any marketplace.

  1. Confirm your exit IPs pass an ASN and reputation check — a full how to test a proxy routine covers this in minutes.
  2. Verify fingerprint consistency: one browser profile per exit geography, with headers that match the IP’s country and language.
  3. Set a requests-per-second budget per session and enforce it in the scheduler, not in hope.
  4. Define your 429 policy before you need it: exponential backoff, session recycle, and a hard stop that protects the pool.
  5. Wire attribution logging: exit IP and response status on every request, exportable per session.
  6. Decide the cadence: how often each listing truly needs refreshing — most price data does not change minute to minute.

Comparison: rotation strategy by page type

Page typeIP strategySession policy
Search & listing pagesRotate per request across the poolNo session continuity needed
Product detail pagesSticky session, 5–15 minutesKeep one IP through the item view
Cart & checkout flowsStatic or long-sticky IPOne identity for the entire funnel
Account dashboardsStatic IP per accountNever rotate under a logged-in session

How to scale without getting banned

  1. Start with one marketplace and one category; prove the architecture before multiplying targets.
  2. Segment traffic: listing sweeps on per-request rotation, product and cart work on sticky sessions, accounts on dedicated static IPs.
  3. Cap concurrency per subnet — ten gentle workers beat one hundred aggressive ones that trip a subnet-wide block.
  4. Back off on 429s immediately: double the delay, recycle the session, and only resume after the backoff window.
  5. Review attribution logs daily: rising error rates on one ASN or subnet mean it is burning — pull it before the marketplace does.
  6. Recalibrate weekly: challenge types change, so re-run your fingerprint and reputation checks on a schedule.

Common mistakes

  • Hammering through a 429. Retrying at full speed after a rate-limit response converts a soft warning into a hard ban and can flag the whole subnet.
  • Bringing a datacenter IP to a reputation fight. No fingerprint polish survives a hosting ASN on a marketplace’s front door.
  • Mismatched fingerprint and geography. A German Accept-Language header on a Texas residential IP is exactly the inconsistency behavioral models hunt for.
  • Flying without logs. When bans arrive, teams without per-request attribution spend days guessing which subnet failed instead of reading the answer off a chart.

FAQ

How do I avoid bans when scraping e-commerce sites?

You reduce probability, you do not eliminate it. Residential IPs with clean ASNs, matched fingerprints, human-paced request timing, disciplined 429 backoff and subnet-level concurrency caps together get most collectors into the zone where blocks are rare and recoverable. The single biggest lever is pacing — most bans are triggered by volume patterns, not by the IP alone.

What is the best proxy for Amazon-style marketplaces?

Rotating residential with sticky-session support is the baseline: per-request rotation for search and listing pages, sticky sessions for product detail and anything cart-related, static IPs for seller or buyer accounts. Datacenter proxies fail at the reputation layer; mobile proxies work but usually cost more than the task justifies. Match the tier to the page type rather than buying one premium tier for everything.

How many IPs do I need?

Size from your request budget, not from a rule of thumb. A rough starting point: one IP can sustain roughly 1,000–3,000 gentle requests per hour before reputation pressure builds, so a 100,000-request-per-day job wants dozens of rotating exits plus a handful of sticky sessions — and your attribution logs will tell you the real number within a week. Treat that first estimate as provisional: concurrency caps, target leniency and pacing all shift the true requirement, which is exactly why the logs matter more than the arithmetic.

Is scraping legal?

It depends on jurisdiction, target and method: publicly available data is generally treated more leniently than logged-in or personal data, and terms of service, copyright and local computer-misuse laws all apply. Get a legal review for your specific case, respect robots directives and rate limits, and never scrape personal data. This article is operational guidance, not legal advice.

Marketplace anti-bot systems are not getting weaker, but they are predictable: they score identity, fingerprint, pacing and history. Build all four layers deliberately and your collector becomes boring to them — which is exactly what you want. FTProxy residential proxies gives you clean consumer-ISP identity with per-request rotation and sticky sessions in one pool, so layers one through four plug together without duct tape.