Rate limits and the customer who scrapes you

Website scraping protection means capping how many requests one identity can make, then deciding what happens when they go over: a 429, a challenge, or a block. The identity is the hard part. Rate limits at the edge and quotas in the application stop a scraper pinning your database while real customers and Googlebot sail through.

Key Takeaways

• Rate limiting is the mechanism; blocking is the policy. Set the ceiling first, decide the punishment second.
• Per-IP counters on four app servers mean four times the limit you think you set — enforce at the edge, not only in NGINX.
• Scrapers rarely cause an outage. They inflate database CPU, wreck cache hit ratio and leak pricing data while your uptime check stays green.
• Never block on User-Agent alone. Verify Googlebot and Bingbot by reverse DNS or you will lose search traffic.
• Shared NAT, mobile carriers and office wifi put hundreds of real users behind one IP address. Key on a session or an API token where you can.
• Cache the 429 and you have built your own outage. Send Cache-Control: no-store on error responses.
• Expect leakage from residential proxies and headless browsers. Perfect blocking is not the goal — making scraping expensive is.

What does website scraping protection actually mean?

It means deciding, per route, how many requests one identity may make in a window, and what happens at the ceiling — a 429 with a Retry-After header, a JavaScript challenge, a slower response, or a hard block. Identity is where teams get stuck: an IP address, an API key, a session cookie, a device fingerprint, or some combination.

Two things get confused constantly. A volumetric attack is thousands of requests per second from a botnet. A scrape is often two requests per second from one host, running for nine hours. The first looks like an emergency. The second looks like nothing at all until you read the database graphs. They need different controls, and lumping them together is why so many protection rules end up blocking legitimate users while the scraper walks straight through.

Why does scraping hurt a site that looks fine in Grafana?

Because the damage hides in the expensive paths. A scraper walks search with every filter combination, hits product pages that miss cache, and pulls price or availability data straight from the database. Request rate stays unremarkable; database CPU, query time and egress climb steadily. Your homepage uptime check keeps returning 200.

The symptoms we see in practice: cache hit ratio falls off a cliff, p95 latency doubles on a quiet Tuesday, and someone eventually notices a competitor is selling at prices scraped from their own catalogue. Analytics get polluted too — session counts inflate and conversion rate drops with no real change in demand. If the crawler is doing credential stuffing alongside the crawl, you also get a spike in failed logins. That is a security incident wearing a crawler's clothes. How much headroom you have for this depends on your hosting model, which we compared in shared hosting versus VPS versus cloud.

Do you actually need to block scrapers?

No, not always. A brochure site with fifty static pages behind a CDN is already mostly immune: caching serves repeat requests without touching the origin, so a scraper costs you a little egress. Adding aggressive limits there buys nothing and risks blocking real visitors.

You do need protection when any of these are true: search results or filters are dynamic, prices and stock are visible, a login form exists, you run a public API, or bandwidth and database load are billed to you. Then the goal is not to stop all scraping. It is to make scraping slow, expensive and visible. Say that out loud before you write a rule, because it changes what you build.

How does request-level rate limiting work under the hood?

Most implementations use a token bucket or a fixed or sliding window counter. A bucket holds N tokens and refills at a fixed rate; each request takes one. When the bucket is empty, requests are rejected or delayed. NGINX does this with limit_req_zone and limit_req, holding counters in shared memory.

The detail that catches teams out: that shared memory is per node. Four app servers behind a load balancer, each with a 10 r/s limit, hand an attacker 40 r/s. You can centralise the counter in Redis, but that adds a network hop to every request. The cheaper answer is to enforce at the edge — CDN or load balancer — where all traffic passes one place, and keep the origin limit as a blunt backstop.

Window choice matters too. A fixed window resets on the minute, so a client can send the full allowance at 12:00:59 and again at 12:01:00, doubling the intended rate inside one second. Sliding windows and token buckets smooth that out. Cloudflare's rate limiting rules count requests over a period you define and can match on path, method and header; AWS does something similar with rate-based rules in AWS WAF, including a scope-down statement so the rule only counts the paths you care about.

How do you set up website scraping protection, step by step?

Work from measurement to enforcement. Every step below is reversible, and you should be able to undo it in minutes — a protection rule that blocks paying customers is worse than the scraper.

1. Measure first. Pull a week of access logs and find the top talkers by IP and path:

   cut -d' ' -f1 access.log | sort | uniq -c | sort -rn | head -20

   Look for one address with thousands of hits at a suspiciously even interval.
2. Pick the key and the ceiling. Per-IP for anonymous traffic, per API token or per session for logged-in users. Set the ceiling above your busiest real user, not above your average.
3. Add an edge rule. In Cloudflare or AWS WAF, create a rate-based rule that returns 429 and logs the match. This is the layer that aggregates across your whole fleet.
4. Add an origin backstop in NGINX. Declare the zone in the http context, then apply it to expensive locations only — not to images, CSS or fonts:

   # http context
   limit_req_zone $binary_remote_addr zone=perip:10m rate=10r/s;
   
   # inside the server or location block
   limit_req zone=perip burst=20 nodelay;
   limit_req_status 429;
   add_header Retry-After 30 always;

   Then check the config and reload rather than restart: nginx -t && systemctl reload nginx.
5. Allowlist verified bots and your monitors. Use a map on the client address for your own probes, and verify search crawlers by reverse DNS lookup rather than by User-Agent string.
6. Add application quotas. Authenticated endpoints should meter per key in the app, where you know who the caller is, not per IP at the edge.
7. Instrument it. Log every 429 with the key that triggered it, and alert when the 429 rate crosses a threshold you set deliberately.

Which configuration values actually matter?

Five settings decide whether a limit protects you or punishes your users: the rate, the burst allowance, whether excess requests are delayed or rejected outright, the key you count against, and what the client is told. Get burst wrong and you will block a real customer who simply opened six tabs at once.

Setting	What it controls	How it goes wrong
rate	Sustained requests per second per key	Set at your average, it blocks users during peak hour
burst	Excess requests allowed before rejection	Too low breaks normal page loads with many assets
nodelay	Whether burst requests are served at once or queued	Omitted, requests queue and p95 latency climbs
key	The identity the counter is attached to	Per-IP punishes shared NAT and mobile carriers
limit_req_status	The code returned when the bucket is empty	Returning 503 hides abuse inside your error budget
zone size	Shared memory holding the counters	Too small and state is evicted, resetting limits

How do you verify a limit works without locking out real users?

Test on staging with the same rule, hammer one endpoint, and confirm you get 429s at roughly the expected count. Then watch production for 24 hours before tightening: compare the 429 rate against successful sessions, and check that no office or carrier address shows up disproportionately.

for i in $(seq 1 60); do
  curl -s -o /dev/null -w "%{http_code}\n" https://staging.example.com/search
done | sort | uniq -c

Three checks people skip. First, confirm the CDN is not caching the 429 — a cached error served to everyone is a self-inflicted outage, so send Cache-Control: no-store on error responses. Second, run your normal monitoring against the protected endpoint so you find out whether your own health checks trip the rule. Third, keep a canary: one known-good request from a residential connection that you check daily. If your scraper detection breaks legitimate traffic, that canary tells you before your customers do. Ongoing tuning of this kind is routine work, and it is the sort of thing our website maintenance service handles for teams who do not want to own it.

What are the failure modes, and how do you debug them?

Start with the 429 rate metric, then the key. If one address is generating thousands of rejections, you either have a scraper or a shared NAT — check the ASN and whether the requests carry a session cookie. If real users report blocks, the key is wrong, not the limit.

• Shared IP addresses. A mobile carrier or a corporate office puts hundreds of users behind one address. Key on session where you can.
• Rotating residential proxies. Per-IP limits barely touch them. You need behavioural signals: request interval regularity, path coverage, missing static asset fetches.
• Headless browsers. They run JavaScript and solve simple challenges. Some leakage is normal and acceptable.
• Cached error responses. A 403 or 429 cached at the edge gets served to everyone. Always set no-store on errors.
• Counters that do not aggregate. Per-node limits multiply with fleet size, as above.
• Your own uptime checks. Allowlist monitoring explicitly, by address and by header, or you will page yourself at 3 a.m.

One more pattern worth knowing: patient scrapers run during your quiet hours precisely to stay under the limit. A rate limit alone will not catch them. You also need to watch coverage — how many distinct pages one client fetches per hour — because that is the signal a slow crawler cannot hide.

What does it cost, in money and attention?

Edge rate limiting usually sits inside a paid WAF or bot-management tier priced on requests or rule count, so confirm current figures on the vendor's own calculator before committing — those numbers move. The origin savings are real: less database work, less egress, smaller instances. The hidden cost is engineer time and false positives, which surface as lost conversions rather than alerts.

Ongoing overhead is small but never zero. Someone has to review the rejection logs, retune burst after a redesign, and remember to allowlist a new monitoring node. In practice that is a few hours a month, and it is exactly the kind of quiet maintenance work that gets skipped when a team is shipping features.

What security and privacy issues come with blocking scrapers?

You are now logging IP addresses and request patterns at a finer grain than before, which raises retention and privacy questions depending on where your users are. Keep retention short, restrict who can read those logs, and do not build a shadow profile of ordinary visitors just to catch one crawler.

Two policy points matter more than the technical ones. Accessibility: CAPTCHAs and aggressive challenges lock out screen-reader users and anyone on a slow connection, so reserve them for genuinely suspicious paths. Restraint: never block search crawlers, public archives, or a competitor doing a one-off manual check. A rule that blocks a rival's marketing intern is not a security control.

What mistakes do teams make most often?

• Blocking on User-Agent alone. Any scraper can set it to anything, and real crawler identity is verified by reverse DNS lookup.
• Setting the limit from averages instead of from the busiest real user.
• Applying the limit globally, including static assets, which breaks ordinary page loads.
• Returning 503 instead of 429, then wondering why the error budget burned.
• Forgetting health checks and monitoring probes, then getting paged by your own rule.
• Treating it as a one-off project when limits need review after every traffic-shape change.

What does this look like in a real scenario?

A booking site with live availability and a public API. Every night around 01:00 the database CPU graph spikes and query latency climbs; by 03:00 it is normal again. Nobody has complained, so it sits in the backlog for weeks.

The access logs tell the story: one network is fetching every departure date combination for the next eighteen months, spaced evenly enough to look human. The fix is unglamorous. An edge rate rule keyed on address and path with a sensible burst, a second limit on the API keyed on the token, and a weekly report of the top twenty talkers. Booking traffic is untouched, the CPU spike disappears, and the same rules catch the next crawler without anyone editing config. That pattern holds across most sites we look at — you can see the kind of builds it applies to in our trekking and travel work.

What are the alternatives to rate limiting?

Rate limiting is rarely the whole answer. Requiring authentication for expensive endpoints removes anonymous abuse entirely. Serving stale cached data for a few minutes absorbs most crawl load. Honeypot links that humans never click catch naive crawlers cheaply. Tarpits that slow responses instead of rejecting them waste a scraper's time without generating errors. Per-account quotas meter known integrators fairly.

When the data is not sensitive and the pages are static, the simpler option wins: cache harder at the edge, set a generous limit as a backstop, and spend your engineering time elsewhere. If you are unsure which of these applies to your stack, the questions we get asked most cover the common cases, and our team can help you pick the smallest control that actually holds.

In short: cap requests per identity, enforce at the edge, keep an origin backstop, verify your crawler allowlist by reverse DNS, and log every rejection. Do that and a scraper becomes an inconvenience rather than an outage — and you never have to guess why the database was busy at 2 a.m.

People also search for

• How to stop a scraper hammering my website
• What an overloaded website actually costs a business
• Should I rebuild my site or keep patching it
• Do I need a WAF if my site runs on WordPress
• NGINX limit_req burst and nodelay explained
• Blocking bots without blocking Googlebot

If you would rather not tune burst values and read access logs yourself, our team can help you design the smallest set of limits that protects your traffic — edge rules, origin backstops and alerting included. Start with a review of what you already have by getting in touch, or look through the wider services we run for clients in Nepal and abroad.