Sammy Straus posted his feed reader, sprinklz.io, to Hacker News and picked up around a hundred upvotes. Then his site slowed to a crawl, and kept getting worse several days after the post dropped off the front page. The logs told the real story: most requests never ran the JavaScript needed to render the page at all, and nearly half of all traffic came from a single service that polls sites many times per minute to generate an AI newsletter.

If you run anything public facing, this is your problem too. AI bot traffic quietly eats server capacity, distorts your analytics, and makes capacity planning guesswork. Below, we break down how Straus found the culprits, what he blocked, and the practical controls worth putting in place before your own numbers stop making sense.

The Hacker News Bump That Turned Into an AI Scraping Nightmare

For developer Sammy Straus, publishing sprinklz.io to Hacker News started as a standard validation milestone. The customizable feed reader picked up around a hundred upvotes because users resonated with its ability to deprioritize specific topics. The celebration ended quickly when the infrastructure hit severe performance degradation that defied standard operational logic. Server loads remained abnormally high days after the initial traffic spike subsided, turning a successful launch into an investigation.

Basic metric tracking revealed a glaring disconnect between active human users and total backend hits. The application is built as a single page application, yet top level endpoint logging exposed a massive flaw. A majority of incoming requests bypassed JavaScript execution entirely. Aggregated NGINX logs exposed the culprit, showing that nearly half of all requests originated from a single automated service continuously polling the site to feed an AI newsletter.

Uncovering the Culprits Behind Exorbitant Server Load

Analyzing top-level request logs and missing JavaScript execution

Straus did the unglamorous thing first. He added top-level logging for every endpoint, counting how many times each URL path was hit. That single change turned a vague “the site feels slow” complaint into a measurable pattern, which is the same move any operations leader should make before throwing money at capacity.

The pattern was obvious once the data existed. A majority of requests never even attempted to fetch news articles, which on a single-page application means they never ran the JavaScript required to render anything. Real users trigger a predictable cascade of downstream calls. These requests stopped at the front door.

That mismatch is the tell. When your request volume and your actual business activity move in opposite directions, you are not looking at demand. You are looking at machines, and no amount of extra server capacity will fix the underlying problem.

Identifying high-frequency scrapers and AI slop newsletter services

Aggregate NGINX logs narrowed it down further. One specific user accounted for nearly half of all traffic, a service that describes itself as listening for changes to a website and summarizing them into an AI newsletter. In practice, “listening for changes” meant querying the site many times per minute. Straus blocked it.

The rest of the traffic sorted into categories worth knowing by name. Claude bot, the OpenAI crawler, and Google’s search indexer all identified themselves honestly. Vulnerability scanners showed up too, probing paths like /wp-content/phpinfo.php on a site that has never run WordPress.

Those categories demand different responses, and treating them identically is how teams either break their search visibility or leave the door open:

• Search indexers: Keep them. Blocking Googlebot to reduce load is trading revenue for a cheaper hosting bill.
• Identified AI crawlers: Discourage with robots.txt if you do not want your content training models or feeding summaries.
• High-frequency pollers: Rate limit or block outright. A service hitting you every few seconds is a cost you are subsidising for someone else’s product.
• Vulnerability scanners: Not a load problem, a security problem. Straus had Claude mimic the scanner paths to confirm nothing was leaking.

The lesson for anyone running public-facing systems is that AI bot traffic is not one thing. It is four separate problems wearing the same uniform, and you cannot manage what you have not separated.

Standard Crawlers Versus Aggressive AI Bot Scrapers

Recognizing well-behaved bots like Claude, OpenAI, and Google

When you dig into your server logs during an unexpected traffic spike, you will find a distinct split between automated visitors. Some systems play by the established rules of the web, while others consume resources with zero regard for your infrastructure costs. Knowing the difference dictates how you configure your perimeter defenses without breaking your public reach.

During his investigation, Straus found that several of the traffic sources were standard known crawlers that kindly identified themselves as such. These included Claude bot, OpenAI bot, Google’s search indexer, and several others. They announce their presence honestly in the user-agent string and generally respect standard throttling rules. You want search indexers hitting your public pages so your customers can find your documentation or product announcements.

The operational fix here is straightforward configuration rather than a blanket block. By updating your robots.txt file, you can discourage most of the unwanted AI scanners while explicitly allowing Googlebot and other legitimate search-related traffic to index your site. Well-behaved systems read these directives and back off, keeping your server bandwidth reserved for actual human buyers.

However, relying solely on user-agent strings is a rookie mistake because aggressive scrapers easily spoof them. You have to monitor actual request rates alongside self-reported identities to see who is telling the truth. If a client claims to be a standard indexer but hits every endpoint fifty times a second, treat them as hostile.

Spotting opportunistic vulnerability scanners probing for WordPress paths

Traffic spikes following a public launch attract more than just polite indexers and data harvesters. Automated script kiddies and opportunistic bots scan the web constantly for known content management system vulnerabilities, looking for an easy backdoor into any exposed server they can find.

Straus noticed this exact pattern while reviewing his logs, noting that he picked up a few vulnerability scanners along the way. Specifically, automated scripts were trying to access paths like -/wp-content/phpinfo.php on his network. The fundamental absurdity of this probe is that his website is not a WordPress site, meaning the bots are firing blind shots at every IP address that flares up on Hacker News.

Leaving these automated probes unchecked creates unnecessary CPU overhead as your application generates endless four hundred and four error pages for paths that do not exist. Operations teams need to route these requests to lightweight edge firewalls or block them via NGINX rules before they ever touch application logic. Handling a missing file at the reverse proxy layer uses a fraction of the compute power required to parse it inside your main application stack.

Once Straus secured his endpoints, he used Claude to query all the vulnerability paths and mimic the scanners to make sure he was not leaking anything. That proactive validation step turns defensive security into a repeatable test rather than a blind guess. You test your own defenses against the exact payloads the wild scrapers use, verifying that your configuration actually holds up under pressure.

Practical Mitigation Steps for Unexpected Bot Floods

Straus fixed this in an afternoon with free tools and no new infrastructure spend. That matters, because the instinct when a server buckles is to add capacity. Paying for more compute to serve scrapers is the worst possible outcome.

Configuring robots.txt and blocking abusive scrapers

Start with the single worst offender, not the long tail. Straus identified one service, blocked it outright, and recovered most of his headroom immediately. Blocking by user agent or IP range at the reverse proxy layer costs nothing and takes minutes.

Then set policy for everyone else. Straus “added robots.txt to discourage most of the AI scanners, but allow google-bot and other search-related traffic.” That distinction is the whole point: you want to stay indexed by search engines while refusing to subsidize someone else’s training corpus or summarization product.

Understand the limits. robots.txt is a request, not a wall. Well-behaved crawlers honor it, aggressive ones ignore it, so pair it with rate limiting at the edge. Straus also added “reasonable” limits on top of the robots directives, which is the correct belt-and-braces approach.

• Per-IP rate limits: cap requests per minute at the proxy, with a burst allowance for real users loading assets.
• User-agent rules: deny known scraper agents explicitly, allow search indexers explicitly, challenge everything unidentified.
• Endpoint logging: keep it running permanently so the next anomaly shows up in hours, not days.

Using LLMs and automated scripts to simulate vulnerability scans

Scrapers were not the only uninvited traffic. Straus picked up vulnerability scanners probing paths like /wp-content/phpinfo.php on a site that is not a WordPress site. Those probes are automated, constant, and aimed at every public endpoint on the internet.

His response is worth copying. He “asked claude to query all the vulnerability paths and mimic the scanners to make sure I wasn’t leaking anything.” Instead of ignoring the noise, he turned the attacker’s own path list into a test suite and confirmed nothing was exposed.

This is a repeatable pattern for any operations team with public-facing systems. Pull the probed paths out of your access logs, hand the list to an LLM, have it generate and run the same requests, then review what came back. You are validating your own perimeter with free reconnaissance someone else already did for you.

Do it on a schedule. Probe patterns shift as new CVEs land, and your log file will tell you what changed before any vendor advisory reaches your inbox.

Source: sammystraus.com