Box runs scrapers 20 hours a day and every provider eventually calls it abuse

Third provider this year send me warning about "network abuse". Nobody complain about content, nobody send DMCA. Is only that box make many outbound connection all day and their monitoring not like it.

What i run is not attack, is scraper. Collect price and listing from public site. Boring. But from provider side it look like something bad, and cheap VPS company have automatic system that see 400 connection per second and suspend first, ask later.

I need host where this is normal and nobody panic. Who is running this kind of workload and where?

Depends what you're running. Concurrency and target — are you hitting a hundred sites or hammering three?

Maybe 60 different site, spread. Concurrency around 300. Not hammer one thing, is wide and slow per target.

Then the host's IP should never be the one making the requests, and this problem disappears.

Your box orchestrates. Something else egresses. Once the outbound connections leave through a proxy pool, the provider sees a machine with modest steady traffic to a handful of endpoints, which is the least interesting thing on their network. You are currently asking a hosting company to be comfortable with a traffic pattern that their abuse system is specifically built to flag, and then being surprised when it flags it.

There's a second reason to do it that nobody's mentioned: your scraper's IP is also your host's IP, so every site you touch now knows which provider you buy from, and the ones that care will block the whole range. You are burning the address you also serve from.

Is fair. I do use proxy for some target but not all, because cost. Maybe this is false economy.

FWIW there are two separate limits you'll hit and providers are honest about neither.

The one you've hit is connection rate, which is usually a template alert with a low threshold on cheap shared-CPU plans. The one you'll hit next is CPU steal, because 300 concurrent workers on a $4 box means you're a noisy neighbour and the hypervisor will throttle you long before support notices.

# if the second column is above ~5 you are not getting the CPU you paid for
vmstat 5 5 | awk 'NR>2 {print $15, $16}'   # id, st

ExtraVM has been fine for this in my experience and their abuse handling is a human rather than a script, which is most of the battle. Virtualine is the cheapest thing that hasn't fallen over for me, though I'd size up a tier from whatever looks adequate — the price is low enough that going up one is still cheap and the steal difference is real.

One thing to be careful of: the bulletproof tier is not automatically the scraping-friendly tier, and people conflate them constantly.

BPServ, AbeloHost's offshore line, BPW — what those buy you is tolerance for what's on the box when a complainant writes in. Network abuse is a different department and a different incentive: outbound connection floods cost them upstream peering relationships, which is their money rather than their reputation. Several hosts that will cheerfully ignore a lawyer's letter will suspend you in an hour for saturating a port.

Ask before you buy, in writing, and use the words "sustained outbound HTTP concurrency, roughly N connections" rather than "scraping" — one of those is a capacity question and the other one is a category they may have a policy about.

I've had a straight yes from two providers doing exactly that and a straight no from one, which is a better outcome than finding out in week three.

Ten years ago you'd rent a box, run whatever you wanted, and the only limit was the port speed you paid for. The reason that ended isn't that providers got strict, it's that everyone's monitoring got automatic and nobody reads the alerts before acting on them. You're not arguing with a policy, you're arguing with a threshold somebody set in 2019 and forgot.

"A human rather than a script" — based on how many abuse interactions? One good experience isn't a policy.

Three over about two years. Small sample, and I said "in my experience" for that reason.

Six week update.

I do the proxy thing properly, all egress now go through pool, box itself is quiet. No warning since. Cost more per month yes, but i not move server again which cost more in my time.

Also i ask new provider in writing like hexforge say, using the words about concurrency and not the word scraper. They answer in one day and say is fine up to some number. Having that email is worth something.

Anyway the answer to my own question is: is not the host, is that i was doing it from the host.