A VPS for legal web scraping and data collection

What this VPS is for

This page is about hosting a crawler that collects public data responsibly: price lists, open datasets, your own listings across marketplaces, research corpora, changelog and status pages, publicly posted job ads. A small always-on server runs the job on a schedule, from a stable IP, without tying up your laptop or your home connection.

It is deliberately not a tool for the abusive end of scraping. Bypassing a login wall, defeating a CAPTCHA, credential stuffing, or hammering a site until it falls over are out of scope and against our Acceptable Use Policy. The line is simple: collect what is public, at a rate the site can absorb, and be honest about who you are.

Crawl politely — the etiquette

Three habits keep a crawler both effective and welcome:

• Read robots.txt first. Fetch https://example.com/robots.txt, honour the Disallow rules for your User-Agent, and respect any Crawl-delay.
• Identify your bot honestly. Send a descriptive User-Agent with a contact URL or email, so a site operator who notices your traffic can reach you instead of guessing. Never impersonate a real browser or a search engine’s crawler.
• Rate-limit and back off. Add a delay between requests, cache what you have already fetched, and slow down — or stop — when you see HTTP 429 Too Many Requests or 503.

A polite Python scraper

The example below uses httpx and checks robots.txt before fetching, sends an honest User-Agent, spaces out requests, and caches responses to a local file so a re-run does not re-hit the site.

# pip install httpx
import time, json, hashlib, pathlib, urllib.robotparser
import httpx

BASE = "https://example.com"
UA = "TLDBunkerResearchBot/1.0 (+https://tldbunker.com/bot)"
CACHE = pathlib.Path("cache"); CACHE.mkdir(exist_ok=True)

# 1. Respect robots.txt for our User-Agent
rp = urllib.robotparser.RobotFileParser()
rp.set_url(f"{BASE}/robots.txt"); rp.read()

def polite_get(url: str, delay: float = 3.0) -> str | None:
    if not rp.can_fetch(UA, url):
        print("disallowed by robots.txt:", url); return None
    key = CACHE / (hashlib.sha256(url.encode()).hexdigest() + ".html")
    if key.exists():                       # 2. serve from cache, no re-fetch
        return key.read_text(encoding="utf-8")
    with httpx.Client(headers={"User-Agent": UA}, timeout=20) as c:
        r = c.get(url)
        if r.status_code in (429, 503):    # 3. back off when asked to
            print("throttled, backing off"); time.sleep(60); return None
        r.raise_for_status()
    key.write_text(r.text, encoding="utf-8")
    time.sleep(delay)                      # 4. space requests out
    return r.text

html = polite_get(f"{BASE}/public-listing")

For pages that only render with JavaScript, swap the client for Playwright. It drives a real headless browser, so it is far heavier — reserve it for pages that genuinely need it, and keep the same politeness rules:

pip install playwright
playwright install chromium   # installs the headless browser

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page(user_agent=UA)
    page.goto("https://example.com/js-rendered", wait_until="networkidle")
    print(page.title())
    browser.close()

Run it on a schedule

A crawler earns its keep by running unattended. The simplest option is cron — this runs the scraper once a day at 04:00, off-peak for many sites:

crontab -e
# m h  dom mon dow  command
0 4 * * * cd /home/scrape/job && /usr/bin/python3 run.py >> run.log 2>&1

For anything longer-running or that you want restarted on failure, a systemd timer is cleaner. Create scrape.service (a Type=oneshot unit that runs your script) and a matching scrape.timer with OnCalendar=*-*-* 04:00:00, then:

sudo systemctl enable --now scrape.timer
systemctl list-timers scrape.timer

Sizing and proxies

A plain-HTTP crawler is light: the Standard plan handles most jobs, since the work is mostly network I/O and parsing. Headless-browser scraping with Playwright is CPU- and RAM-hungry — budget more memory, or move to a larger plan if you run several browser contexts at once.

For higher-volume legitimate crawling, a common and honest pattern is a VPS plus a pool of proxies: the server runs the scheduler and logic while requests egress through proxies to spread load across IPs. This is about capacity and reliability, not evasion — you still identify your bot and respect each site’s limits. A proxy add-on may be offered for this.

Why the host being no-KYC matters here

Your crawler and its output can be commercially sensitive — the targets you watch, the datasets you build. Renting the machine with a 32-character ID and a Monero payment means the hosting account itself is not another identity record tied to your project. That is a privacy property of the host, not a licence to misbehave: the etiquette above still applies, and so does our AUP.