While testing my browser scraping stack, I sent 1,000 requests to an online store. The browser workers stayed healthy, but 193 responses were HTTP 403s, mostly challenge pages.

It’s easy to look at that result and say the site detected Playwright.

It might have picked up browser signals associated with automation. I can’t tell that from the response alone. The site also saw the IP address, how quickly I requested pages, where I started and what I visited next. My test even blocked scripts, which limited what a JavaScript challenge could do.

I’ve found it more useful to ask what the site could observe, and how those observations fit together.

Hiding in plain sight

I want the browser and session to fit among the traffic the site expects. Scrubbing every field to build a supposedly perfect fingerprint can work against that.

Changing every exposed value can create a combination that hardly any ordinary browser has. If it claims one operating system but renders text and reports graphics capabilities associated with another, those details don’t agree.

Chromium’s popularity makes it a natural starting point for this approach. Chrome alone accounted for 65.09% of worldwide desktop browser usage in Statcounter’s September 2026 figures. Add Chromium-based browsers such as Edge, Opera and Brave, and the family represents a much larger crowd than any competing desktop browser engine. Statcounter desktop browser share, Chrome and Chromium

That large population is a practical reason to build a stealth browser on Chromium, alongside compatibility and tooling. It doesn’t explain every project’s choice. Firefox-based options such as Camoufox exist too, and the site’s own traffic matters more than a global average. An unusual Chromium setup can stand out while an ordinary Firefox session fits in. Camoufox

There are three questions here:

• Can this browser or device be recognised again?

• Does this activity look automated or abusive?

• Does this site’s policy allow this particular request?

A fingerprint can help answer the first two. It doesn’t decide the third on its own. A recognisable browser can be an ordinary returning customer. A common browser configuration can still send an unacceptable volume of requests.

The W3C calls this increasing the anonymity set. When more browsers share the same observable characteristics, any one of them is less distinctive. That’s the idea behind “blending in”. It still doesn’t prove a visitor is human or guarantee access. W3C fingerprinting guidance

What a few lines of JavaScript can collect

You don’t need an elaborate obfuscated script to start observing a browser. Run this in your own page or its developer console:

const snapshot = {
  userAgent: navigator.userAgent,
  platform: navigator.platform,
  languages: [...navigator.languages],
  webdriver: navigator.webdriver,
  logicalProcessors: navigator.hardwareConcurrency,
  approximateMemoryGB: navigator.deviceMemory ?? null,
  touchPoints: navigator.maxTouchPoints,
  timezone: Intl.DateTimeFormat().resolvedOptions().timeZone,
  screen: [screen.width, screen.height, screen.colorDepth],
  viewport: [innerWidth, innerHeight, devicePixelRatio],
  finePointer: matchMedia(’(pointer: fine)’).matches,
  hover: matchMedia(’(hover: hover)’).matches
};
console.log(snapshot);

These values aren’t all exact hardware measurements. Availability varies, and browsers may reduce or standardise what they expose. Unsupported memory reporting is a missing observation, not evidence of automation.

I’d compare these values together with a normal browser on the device being claimed. The input capabilities, viewport, rendering and browser features should make sense together; one unusual field doesn’t tell me much on its own.

navigator.webdriver is an explicit automation signal. Its value depends on browser configuration and launch conditions; attaching a CDP client doesn’t universally set it to true. Removing that one observation also doesn’t remove the others. MDN: navigator.webdriver

Drawing a fingerprint with Canvas

Canvas fingerprinting asks the browser to render something and examines the result. Here’s a small example that draws text and shapes, then hashes the resulting pixels:

async function canvasSample() {
  const canvas = document.createElement(’canvas’);
  canvas.width = 320;
  canvas.height = 80;
  const ctx = canvas.getContext(’2d’);
  if (!ctx) return null;
​
  ctx.fillStyle = ‘#e8edf2’;
  ctx.fillRect(0, 0, 320, 80);
  ctx.font = ‘19px sans-serif’;
  ctx.fillStyle = ‘#245b87’;
  ctx.fillText(’Browser rendering: Aa 0123 Ω’, 8, 30);
  ctx.globalCompositeOperation = ‘multiply’;
  ctx.fillStyle = ‘rgba(220, 100, 40, 0.6)’;
  ctx.beginPath();
  ctx.arc(170, 35, 24, 0, Math.PI * 2);
  ctx.fill();
​
  const pixels = ctx.getImageData(0, 0, 320, 80).data;
  const digest = await crypto.subtle.digest(’SHA-256’, pixels);
  return Array.from(new Uint8Array(digest),
    byte => byte.toString(16).padStart(2, ‘0’)).join(’‘);
}
console.log(await canvasSample());

Use localhost or HTTPS for the Web Crypto call. This is an illustrative probe, not a commercial detector’s implementation.

Fonts, the way text is rasterised, graphics paths, browser versions and privacy protections can all affect the output. The hash gives me a compact value to compare between runs. It won’t tell me whether the browser is human, uniquely identify the device, or explain which input caused a difference.

Compare a normal desktop browser with your container, and repeat the measurement. If the hashes differ, you have something to investigate, though you still won’t know whether that difference mattered to the site that challenged you.

Graphics, fonts and audio

WebGL can expose graphics capabilities and, when available, renderer information:

const gl = document.createElement(’canvas’).getContext(’webgl’);
const info = gl?.getExtension(’WEBGL_debug_renderer_info’);
console.log(gl ? {
  vendor: info ? gl.getParameter(info.UNMASKED_VENDOR_WEBGL) : null,
  renderer: info ? gl.getParameter(info.UNMASKED_RENDERER_WEBGL) : null,
  maxTextureSize: gl.getParameter(gl.MAX_TEXTURE_SIZE),
  extensions: gl.getSupportedExtensions()
} : { webgl: false });

Privacy settings can restrict the extension, and legitimate browsers can use software rendering. Those observations need context. Changing the renderer string also leaves the graphics implementation underneath it unchanged. MDN: WebGL renderer information

Fonts provide another comparison. Without enumerating installed fonts, a script can request a candidate font and compare text widths with a fallback. Different metrics suggest that a different font rendered the text; equal metrics don’t prove absence. A short probe looks like this:

await document.fonts.ready;
const ctx = document.createElement(’canvas’).getContext(’2d’);
const sample = ‘mmmmWWWW0123456789’;
for (const font of [’monospace’, ‘Arial, monospace’, ‘Georgia, monospace’]) {
  ctx.font = `24px ${font}`;
  console.log(font, ctx.measureText(sample).width);
}

Audio fingerprinting works much the same way. An OfflineAudioContext can render a short oscillator through an audio-processing graph into a buffer without playing anything through the speakers. A script can inspect the samples to compare how browsers process the audio. This doesn’t directly reveal your microphone or identify a person. MDN: OfflineAudioContext

These measurements overlap, so you can’t treat each one as an independent vote and add them up. Real people can also have rare combinations. Treating every unusual setup as a bot would block some of them too.

Some of the decision happens before page JavaScript

The server sees the source IP of the connection reaching it, request headers, and transport behaviour. It can associate that IP with a network and previous traffic. A proxy changes part of that view; it doesn’t give the entire session a clean history.

TLS and HTTP behaviour can be fingerprinted too. Copying Chrome’s headers into a Python HTTP client doesn’t make its handshake match Chrome’s. With Playwright driving Chromium, browser requests normally use Chromium’s network stack. Writing the controlling script in Python doesn’t turn those into Python TLS handshakes.

Cloudflare describes combining transport fingerprints with aggregate request signals in its JA4 write-up. It shows how much more a detector can consider than page JavaScript, though it doesn’t tell me which signals caused my failures. Cloudflare: JA4 and inter-request signals

The page you visited first matters too

Think about two sessions. One arrives on a product page from a search result, follows a related category, then opens another product. Another starts with fresh storage and visits hundreds of deep product URLs at a steady rate.

Both sessions could report the same browser properties, but they’re using the site differently.

A site can associate requests it receives with cookies, accounts and other session signals. Scripts can report their current page and events. The referrer may give information about the preceding navigation, subject to browser policy. This doesn’t grant the site a list of everything you’ve visited elsewhere. MDN: document.referrer

People arrive at deep URLs through search, bookmarks and shared links all the time. Starting there doesn’t prove automation, and visiting the homepage first won’t fix every blocked session. I’d look at the sequence of requests and how it compares with ordinary use of that part of the site.

Mouse speed, clicks and the journey between them

Browser scripts can observe pointer positions and timestamps, clicks, scrolling, focus and visibility. This local demonstration records a bounded sample and prints it when you call the returned stop function. It sends nothing to a server:

function inspectInteractions() {
  const events = [];
  const controller = new AbortController();
  const started = performance.now();
  let previous = null;
  const record = value => {
    if (events.length < 100) events.push(value);
  };
​
  addEventListener(’pointermove’, event => {
    const now = performance.now();
    const dt = previous ? now - previous.time : 0;
    record({
      type: ‘move’, ms: Math.round(now - started),
      pointer: event.pointerType,
      pixelsPerSecond: dt > 0 ? Math.round(1000 * Math.hypot(
        event.clientX - previous.x, event.clientY - previous.y
      ) / dt) : null
    });
    previous = { x: event.clientX, y: event.clientY, time: now };
  }, { signal: controller.signal, passive: true });
​
  addEventListener(’click’, event => record({
    type: ‘click’, ms: Math.round(performance.now() - started),
    element: event.target instanceof Element ? event.target.tagName : null
  }), { signal: controller.signal });
​
  return () => {
    controller.abort();
    console.table(events);
  };
}
const stopInspection = inspectInteractions();
// Interact with this page, then call stopInspection().

The script calculates mouse speed, but I haven’t shown that any vendor uses this formula or a particular “human speed” threshold. Event coalescing, device type and browser scheduling affect the samples. People also use keyboards, touchscreens and assistive technology, so a visitor who doesn’t move a mouse isn’t necessarily a bot. MDN: PointerEvent

What interests me is the broader sequence: how quickly pages are requested, which links are followed, whether dozens of sessions move in lockstep, and whether activity continues unchanged through errors and challenges. Adding random mouse movements doesn’t make those patterns disappear.

A plausible fingerprint still doesn’t guarantee access

Cloudflare documents combining headers, browser signals and session characteristics in its bot scoring, alongside multiple detection engines. It also describes a cookie used to incorporate request patterns over a session. This is one provider’s documented design, not a universal recipe for every protected website. Cloudflare detection engines

The site’s configured rules still determine what action to take. Challenge, block and allow decisions can vary by route, rate, account state and other security policies. A challenge page alone doesn’t identify the rule or observation behind it.

The order matters here. The browser has to receive and run a script before its observations are available. Cloudflare explicitly warns against requiring its JavaScript detection result on a visitor’s first request. Any decision on that request has to use what the system knows so far. Cloudflare JavaScript detections

What I would measure next

I’d compare an ordinary browser and the automated browser on a page I control, recording the environment and the exact browser versions. Then I’d change one factor at a time: network, browser configuration, session reuse, resource loading, or request rate.

The JavaScript examples above are observation tools. They don’t reproduce a commercial risk model, and passing them wouldn’t certify a browser as undetectable. Before publishing measured comparisons, I still need to run and record those demonstrations.

I want to know what changed between the working session and the failing one. That could be the browser properties, IP, request pattern or missing session state. The site may also have a rule I can’t identify from its response. Comparing those factors separately should give me more to work with than another unexplained challenge page.

In my last post, I wrote about building a self-hosted browser scraping service around CloakBrowser, Playwright, Xvfb, Docker, and an authenticated proxy.

That version worked. One browser process could run multiple isolated contexts, and each context could have its own cookies, storage, and session state.

The catch was that every context still shared the same browser process.

The question became: how many contexts should I run in one browser, and when should I start another browser?

The answer was sooner than I expected.

The first version had one large failure domain

The original setup looked roughly like this:

Scraping script
      |
      v
Playwright server
      |
      v
One CloakBrowser process
      |
      v
Multiple browser contexts
      |
      v
Authenticated proxy

A single browser with several contexts is much cheaper than starting a new browser for every URL. Each context has its own cookies, local storage, and browsing state, which is what I want for most scraping work.

But the contexts still share CPU, memory, renderer processes, and the network stack. If the browser becomes unhealthy, all of those contexts are affected.

There is also a scheduling problem. A CDP connection belongs to one browser process. Adding contexts creates more work inside that process, but it does not spread the work across the machine.

I needed browser processes to be the unit of capacity.

The new architecture

The new version runs several browser replicas behind one CDP endpoint.

Playwright or Puppeteer clients
              |
              v
       CDP broker
       /    |    \
      v     v     v
 Browser Browser Browser ...
 worker 1  worker 2  worker 3
      \     |     /
       Browser contexts

The default configuration runs four browser workers.

The browser ports are private to the Docker network. Clients connect only to the broker, which is exposed on port 3000.

The client still connects like this:

browser = await playwright.chromium.connect_over_cdp(
    “http://192.168.68.62:3000”
)

The client does not need to know which browser process it is using.

What the broker does

The broker resolves the browser replicas through Docker DNS and checks their CDP discovery endpoints every two seconds.

When a client opens a new CDP connection, the broker chooses a healthy worker with available capacity. It prefers the worker with the fewest active connections, then the one least recently assigned.

The connection stays with that worker until it closes. The broker does not move individual contexts between browsers, so a batch using one CDP connection stays inside one browser process.

The default limits are:

When all workers are busy, a new connection waits in memory. Closing an existing connection wakes a waiting client.

If the queue fills or the timeout expires, the broker returns HTTP 503. A client gets a definite failure instead of waiting forever.

The broker enforces the connection limit, but it does not count or limit the contexts a client creates. My scrapers use ten contexts per connection. Another client could create more, so the current capacity model depends on clients following that convention.

CDP made the networking awkward

CDP is a WebSocket protocol, but clients also expect normal HTTP discovery endpoints.

The broker handles both:

GET /json/version
GET /devtools/browser

The first returns the public WebSocket URL. The second opens the long-lived connection that carries browser commands.

CloakBrowser binds its CDP server to loopback inside the worker container. That keeps the browser endpoint private, but it also means the broker cannot connect to it directly over the Docker network.

I added a small CDP gateway inside each worker. It exposes the loopback-only browser endpoint to the container network and rewrites the discovery response so clients receive the broker’s public address rather than an internal 127.0.0.1 address.

The client sees one endpoint. Internally, there are several browser processes and several proxy layers.

Proxy routing stays inside each worker

Each browser worker uses a local proxy:

CloakBrowser
    |
    v
127.0.0.1:3128
    |
    v
Authenticated upstream proxy

The local adapter reads the PROXY environment variable, adds the upstream Basic authentication header, and forwards HTTP and CONNECT traffic.

The browser only sees

http://127.0.0.1:3128

. Proxy credentials do not appear in the browser command line or CDP responses.

If the proxy variable is missing, malformed, or uses the wrong scheme, the adapter exits. The browser then fails instead of silently making direct requests.

This is safer than treating proxy routing as a best-effort setting.

It is not a complete security boundary yet. Kernel-level egress filtering is not implemented, so a deliberately modified process could still try to bypass the adapter. That is on the hardening list.

The real test

A synthetic test can tell me whether the queue works. It cannot tell me how the whole system behaves against a real site.

For the real test, I used CCL Online.

The test discovered URLs from CCL’s public sitemap, excluded paths disallowed by robots.txt, and navigated to 1,000 URLs through the browser service.

I used:

• Four CDP connections

• Ten contexts per connection

• Forty concurrent contexts

• A 45-second navigation timeout

• domcontentloaded navigation

• Images, fonts, media, scripts, and stylesheets blocked

• Product-page validation using the returned HTML

The test recorded each URL, status code, elapsed time, retry count, title, and error. It also polled the broker during the run.

The results were:

The service stayed up for the entire run. All four workers remained healthy, and no CDP connection waited in the broker queue.

The 403s were from CCL

The 193 HTTP 403 responses were mostly Cloudflare challenge pages. Their page title was:

Just a moment...

The broker did not run out of workers. The browser processes did not disappear. The queue did not fill. CCL accepted many requests and rejected others at its edge protection layer.

The failures also clustered around category and deals pages. Product pages performed better than several of those listing pages.

The test measured two separate things:

1. How quickly my service could process concurrent browser navigations.

2. How the target site responded to that level of traffic through the configured proxy.

The service processed 1,000 navigation attempts at 6.6 pages per second. It returned usable HTTP 200 pages for 801 of them.

I would not describe 6.6 pages per second as a universal scraping rate. It is the result for this machine, this proxy, this browser build, this concurrency level, and this target site.

The browser and the target had different limits

The useful result was not the throughput number by itself.

The broker handled forty active contexts without queueing. The four browser workers stayed healthy. The internal scheduling worked.

CCL’s Cloudflare layer became the limiting factor before the browser pool did.

Adding more replicas would not automatically improve the result. It might increase local capacity while also increasing the number of rejected requests.

There is a difference between sending more browser work and completing more useful scraping work.

The next version of the test should vary the concurrency and measure:

• Successful pages per second

• 403 rate

• p95 navigation latency

• Browser memory

• CPU usage

• Proxy behaviour

• Results by page type

A larger browser pool is useful only when the rest of the system can absorb it.

What I would try to reduce challenge pages

The first thing I’d revisit is resource blocking. This test blocked scripts as well as images, fonts, media, and stylesheets. That saved bandwidth, but it also meant a JavaScript-based challenge had no opportunity to run. The results show which navigations returned challenge pages under those settings; they don’t establish how many would still fail with scripts allowed.

I’d compare the current settings with scripts enabled, then with normal resource loading. I’d also give the page a bounded amount of time to reach a recognisable product or category page, rather than treating the first domcontentloaded event as the final outcome. I’d record both the initial response and the eventual page state. A page that reaches the content after a short wait is a different result from one that remains on a challenge or requires interaction.

Separate concurrency from request rate

Forty contexts tells me how much work can be in flight. It doesn’t set a predictable request rate: fast pages let each context request the next URL sooner.

I’d start with low concurrency and a shared per-domain rate limit across all workers, then increase the load in steps. If challenges increase, the scraper should slow down and allow a cooldown instead of immediately adding more retries. The comparison would be validated pages per minute, challenge rate, and bandwidth. A slower request rate could still finish the useful work sooner if fewer requests fail.

Check what identity each session actually keeps

The test already reuses a context and page across multiple URLs, so it retains cookies within that context. What I haven’t established is how stable the proxy exit IP is across those requests. A persistent context doesn’t guarantee a persistent network identity.

I’d compare the current proxy configuration with a session-sticky configuration, where available, keeping the browser context and exit IP together for a bounded batch. I’d measure the exit address before and after each test batch rather than assume the provider’s rotation behaviour. I’d also check that locale, timezone, and any browser overrides are consistent with the intended session. These are hypotheses to test, not a promise that a consistent session will avoid challenges.

Compare like-for-like pages

The failures clustered around category and deals pages, so an overall success rate hides useful differences. I’d build a fixed sample of product, category, and deals URLs and report each group separately.

For each experiment I’d change one variable, repeat the runs, and alternate their order with cooldowns. Otherwise a change in proxy reputation, time of day, or accumulated traffic could look like an improvement caused by the code.

Try different proxies and proxy types

Once I have a lower-load baseline, I’d compare datacenter, ISP, residential, and mobile proxies, and try more than one provider. I wouldn’t assume the more expensive option would produce better results. The question is which pool works reliably for this target at the request rate I need.

I’d keep the browser settings, URL sample, request rate, and exit country consistent where possible. For each option I’d compare sticky sessions with rotation between batches, while checking the actual exit IPs. Rotating on every request would be a separate experiment, because it changes session consistency as well as the IP pool.

The results should include challenge rate, validated pages per minute, latency, connection failures, bandwidth, and cost per usable record. I’d sample multiple exit IPs from each pool so one unusually good or bad address doesn’t decide the comparison. A cheaper proxy with more challenges might still be economical at a lower rate; an expensive one would need to earn its place through better usable output. These tests would establish what works for CCL under those conditions, rather than a universal ranking of proxy types.

Count the content, not just the status

The current test checks for an h1 on HTTP 200 pages. That’s a useful sanity check, but it isn’t enough to prove the expected data arrived. I’d add page-type checks for product names, prices, or category listings, and classify persistent challenges, explicit denials, missing content, and navigation failures separately.

That would give me a better target than simply reducing 403s: more correct records per minute at an acceptable proxy cost. I’d start with scripts enabled and lower request rates, because those are two concrete limitations of this test. I haven’t run these comparisons yet, so I can’t say which change would improve the result.

What I would change next

The next tests need to cover failures and longer workloads. A short benchmark tells me whether the service can complete that run. It doesn’t tell me what happens after several days of scraping, or when a client disappears halfway through a batch.

Enforce browser capacity

The first change would be enforcing the context limit. Right now, one connection gets one worker, whether that client opens one context or forty. The broker cannot see that difference in its allocation counters. I’d like to track context creation and disposal, cap the number per connection, and test several scrapers competing for capacity.

Test cleanup and interrupted work

The broker releases its connection slot when the WebSocket handler exits. I still need to kill a client abruptly and check what happens to its pages and contexts, rather than relying on the scraper’s normal cleanup path.

I’d also restart a browser worker, the broker, and the whole machine with work in progress. Discovering a replacement worker restores capacity; it doesn’t recover the pages that were being scraped. The broker’s waiting connections live in memory, and it doesn’t store scraping jobs. Resuming interrupted work, recording partial results, and handling duplicates need to be designed into the scraper or a separate job queue.

Make health checks prove more

The current health check asks the browser’s /json/version endpoint for its WebSocket address. That proves CDP discovery is responding. It doesn’t prove the worker can create a context, navigate through the proxy, or return a usable page.

I’d add a small, controlled navigation check and record browser failures separately from proxy errors and target-site blocks. That would make the word “healthy” more useful when investigating a failed run.

Measure sustained use and useful output

The broker exposes snapshots through /health, /workers, and /metrics, but I want a history of queue waits, active contexts, memory, CPU usage, and navigation latency during a sustained workload. The Compose configuration sets shared memory for Chromium, but it doesn’t set total-memory or CPU limits, and its process limit is explicitly unlimited. I’d measure normal usage before choosing those limits and a policy for recycling workers.

I also want to test one, two, three, and four replicas against the same workload. Alongside successful pages per second and the 403 rate, I’d measure proxy bandwidth, retries, and cost per validated record. Four workers worked here, but the useful question is how much correct data they produce for the time and money spent.

Harden the containers

I want network-level egress controls that prevent traffic from bypassing the proxy adapter. I also want to replace the current SYS_ADMIN and --no-sandbox setup with a non-root browser user and Chromium sandboxing.

Those are the changes I’d make before giving the service more responsibility. The existing tests establish a working browser pool; the next round needs to establish how it behaves when clients fail, workloads compete, and the service stays busy for longer.

Summary

It’s already been useful for prototyping and building test scrapers. Having a browser service that’s always up, with decent stealth and proxy routing already configured, means I can connect a new script and start testing without rebuilding that setup for every project.

I now have four browser workers behind one endpoint, with connection limits, queueing, and worker discovery. In the CCL test, they handled 1,000 navigation attempts in about two and a half minutes and stayed healthy throughout. But 193 responses were challenge pages, so the throughput figure alone doesn’t tell me how useful the service is.

The next experiments are about getting more usable data from that capacity: allow scripts, control the request rate, check session consistency, and compare proxy providers and types. Alongside that, I need to test interrupted jobs, enforce context limits, and measure longer-running workloads. The number I want to improve is correct records per minute at a cost I’m willing to pay.

This is what I learned.

The hardware

I’m running this on a HP ProDesk 405 G6 — a small form factor desktop with a Ryzen 4650G and 32GB RAM sitting on a shelf floor (!) at home. It’s not a powerful machine by any measure, but it’s x86_64 Linux, always on, and draws very little power. For a persistent browser service it’s more than enough.

The control layer is not the browser

The first thing worth understanding clearly: Playwright, Puppeteer, and Selenium are control libraries. They are not browsers. Playwright speaks CDP (Chrome DevTools Protocol) to whatever binary you point it at. The binary and the control library are completely separate concerns.

This matters for architecture. When you call playwright.launch(), it starts a browser process locally and connects to it. When you call playwright.connect("ws://..."), you’re connecting to a browser process running somewhere else — the control library stays local, the binary runs remotely. The browser itself has no idea which one you used. From the page’s perspective, the experience is identical either way.

# local — Playwright launches the browser itself
browser = await p.chromium.launch()

# remote — Playwright connects to a browser running elsewhere
browser = await p.chromium.connect("ws://192.168.1.100:3000")

# from here, the API is identical
context = await browser.new_context()
page = await context.new_page()
await page.goto("https://example.com")

The design decision this enables: separate the browser from the code that drives it. Run a persistent Playwright server on a dedicated machine, expose it as a WebSocket endpoint, and write scraping scripts that connect to it as clients. Multiple scripts, multiple projects, one browser service. That’s the architecture.

Why the binary matters

When a browser is put into automation mode — which happens the moment Playwright launches or connects to it — it’s supposed to advertise that fact. navigator.webdriver = true is in the W3C WebDriver spec. It’s not a side effect of Playwright doing something unusual. The spec requires it.

There are three layers at which automation can be detected:

JavaScript-visible properties. navigator.webdriver, the shape of navigator.plugins, the presence or absence of window.chrome. These can be overridden in JS before the page loads, but the overrides are themselves detectable. The property descriptor is wrong. The prototype chain doesn’t match what a native property would look like. Antibot systems check for exactly this.

Binary-level signals. Internal automation flags, the CDP debugging port being open on localhost. Pages can probe open ports via timing differences in connection failures — a connection to 127.0.0.1:9222 that’s refused immediately is different from one that times out.

Network fingerprints. TLS handshake characteristics and HTTP/2 settings are determined by the browser’s network stack, compiled in. You cannot change them from JS, from headers, or from Playwright settings. They require recompiling the browser.

The patching hierarchy follows from this: JS override is detectable. Binary-level patch is not, because there’s nothing overriding anything — the signal was never emitted. Projects like CloakBrowser (Chromium) and Camoufox (Firefox) work at the source level precisely because that’s the only level where you can actually remove the signals rather than mask them.

The practical implication: for serious work, the choice of binary matters as much as the choice of control library. I went with CloakBrowser.

Getting it into the container means installing Playwright’s own Chromium first to create the versioned slot directories, then replacing the binaries:

# Install Playwright's Chromium to create the versioned slot directories
# then replace both with CloakBrowser
RUN npx playwright install chromium \
    && SLOT=$(ls /root/.cache/ms-playwright/ | grep '^chromium-') \
    && CHROME_DIR="/root/.cache/ms-playwright/$SLOT/chrome-linux64" \
    && rm -rf "$CHROME_DIR" \
    && mkdir -p "$CHROME_DIR" \
    && cp -r /browsers/chromium/. "$CHROME_DIR/" \
    && chmod +x "$CHROME_DIR/chrome" \
    && HS_SLOT=$(ls /root/.cache/ms-playwright/ | grep '^chromium_headless_shell-') \
    && HS_DIR="/root/.cache/ms-playwright/$HS_SLOT/chrome-headless-shell-linux64" \
    && rm -rf "$HS_DIR" \
    && mkdir -p "$HS_DIR" \
    && cp -r /browsers/chromium/. "$HS_DIR/" \
    && mv "$HS_DIR/chrome" "$HS_DIR/chrome-headless-shell" \
    && chmod +x "$HS_DIR/chrome-headless-shell"

Why two slots? That’s the thing that took the longest to figure out — more on that below.

Headed, not headless

The default for automated browsers is headless — no display, no window, no rendering overhead. It’s also one of the more reliable detection signals. The HeadlessChrome string in the User-Agent, differences in canvas rendering, different screen property values, different WebGL capabilities. Headless browsers have a distinct fingerprint that antibot vendors have studied for years.

The alternative is running headed but without a physical display, using Xvfb (X Virtual Framebuffer). The browser renders into a virtual display. From the inside, nothing about it looks headless — because it isn’t. The rendering pipeline, the screen dimensions, the font stack all behave as they would on a real desktop.

The Dockerfile sets this up:

# Start Xvfb on display :99 with a 1920x1080 24-bit framebuffer
Xvfb :99 -screen 0 1920x1080x24 -ac

# Browser processes use that display
ENV DISPLAY=:99

The tradeoff is slightly higher resource usage. For a dedicated scraping machine it doesn’t matter. The decision is simple: headed via Xvfb is strictly better for evasion. The only reason to use headless is convenience.

The output of the code I generated to test it.

Why supervisord inside Docker rather than a simpler setup

Docker gives you portability and reproducibility. The browser setup is complex enough — Xvfb, Playwright server, patched binary, runtime dependencies, font packages — that managing it by hand on each machine doesn’t make sense. One Dockerfile, one build, same result anywhere.

The wrinkle is that a browser service is not a single process. It’s at minimum two: Xvfb and the Playwright server. Docker’s default assumption is one process per container. You need a process supervisor to manage startup ordering and handle crashes cleanly.

[program:xvfb]
command=Xvfb :99 -screen 0 1920x1080x24 -ac
autorestart=true
priority=10          ; starts first

[program:playwright]
command=/start-playwright.sh
autorestart=true
priority=20          ; starts after xvfb
startsecs=3          ; give xvfb time to be ready

Supervisord handles startup ordering via priority values and restarts processes that crash. If Xvfb goes down, the Playwright server goes down with it — supervisord brings them back up in the right order.

One detail worth noting: getting environment variables to propagate correctly through supervisord into child processes is more fragile than it looks. The Playwright startup environment variables live in a shell script (start-playwright.sh) rather than supervisord’s environment directive, because the script is explicit about what’s set and reliable in a way the directive wasn’t.

The thing that actually took the longest

Not the architecture and not the detection evasion. The most time-consuming part was a single discovery: Playwright maintains two separate Chromium slot directories, which caught me out and made me see I wasn’t running headful and why I was getting 403s.

chromium-1223/chrome-linux64/ — the full Chromium binary.
chromium_headless_shell-1223/chrome-headless-shell-linux64/ — a stripped headless-only build.

When running headless, Playwright prefers the headless shell. I’d replaced the full Chromium slot with CloakBrowser. Playwright was ignoring it and launching the headless shell from the untouched slot. The version reported was 148 (Playwright’s bundled build), not 146 (CloakBrowser). Nothing in the error output explained this. I found it by watching ps aux during an active connection and reading the full binary path in the process arguments.

# what I expected to see
/root/.cache/ms-playwright/chromium-1223/chrome-linux64/chrome

# what was actually running
/root/.cache/ms-playwright/chromium_headless_shell-1223/chrome-headless-shell-linux64/chrome-headless-shell

Fix: replace both slots. The headless shell slot needs its binary renamed to chrome-headless-shell to match what Playwright expects. This is documented nowhere I could find.

Concurrency: contexts not instances

A single browser instance runs multiple isolated contexts. Each context has separate cookies, separate storage, separate session state — effectively separate browser profiles sharing one process. For most scraping work, one instance with a pool of contexts is the right model.

The async queue pattern works well. Workers pull URLs from the queue, create a context, scrape, close the context, move on. A 403 doesn’t block the worker — it requeues the URL with a backoff delay and immediately picks up the next job.

async def worker(worker_id, browser, queue, results):
    while True:
        url, attempt = queue.get_nowait()
        result, should_retry = await scrape(browser, url)

        if result:
            results.append(result)
        elif should_retry and attempt < MAX_RETRIES:
            await asyncio.sleep(2 * attempt)
            await queue.put((url, attempt + 1))

        queue.task_done()

# spin up N workers against the same browser instance
workers = [
    asyncio.create_task(worker(i, browser, queue, results))
    for i in range(CONCURRENCY)
]
await queue.join()

At 16 concurrent contexts on the ProDesk the machine copes fine. Proxy credentials go in new_context() per context — residential proxies with rotating IPs for each context and retries in my script, meant any bad IPs that failed or got blocked just retried with a new context and new IP.

A full connection looks like this:

async with async_playwright() as p:
    browser = await p.chromium.connect("ws://192.168.1.100:3000")

    context = await browser.new_context(
        proxy={
            "server": "http://proxy-provider:port",
            "username": "your-username",
            "password": "your-password",
        },
        locale="en-GB",
        timezone_id="Europe/London",
        viewport={"width": 1920, "height": 1080},
        user_agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
                   "(KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36",
    )

    # block resources you don't need — cuts proxy bandwidth significantly
    await context.route(
        "**/*",
        lambda route: route.abort()
        if route.request.resource_type in ("image", "media", "font", "stylesheet")
        else route.continue_()
    )

    page = await context.new_page()
    await page.goto("https://example.com", wait_until="domcontentloaded")
    # do your scraping
    await context.close()

Scrapy integration

The natural next step for me is wiring this into Scrapy via scrapy-playwright, which supports remote browser connections via PLAYWRIGHT_CONNECT_URL. The connection works but the problem is proxy auth — Chromium doesn’t correctly apply proxy credentials passed via context kwargs when scrapy-playwright manages contexts on a remote connection. The workaround is a local CONNECT proxy inside the container that accepts connections from the browser and injects upstream credentials without SSL interception. It works, but it’s a workaround for something that shouldn’t need one.

TODO: Resolve this properly and document a clean Scrapy setup.

What’s next

One machine, one endpoint and one pool of contexts so the obvious scaling step is multiple machines. Docker Swarm or similar, multiple browser server nodes behind a single endpoint, context creation distributed across the pool.

TODO: Multi-node browser distribution via Docker Swarm.

The repo is at https://github.com/jhnwr/browser-service. CloakBrowser needs to be sourced from their releases page — the binary isn’t something I can redistribute. The Dockerfile handles everything else, including replacing both Playwright Chromium slots with the patched build.

It’s not complicated. You create a folder, initialise the project, get your settings right, install the right packages, build out page objects, wire up your items. Every time. From scratch. And if you’re building against multiple domains — which is common — you do most of it again. Nothing about it requires thought. It’s just time, and repetition, and an open invitation for small mistakes to creep in.

I wanted to see if AI could reliably do this, and do it efficiently.

That was about 6 months ago. Since then I’ve built several MCP servers (not good for project building), what feels like hundreds of CLI tools, a set of agent skills that scaffold full Scrapy projects in Claude, and I’m currently testing OpenSpec — a project specification framework that guides the AI through the entire build from scratch. This post is what I’ve learned. Not a polished success story. More of an honest log of what worked, what broke, and what I’m still figuring out.

The first attempt was a mess

The obvious thing to try first is just... asking. Open a new Claude conversation, describe what you want, and see what comes back.

What came back was not great.

The problem is that the AI is working from documentation. It knows the Scrapy docs, it knows the scrapy-poet docs, and it can produce code that looks like it belongs in a tutorial. But it does not know how I build spiders. It doesn’t understand the page object pattern the way I use it. Specifically, it kept trying to inject page objects incorrectly — putting logic in the wrong place, using wrong imports, handling custom items and returns in ways that would never run cleanly. Every time I fed it back the error, it would try to fix it, make a different mistake, and produce something worse than what it started with.

That death spiral — error, self-correction attempt, deeper error — is the defining failure mode of unstructured AI code generation. And the fix is not better prompting. The fix is structure.

Building the skill stack

My first structured approach was to build a set of agent skills — small, specialised instruction sets that each handle one part of the build process. The idea was to break the project into logical units and give the AI a tight brief for each one.

Getting the granularity right was harder than I expected.

A skill that tries to do too much loses its focus. Early on I tried combining project scaffolding with dependency installation and settings configuration all in one. It mostly worked, but the more decisions the skill had to make, the more it started cutting corners or making assumptions I hadn’t approved. A skill that tries to do too little produces too little — you end up with a lot of steps that don’t add up to a working project.

What I landed on was six skills:

1. Create the project/install dependencies

2. Build the spider

3. Create the item

4. Create the base of the page objects

5. Generate the page object code

6. Wire everything together

Each one maps to a stage I would naturally think about when building a spider by hand. That’s the test I’d apply to anyone trying to do something similar: if you can’t describe the skill as a step in your own mental model of the task, it’s probably either too big or misaligned with how the work actually flows.

One other thing I learned: some tasks belong in a script, not a skill. The clearest example is HTML cleanup. I have specific things I want to strip from the page before extraction — and I cannot let the AI decide which ones. If it gets that wrong, data disappears and nothing in the pipeline tells you why. So I wrote a Python script that handles it deterministically. The skill calls the script. The AI never makes that call itself. I tried doing the same thing for schema extraction at one point, pulling structured data out of JSON-LD and similar, but it ended up being one layer of abstraction too many. The script approach is only worth it when the decision about what to do is too important to leave to inference.

Order mattered

One of my early mistakes nearly convinced me the whole approach wasn’t going to work.

I hadn’t specified the order in which the project should be built. I’d listed everything that needed to happen, but not when. The AI tried to create page objects before the actual project existed. Which means it couldn’t use the Scrapy downloader, couldn’t get access to any pages, and had no project structure to put the page objects into. It tried to fix the errors, failed, and looped.

The fix was obvious in hindsight: if you’re encoding how to build something, encode the sequence. Not just the steps. Scrapy projects I build have a strict execution order — project first, then spider, then items, then page objects, then wiring. That order has to be explicit, because the AI will not infer it reliably from context alone.

For skills this was an orchestrator who controlled everyting, and for the spec driven development it was including the order in a simple manner.

Where I’m heading: OpenSpec

The skills approach works. But I’ve been testing something I think will work better.

The difference is conceptual. Skills handle specific tasks. OpenSpec is a framework for a set of guidelines for how the whole project should be built — closer to a reference document than a set of instructions. It encodes the gotchas, the order, the patterns I want, the decisions I’ve already made. When I say “do this, not that,” it goes in the spec. When I say “when you’re doing X, make sure Y comes first,” that goes in the spec. As models get better, I feel like specs will start to gain the edge even more.

The advantage over skills is that you can copy the spec into any new project folder and the AI has everything it needs to build the project in my way. You’re not threading together a chain of skill calls. You’re handing the AI a complete picture of what a good project looks like and asking it to produce one.

I’m still tweaking it. Different models interpret the same instructions differently — I’ve been testing with Claude Sonnet 4.6 and I haven’t tried GPT-5.4 or 5.5 yet, which would be an interesting comparison. Some rules that work perfectly with one model need to be more explicit with another. So the spec is not done; it’s a living document.

What the workflow looks like today

Right now, starting a new Scrapy project looks like this:

1. Create a new project folder

2. uv init, then install the Scrapy packages I need

3. Copy the spec into the folder

4. Prompt: “Create a Scrapy project with custom items. You decide the fields. Here’s the product URL, here’s the list page URL.”

That’s it. Sometimes I’ll add: “Also create an SQLite pipeline and save items to the database.” SQLite is just cleaner than JSON files accumulating everywhere, and it’s easy to query and inspect.

The thing that surprised me most — and it’s a modest surprise — was how well adding new domains worked. When a second domain uses the same item schema, I can drop it into the existing project without rebuilding anything. The AI handles it cleanly, and that near-freebie was one of the genuine wins in the whole process. Running multiple spiders across multiple domains from a single project, without repeating the setup work, was the original goal.

What nobody talks about

There’s a lot of conversation right now about the perfect agentic workflow. How to chain tools together, how to get the AI to work more autonomously, how to reduce the amount of human intervention in any given process, and how to delegate to subagents.

What I’d push back on slightly is the idea that rigid rules are what you’re building toward.

Web scraping is not a stable target. Websites change. Anti-bot systems evolve. The structure of a page that worked last month might be completely different next month. A scaffold that’s too rigid will produce projects that need constant manual patching once they hit the real world. What you actually want is rules that are rigid in the right places — the architecture, the patterns, the execution order — and flexible in the places where the domain demands it.

That balance is harder to get right than just making the AI do more. And I don’t think enough people who are building AI workflows for technical work are talking about it honestly.

I’m going to keep iterating on the spec and will share a more detailed breakdown once it’s further along. If you’re building something similar — or tried it and hit different walls — I’d be interested to hear what you ran into.

John Rooney is Head of Developer Advocacy at Zyte. He writes about web scraping, AI tooling, and developer workflows at JHNWR on Substack.

However, the challenges I face now are vastly different to those I used to see, even just a few years ago. We are all aware of the changes that AI brings, and I feel it very acutely and this is represented in the work I do and what I spend my time thinking about and learning.

The difficulties in web scraping are generally the same, regardless of who you are and what you are trying to achieve - the solo developer who needs data for their non profit faces the same antibot protection, the same browser fingerprinting and techniques and the same parsing woes and maintanence issues that the largest of businesses do, except they see it on an entirely different scale.

At the lower end of the request count, in the early thousands or 10’s of thousands, these issues don’t represent as much of a time sink. It’s feasible to manually fix your scraping code when something changes, and update your stealth browser when it starts to get detected.

Yet when scaled both vertically to millions of requests, and horizontally, to hundreds of websites, these common issues are amplified to a point where it’s nearly impossible to manage without significant investment in expertise and time. And it’s this issue that I find more and more interesting to me and what problems I want to solve.

It makes sense really. The cutting edge AI tooling provides a whole world of possibilities to solving these issues and as a relatively new way of working, the enjoyment for making headway in this space is much more real than solving the same things over and over.

But bypassing antibot protection and working out how to scrape data from websites is still a real issue. And a very real one to those who are working on a small or very limited budget. But the thing for me is, and I know working for Zyte has coloured my perception to this somewhat and that needs to be taken into account, I just find that generally speaking this is a solved problem - let me explain.

As I’ve spent more time with people who are literal experts in antibot tech, and the more I’ve seen of how to get access the more that problem moves away from the space I want to work in.

Without wanting to downplay it, gaining access programmatically to websites to scrape data is generally easy, until you scale up. You either need a browser or you don’t. And if you do, you need smart session management paired with good IPs. It has become much more now a pay to play experience, with the infrastructure costs growing and the management time for these things increasing.

This isn’t to say there aren’t real and interesting problems to solve - of course there are, but it just isn’t a world I find myself in, nor one I want to exist in any more than the periphery of (no offense to my colleagues who keep our product going, they are incredible).

All of this only applies to a certain percentage of websites, and a lower percent than you might think. A LOT of sites are very easy to scrape, with minimal scraping techniques required outside of using the modern libraries, like a good TLS fingerprint friendly HTTP client. Take shopify for example, huge company and powers millions of online stores, yet you just need to add “/products.json?limit=250&page=1” to the main shopify url and there you go, all the product data, neatly in JSON format. You can scrape a full store in 10 requests.

So where does that actually leave me and what challenges and issues I enjoy solving and creating things for now? It’s AI and Agentic, workflows and efficiency, effectiveness and simplicity. And some more linkedin buzzwords.

Jokes aside, I’ve been spending a lot of time optimising scraper creation, designing workflows for multiple usecases, creating CLI tools with specific uses to steamline agents with the goal to make web data more accessible to everyone, and accessible in whichever place or format that person wishes to work in.

This means I lean heavily on our web scraping API. But the more I do (and yes I can see the costs) it just makes sense to me to hand that part off to a SAAS/3rd party so I don’t need to be in that world.

I hope this resonates with you all in some way. I still scrape a lot, I live it everyday, I’ve just naturally shifted to a different area that one I used to be in. I barely look at what websites I’m scraping anymore, as I never consider them an issue. I do look at my AI and my agents to see what is possible, how to work with them in the best way, and what tools I can create to make it easier and more efficient.

These are the sorts of things I’ll be writing about here. I will of course still write about the tehcnical side of scraping at scale, what is happening in the world of web data, but as and when I find somthing interesting to share.

See you soon. John

Subscribe now

A CSS sprite is an image that contains multiple smaller images combined into a single file. Instead of making separate HTTP requests for each individual image, you load one file and use CSS background-position to show only the portion you want.

The classic use case is UI icons. If you have twenty toolbar icons, you arrange them in a grid in a single PNG and reference them like this:

.icon-search {
    background-image: url('/sprites/ui.png');
    background-position: -40px 0px;
    width: 20px;
    height: 20px;
}

The negative offset shifts the image left or up so that the correct icon appears within the element’s visible area. Change the offset and you get a different icon from the same file. One network request, many images. It was a significant performance optimisation in the era before HTTP/2, and it’s still widely used today.

The same technique works for anything graphical — including digits.

Using sprites to hide data

If you render numbers as sprite composites rather than text, they’re visible to humans but absent from the HTML as readable values. A price of $9.99 becomes three empty divs, each pointed at a different offset in a digit spritesheet. The browser paints the correct pixels. A scraper reading the DOM finds nothing useful.

This is the technique I came across recently on a marketplace site. What made it particularly interesting was an additional layer on top: the CSS class names were randomised on every page load, making any static mapping approach useless. This post is about how the system works and how to break it.

Subscribe now

How its constructed

The technique has three moving parts.

The spritesheet is a single image — in the case I examined, 600×84px — containing all ten digits laid out at fixed coordinates, sometimes with the same digit appearing at multiple positions in different sizes or styles.

The CSS maps obfuscated class names to background-position values pointing into that image. Each price digit is rendered by an empty element with three classes applied: one controlling layout, one setting the background image, one setting the position. Only the position class carries the information:

.rJtKq { width: 7px; float: left; height: 15px; }
.kAsHe { background-image: url(//cdn.example.com/a8f3k2.jpg); }
.dEoRz { background-position: -280px -65px; }

The data is stored in a JavaScript variable as a sequence of class names rather than a number. A price of $3.99 is encoded as:

rJtKq kAsHe dEoRz;V;mZkRg kAsHe rJtKq;mZkRg kAsHe rJtKq

Semicolons separate digit positions. V is the decimal separator. Each segment contains the three classes for that digit position — only the one with a background-position rule matters for decoding.

What makes this resilient is that the class names are randomised on every page load. The class pointing to -280px -65px might be called dEoRz on one request and something completely different on the next. The spritesheet URL also appears to be hashed, suggesting it rotates too, though this is harder to confirm without observing multiple sessions. There is no static dictionary you can build and reuse.

Breaking it

The obfuscation rotates the names, but it cannot rotate the pixels. The digit glyphs are fixed — a 3 always looks like a 3. That is the seam.

The approach is:

1. Parse the inline <style> block to extract the current session’s class → background-position mapping

2. Download the spritesheet

3. For each background position, crop the corresponding pixel cell and compute a fingerprint

4. Match the fingerprint against a reference dictionary of known digit glyphs

The fingerprinting works because the digits are rendered at a fixed small size — 7×15px in this case — in a consistent pixel font. You don’t need OCR. You binarise the cell (converting each pixel to dark or light, which strips out JPEG compression noise) and hash the result:

def cell_hash(img_array, x, y, w=7, h=15):
    cell = img_array[y:y+h, x:x+w]
    binary = (cell.mean(axis=2) < 128).astype(np.uint8)
    return hashlib.md5(binary.tobytes()).hexdigest()

The hash is stable across page loads even though the class names and spritesheet URL change. As long as the site doesn’t change its font rendering — which is a far more expensive operation than rotating class names — the reference dictionary stays valid indefinitely.

Building the reference dictionary

You build it once from a single page load where you have ground truth for at least a few digit glyphs.

The source of that ground truth is the site itself. Discounted listings expose plain-text prices in the JavaScript because the server needs to render a “was / now” price and the discount calculation happens client-side. Those raw numbers leak into the page data even when everything else is sprite-encoded. You take those known values, map their class names to pixel hashes, and use them to seed the reference.

In the page I examined, that gave me confirmed hashes for most digits immediately. The remaining ones I identified by reading the pixel patterns as ASCII art — at 7×15px, the shapes are distinctive enough to read directly:

.  .  .  #  #  .  .
.  .  #  .  .  #  .
.  .  .  .  .  #  .
.  .  .  .  #  .  .
.  .  .  .  .  #  .
.  .  .  .  .  #  .
.  .  #  #  #  .  .

One thing worth noting: some digits appeared at multiple positions in the spritesheet with slightly different rendering, producing different hashes for the same digit value. The reference dictionary needs to accommodate this — multiple hashes mapping to the same digit.

Decoding

Each page load requires exactly two HTTP requests: one for the HTML, one for the spritesheet. The decode process is then pure in-memory work:

1. For each encoded price string, split on ; to get digit segments

2. For each segment, find the class that has a background-position rule in this session’s CSS

3. Crop the pixel cell at those coordinates in the spritesheet

4. Hash it and look up in the reference dictionary

5. Assemble the digits around the decimal separator

This runs in milliseconds per page regardless of how many listings it contains.

A regex trap worth knowing about

One bug that’s easy to miss: in this site, the entire product listing array is serialised as a single line of JavaScript. When I first wrote the extraction regex using .*? to locate precoCss fields, it worked in testing but silently broke on real data.

The lazy wildcard .*? still crosses object boundaries when everything is on one line. So a listing with no precoCss of its own — one of the plain-text discount listings — would match the precoCss from the next object in the array and get assigned the wrong price entirely. The failure was silent because a price was returned, just the wrong one.

The fix is to extract individual JSON objects before parsing fields within them:

objects = re.findall(r'\{[^{}]+\}', html)
for obj in objects:
    price_css = re.search(r'"precoCss":"([^"]+)"', obj)

[^{}] cannot cross object boundaries regardless of line structure. It’s a good habit whenever you’re using regex on concatenated or minified JSON.

Why this is interesting

Most anti-scraping measures create friction — they slow you down, require infrastructure to solve challenges, or force you to render JavaScript. This one is architecturally different. It accepts that the page will be fetched and instead attempts to make the data unreadable once you have it.

It’s also cheap to deploy. Generating randomised class names and a matching CSS block on each render is a small server-side operation. The spritesheet itself is static — only the filename rotates.

The weakness is that it relies entirely on keeping the class-to-position mapping secret, while necessarily including that mapping in the page it’s trying to protect. The CSS, the spritesheet, and the encoded data are all present in the response. Once you recognise the pattern and fingerprint the pixels rather than chasing the class names, the randomisation stops mattering.

I think whilst easy enough to decode, it serves it’s main purpose - putting a barrier up to stop the majority.

John.

Quick reference: how to handle this in the wild

If you encounter a site using CSS sprite obfuscation for data, here is the end-to-end process.

1. Confirm you’re dealing with sprites

Inspect the element containing the data you want. If you see a row of empty or whitespace-only elements each with opaque class names, and the corresponding CSS maps those classes to background-position offsets into an image, it’s sprites. Check the page source for a JavaScript variable containing semicolon-separated class name sequences — that’s the encoded data.

2. Find your ground truth

Look for any place on the page where the real values leak as plain text. Discount or “was/now” prices are a common source, since client-side arithmetic requires the raw number. Aggregate stats (min, max, average) are another. You only need a handful of known values to seed the reference — ideally covering a spread of different digits.

3. Build the reference dictionary

For each known value, use your ground truth to identify which CSS class corresponds to each digit. Download the spritesheet, crop the pixel cell at each class’s background-position offset, binarise it, and store an MD5 hash of the result mapped to the digit. Do this once and save it to disk.

def cell_hash(img_array, x, y, w, h):
    cell = img_array[y:y+h, x:x+w]
    binary = (cell.mean(axis=2) < 128).astype(np.uint8)
    return hashlib.md5(binary.tobytes()).hexdigest()

If you can’t find enough ground truth, render the cells as ASCII art (dark pixel = #, light pixel = .) and read them directly. At small sizes with a pixel font, digits are identifiable by eye.

4. Decode on each page load

For every page you scrape: parse the inline <style> block to get the current session’s class → background-position mapping, download the spritesheet, hash each cell, and look up against your reference dictionary. The class names will be different every time — the pixel hashes won’t be.

5. Watch for cross-object regex bleed

If the encoded data is embedded in a minified or single-line JavaScript array, be careful with .*? in your extraction regex. It will cross object boundaries and silently assign the wrong data to the wrong record. Extract individual objects with re.findall(r'\{[^{}]+\}', html) first, then parse fields within each one.

6. Know when to rebuild the reference

The reference dictionary stays valid as long as the site’s font rendering doesn’t change. Class name rotation, URL hashing, and spritesheet filename changes don’t affect it — only a change to the actual glyph shapes would. If you start getting ? characters in your output, re-examine a page with known values and check whether the pixel hashes have changed.

One year ago I decided that I needed a change in my day job. Using contacts in the web scraping industry I was able to line up a few interviews and eventually as of April last year started to work for Zyte full time as head of Devrel.

This was a dream role for me - I had the opportunity to create content full time, share more about data extract and learn from developers who did it full time as well as expand my own skillset into areas like conference speaking.

However, this meant I needed to take a step back from my own channel for a while. I wanted to focus on my job, and while I felt like I had achieved most of I wanted to from my own channel in my head I was always planning to return, I just didn’t know what it would look like.

So last week I released a new video for the first time in about 8 months - the longest gap since the channel was created in 2019.

I am back creating, scraping data and sharing my methods, but with more knowledge, and more industry insights than before.

I’ve also moved my blog/mailing list to substack, and have chosen to put my written content here.

So whats next?

I have many video ideas, and have a few created already ready to go. Back to weekly cadence, back to creating and sharing, and a new feeling of no pressure for views/subs - and no more sponsors.

See you all next week.

John

Subscribe now