Topics include:
• JEV and probabilistic classification
• Using AI to decide when a website needs browser rendering
• Quality of input signals, cost, speed, and debugging
• Scrapy workflows, notebooks, and data review
• Product variants, extraction, and the cost of extra requests

Subscribe for more conversations about web scraping, data extraction, and the tools and techniques behind them.

#WebScraping #AI #DataExtraction

They also share new Scrapy projects, reflect on the power of open-source software and Linux, and look at the rise of proactive desktop agents. Plus, there’s time for model pricing, developer workflows and a well-earned appreciation of croissants.

If you’ve built something interesting, figured out something the hard way, or have a perspective on where this space is going, we’d like to give you a stage to share it.

What we’re looking for

The theme for 2026 is the intersection of AI and web data. That covers a lot of ground: agentic scraping workflows, scrapers and antibot systems, large-scale extraction pipelines, LLMs as data parsers, and the legal and compliance questions that come with all of it. We’re also interested in practitioners who work with web data further down the stack; data engineers and ML teams who depend on extraction quality but rarely get to talk about it at a scraping-specific event.

What we’re not looking for: product pitches, introductory tutorials, or talks that would fit equally well at any generic developer conference. Our audience works in this space every day, and they can tell the difference between a talk that’s been written for them and one that’s been repurposed from somewhere else.

The strongest applications we’ve received over the years have come from people describing a specific problem they ran into, what they tried, what failed, and what eventually worked. Stories that are honestly told consistently outperform polished vendor narratives.

What you get

As this is an industry event run by Zyte, we cannot pay for travel or accommodation. What we do offer:

• A free conference ticket for you, plus additional free tickets for your team

• A referral discount code to share with anyone in your network

• Your talk published on the Extract Summit YouTube channel

• Promotion across Zyte’s channels before, during, and after the event

For many people who speak here, the value is the audience. These are practitioners - the people making technical decisions, choosing tools, and building the infrastructure that the industry runs on. It’s a small conference by design, which means the conversations in the hallway are as useful as the ones on stage.

The events

Austin, Texas 7 & 8 October & Dublin, Ireland 10 & 11 November

Talks are approximately 20 to 30 minutes. Day 1 is the workshop day that we run, and will include lightning talks, and day 2 is the main event.

How to apply

CFP is here.

The application is short, tell us who you are, what you want to talk about, and why this audience is the right one for it. If you’ve given the talk before, include a link to the recording but if you haven’t no problem, a clear outline and a sentence on why you’re the right person to give it will do the job.

If you know someone who should be speaking at Extract Summit, send them this post. Past speakers: we’d love to hear from you again, and we’d equally love a referral to someone new in your network who has something worth saying.

Questions? Reply to this post or find us in the Extract Data Discord and speak to John aka jhnwr

Google dropped Gemini 3.0 Pro this week! We were quick to jump on to see how it performed for us, both internally, but also for users of our Web Scraping Copilot extension. Up to this point we had been recommending at least GTP-5 mini but I’d become a firm Gemini 2.5 Pro enjoyer so was pleased to see that 3 Pro was even better in our testing. I’ve been using it ever since and have been very impressed. Feels very much like Google are winning here now.

We’ve been working on some great content too, summarising our favourites from Extract Dublin and have plenty of cool stuff in the pipeline to share with you for the rest of the year.

Gemini 3 Pro

So far, VERY good.

Our Head of R&D Konstantin Lopukhin summarised his thoughts for us in this post - “It achieved the highest quality scores, improving on the already strong performance of Gemini 2.5 Pro and edging out leading models like GPT-5 and Claude.”

Whilst it is still in preview, we are recommend trying it with our VS Code Extension, as I shared here:

Dead Internet Theory?

Bots talking to Bots.

Dead Internet Theory is the hypothesis that the majority of internet traffic, content, and interactions are no longer generated by humans, but rather by bots and artificial intelligence. Originally dismissed as a fringe conspiracy, the theory argues that this “synthetic” activity—ranging from automated social media comments to AI-generated articles—is now drowning out organic human discourse, largely to manipulate engagement metrics and public perception.

This has given rise to bizarre viral trends like “Shrimp Jesus,” a series of nonsensical AI-generated images pairing religious figures with marine life.

“When the idea first started circulating around 2018, it felt like a paranoid exaggeration. But the rapid rise of generative AI has shifted this conversation from a fringe idea to an observable reality — with profound consequences.”

Read more on our blog

Beyond the block: The front line of data access

A deep dive into the evolving battle for web data access—featuring insights from Castle, Scrapoxy, and Zyte at Extract Summit 2025. Learn how AI, anti-bots, economics, and authentication standards like Web Bot Auth are transforming scraping, security, and the future of the open internet.

We were lucky enough to have representatives from both sides of the industry, who shared their knowledge with us.

“Big platforms are asking questions. They don’t really know what to do with AI agents.”

– Antoine Vastel, head of research, Castle

“The goal of the anti-bot is to raise the bar every time... to make you access the data in a more expensive way.”

– Fabien Vauchelles, creator, Scrapoxy

Read more on our blog.

Do I need an Unblocker API?

If you are a web scraper who has ever had to search for “How to handle bot detection?,” you may have seen hundreds of “instant unblocker” tools promising sky-high success rates and unlimited bandwidth for pennies per request.

On paper, they sound great: plug in an API endpoint, send your URL, and get data back.

• They often charge per GB rather than by success

• Have static fingerprints (mean less unblocking)

• Hidden maintenance costs

“When a ‘$50 unblocker’ leaves your spiders failing 30% of the time, your effective cost per successful page triples... Every ban you chase costs uptime and erodes predictability—the one thing that enterprise data operations rely on.”

More detail on our blog

Projects to Try

The thing we love the most is building. I want to share a few of the teams recent projects that we think you should try. Code and write ups on our dev.to page.

Build Your Own Holiday Deal Tracker with Python, Zyte API & IFTTT

• A great beginner friendly project that tracks prices for your favourite items over Black Friday and Cyber Monday deals, with notifications by Ayan Pahwa

n8n Web Scraping || Part 2: Pagination, Infinite Scroll, Network Capture & More

• Web Scraping Automation with N8n, for those who love N8n this is a fantasic way to implement web data into your automation pipeline, by Lakshay Nasa

From the Community

• Started with weekly live session on Extract YT channel, every Wednesday at 3:00 PM UTC- https://www.youtube.com/@ExtractSummit/live

• Watch the past streams - https://www.youtube.com/@ExtractSummit/streams

That’s a wrap for this week! John.

The conversation that sparked this reflection

Earlier this week, I asked my new Dev Engagement Manager, John Watson Rooney (yes, the YouTube influencer, welcome, John), why some developers still hesitate to use AI-powered automated web scraping tools like Zyte API. In my interactions with developers in the Extract Data Discord, they often reply

"I want full control over my code."

That sentiment keeps resurfacing in every scraping community I frequent. This got me thinking about Stephen Ango's brilliant piece "Don't Delegate Understanding", where he warns against outsourcing critical thinking to systems that extract value from your ignorance. But I think we need to examine what we mean by "control" and whether we're fighting for the right things.

AI-powered scraping tools actually give you more strategic control than traditional approaches, not less. Let me show you what I mean.

The smart delegation you're already doing

Think about your current web scraping stack for a moment. If you're running any serious data extraction operation, you're probably already using:

• Third-party proxy services for IP rotation instead of building your rotating proxy management system.

• Cloud providers for infrastructure, rather than managing your data centres

• Browser automation frameworks you didn't build (Selenium, Playwright, Puppeteer)

• Anti-detection services with proprietary algorithms for fingerprint and CAPTCHA management

You've delegated the implementation of these incredibly complex systems to specialists. And that's smart! Building a rotating proxy management system or developing sophisticated anti-bot detection is not your core competency.

This isn't about losing control—it's about choosing where to apply your control efforts.

The AI paradox in our development workflow

Here's where it gets even more interesting. We developers are already using AI tools in our development workflow:

• Cursor for AI-assisted coding

• ChatGPT for problem-solving

• Claude for code review

• Copilot for autocompletion

We've embraced AI augmentation in our development workflow, recognising that these tools amplify our capabilities rather than diminish them. We maintain agency over the strategic decisions while letting AI handle the tactical implementation.

Yet when it comes to web scraping, there's resistance to AI-powered tools—even when they offer more granular control than traditional approaches.

What control looks like with modern web scraping tools like Zyte API

Let me be precise about it. With Zyte API, the system automatically handles proxy rotation (switching from data centre to residential proxies when needed, cost-optimised per website), anti-bot protection (bans, captchas, fingerprinting), and browser automation (which you can toggle on/off and customise with your own actions). The AI-powered extraction means no CSS selectors or XPaths to write or maintain, and when websites change their layouts, the system adapts automatically.

You make an API call and get structured data back:

You get back structured product data, article content, job listings, or SERP results according to standardised schemas. When you need something custom beyond the standard schema, you define it as a custom attribute, and the AI handles the extraction logic.

But here's the key: you maintain complete control over what actually matters. You define custom attributes for any data fields beyond the standard schemas. You control data quality standards and validation logic. You decide when to use AI extraction versus when to fall back to custom parsing for edge cases – it's as easy as adding a single argument to the above snippet to request the page HTML.

You're not giving up control—you're gaining leverage. You control the what and the how (data requirements, business logic, custom attributes), while the system handles the where and the when (infrastructure, adaptation, maintenance that used to consume most of your development time).

The evolution: From Scrapy to AI-powered templates

Understanding Zyte's evolution helps explain why this approach works. The journey went:

1. Scrapy (2008): Open-source framework that prioritised developer productivity and accessibility over raw performance

2. Scrapy Cloud: A hosted platform that eliminates infrastructure management

3. Zyte API: AI-powered extraction that eliminates parsing code

4. Zyte AI Scraping: Full automation with configurable templates

The latest innovation combines all of these. Using AI-powered spider templates, you can start extracting data from e-commerce websites within seconds, without the need for ongoing maintenance. These templates can be customised by developers and then configured by non-developers through a user-friendly UI.

What's remarkable about this system is that templates ensure scraping adheres to best practices, maintaining data accuracy, while developers save time by reusing templates and non-developers access data without technical barriers.

Two paths, same destination: strategic control

Whether you choose surgical AI extraction (Zyte API) or full pipeline automation (Zyte AI Web Scraping), both approaches solve the same control paradox in different ways:

Zyte API: Infrastructure + Parsing Automation

You maintain spider architecture, crawling logic, and data pipelines. The API handles proxy management, anti-bot protection, browser automation, and parsing. Since extraction is AI-powered, website layout changes don't break your spiders.

Zyte AI Web Scraping: Complete Template Automation
You define data requirements and target sites. Set up crawlers/spiders in minutes, customise through Page Objects when needed, using pre-built templates, scale to hundreds or thousands of sites effortlessly.

In both cases, you focus on strategic decisions (business requirements, data quality, custom attributes) while delegating tactical implementation (infrastructure, parsing, maintenance).

Understanding vs. implementation

This connects to Stephen Ango's brilliant piece about not delegating understanding, where he warns against outsourcing critical thinking to systems that extract value from your ignorance. But I think we need to distinguish between delegating understanding and delegating implementation.

When you use a third-party proxy service, you should understand how proxy rotation works, why you need it, and how it fits into your architecture. But you don't need to implement your own global proxy network.

Similarly, with AI-powered extraction, you should understand your data requirements, your target schemas, and your business logic. But you don't need to reinvent parsing algorithms or maintain CSS selectors that break every time a website updates.

Understanding you should keep:

• Your data requirements and business logic

• How web scraping fits into your architecture

• The websites you're targeting and their compliance requirements

• Your data quality standards and validation needs

Implementation you can delegate:

• DOM parsing algorithms and selector optimisation

• Browser fingerprinting and anti-detection strategies

• Proxy rotation and session management

• Template maintenance and website adaptation

The resistance to AI-powered web scraping tools often stems from conflating these two very different types of control.

The question isn't whether to delegate—it's what to delegate and what to keep.

How modern tools preserve your options

Maybe the resistance isn't really about control at all. Maybe it's about trust, transparency, and the fear of building on shifting ground.

Traditional scraping gives you the feeling of control because you can see every line of code. But that code is often fragile, constantly breaking, and requires endless maintenance. You control the code, but the code doesn't actually control the outcome.

Modern AI-powered tools like Zyte API are designed to preserve your control options:

• Override automatic extraction for specific fields when needed

• Fall back to custom parsing for edge cases

• Mix AI extraction with selectors for hybrid approaches

• Define custom business logic for data validation and processing

• Maintain multiple providers to avoid vendor lock-in

You're not locked into black-box decisions. You have escape hatches and fallback options at every level.

Where the resistance really lies

After talking to hundreds of developers, I think the hesitation comes from a few specific places:

1. Black box anxiety: Not understanding how AI extraction works under the hood

2. Vendor lock-in fears: Worry about being dependent on proprietary systems

3. Cost uncertainty: Unclear pricing models compared to predictable infrastructure costs

4. Quality scepticism: Doubt that AI can match hand-crafted selectors

5. Control misidentification: Confusing tactical control (writing selectors) with strategic control (defining requirements)

Most of these concerns are addressable through education and transparency. For instance, Zyte's composite AI architecture combines probabilistic LLMs with deterministic ML models, so you can balance control, cost, and flexibility rather than relying purely on unpredictable LLM behaviour.

Spreading awareness: making the abstract concrete

The real challenge isn't technical—it's helping developers understand what modern AI-powered scraping actually looks like in practice. Too many still think it means "throw HTML at an LLM and hope for the best."

The reality is much more sophisticated: AI-powered spider templates come with built-in support for products, articles, jobs, and SERPs, with pre-built crawling and parsing logic including pagination and navigation. You can mix ML for common fields with selectors or custom LLM prompts for anything outside the standard schema.

This layered approach gives you more control than traditional scraping, not less.

An invitation to rethink control

I'm genuinely curious about your perspective on this. Are we clinging to an outdated notion of control? Is there wisdom in the resistance that I'm missing?

What would convince you that AI-powered extraction gives you more control, not less?

Here are the questions I keep wrestling with:

1. Where do you draw the "must-keep-control" line in your stack?

2. Have AI extractors ever burned you? What happened?

3. Would more transparency into AI decision-making change your perspective?

4. What's the difference between controlling implementation and controlling outcomes?

Join the conversation

This is exactly the kind of nuanced discussion I want to have more of. If you're interested in diving deeper into these topics, John hosted a live discussion on YouTube where we explored different approaches to AI-powered extraction and how you can use the custom attributes feature in the Zyte API.

I'd love to hear your thoughts, whether you're team "full control," team "smart delegation," or somewhere in between.

What's your take? Do we need to rethink what "control" means in the age of AI?

Hit reply—I genuinely want to understand where the resistance comes from and how we can build better tools that give developers the control they actually need.

TL;DR: Total control is often a comforting myth. The real question isn't whether to delegate, but what to delegate and what to keep. We can obsess over tactical control (CSS selectors that break constantly) or focus on strategic control (data quality, business requirements, outcomes). AI-powered tools aren't taking control away—they're shifting where we apply our control efforts.

P.S. If you want to understand what modern AI-powered web scraping- Zyte AI Web Scraping actually looks like under the hood, I've written detailed articles about the evolution of web scraping infrastructure, how to build 30 scrapers in 30 minutes, and deep customisation techniques. Check out the Zyte API docs or try the AI spider templates that let you scrape 30 e-commerce sites in 30 minutes. But this piece is about the bigger questions these tools raise about control, autonomy, and what we choose to optimise for.

It was a role that aligned perfectly with my lifelong passion for education and empowerment, a role that feels like home because it lets me lean on my natural gifts: understanding people, listening to their stories, wondering about the systems that shape us.

In 2021, my son, Cyan, was born, and I found myself navigating the beautiful new chaos of motherhood. Just six months later, I joined Zyte and found myself balancing late-night feedings with community calls, and discovering unexpected connections between my two disparate roles.

My approach to both work and motherhood is shaped by my roots and by self‑exploration. I grew up watching my grandparents run a small village school for farmers’ children, while my mother and aunts filled classrooms as teachers. “When you grow up, do something for under‑privileged children,” my mother told me—and, for reasons I still can't fully explain, that calling stuck.

As a child, I questioned everything, but in traditional schooling systems, curiosity is treated as trouble, so I learned early what it feels like to be squeezed into a box. Years later, when Cyan arrived, he brought awe and fresh curiosity back into my days. Seeing the world through his eyes reminds me that we are meant to wonder, to test the rules, and to change the ones that no longer serve us. Mothering him nudges me to keep challenging any system that limits potential—whether the patriarchal expectations that once guided my path or the economic barriers that gatekeep knowledge.

As these parallel journeys of motherhood and developer advocacy have unfolded, I've discovered that the learning flows in both directions, creating a virtuous cycle that continues to shape both aspects of my life.

1. Deep listening reveals hidden potential

When Cyan arrived I was full of questions, and worries: “How do I raise a child when I’ve never done it before?” My father gave me essential advice when I was a new parent: "Every child is born with their own gifts. Your job is to pay attention and stay connected to him. He will tell you what he needs." I didn’t have to perform miracles; I just had to be present.

So I watched and listened. Cyan fell in love with excavators. Diggers had always dotted the roadside, but I truly saw them only when he pointed them out. We counted scoops, spun stories, traced tyre tracks in the dust—letting his curiosity set the syllabus while I stayed present.

That practice of attentive listening became my best tool at work. On developer calls now, I skip the slide deck and begin with honest questions: “What are you building? What’s been hard?” Then I wait. The spark in a developer’s eyes when they feel heard does more than any product pitch ever could.

Motherhood didn’t just influence my advocacy; it built the muscle for it. The habit of unhurried listening and attention that guides Cyan now guides every community chat. Deep attention turns worries into wonder and strangers into partners, whether on the playground or in the pull‑request queue.

2. Creating conditions for natural growth

Before becoming a mother, I worked with companies like IBM Watson Education on personalization in learning. We were trying to solve a fundamental problem: traditional schooling often averages out rather than celebrates each child's unique gifts and needs.

This experience informed my approach to developer advocacy at Zyte. Rather than creating rigid pathways for developers to follow, I focused on building environments where exploration and collaboration could flourish naturally. When I created small “squads" to support my developer advocacy work, it wasn't just about distributing tasks—it was about creating spaces where developers could connect and inspire each other.

This philosophy extended to one of my most personal decisions as a mother: naming my son. I chose the name Cyan because it's a color without any identity attached to it—the color of the ocean, open and receptive to everything. This wasn't accidental; I wanted a name that wouldn't impose prescriptive ideas or limitations, just as I strive to avoid imposing rigid frameworks on the developers I work with. From the beginning, I wanted him to have the freedom to define himself.

When Cyan came along, I applied this same philosophy to parenting. Instead of imposing a curriculum of what I thought he should learn, I worked to create conditions where his natural interests could thrive. When we noticed his love for tools, his teachers designed counting exercises with images of screwdrivers and drills rather than the usual ice cream cones or apples.

This approach represents a deliberate break from the systems I experienced growing up, with someone directing the path my life should follow. By creating conditions to celebrate one’s natural gifts, rather than prescribing the road to follow, I'm working to dismantle these limiting structures, in both my professional community and the world closer to home.

3. It takes a village

One of the most profound parallels between motherhood and developer advocacy is the recognition that neither can be done in isolation.

But, when we started planning our developer community at Zyte, our CEO, Shane, offered advice from his experience with building the Scrapy community: “Neha, you have to acknowledge that you can’t build this alone.” His words hit me hard.

I began to see the community as something to nurture rather than control—a collaborative effort that would grow through the contributions of many, not just my own efforts.

My role wasn't to dictate but to lead through intuition and constant feedback, creating an accountability system that encouraged growth without imposing fear.

Bit by bit, I traded control for connection. My role shifted from architect to caretaker, guiding with intuition and clear feedback while letting others plant their own seeds.

This same principle applies to motherhood. Despite the cultural narrative of the self-sufficient mother, I've learned that raising a child requires a support network. Just as I built "squads" to support my developer advocacy work, I've cultivated relationships with family members, teachers, and other parents who contribute to Cyan's development in ways I alone cannot, who see what I miss and step in when I’m tired.

In both contexts, acknowledging our interdependence isn't a sign of weakness but of wisdom. It allows us to leverage diverse perspectives and strengths, creating richer experiences for both children and developers than any single person could provide. Shared effort makes ideas richer, children braver, and code better. We bloom together, or not at all.

4. Intrinsic motivation beats external pressure

The tech industry promulgates what I see as a "scarcity mindset," creating fear around keeping up with the latest technologies: "If you don't learn this new framework, you'll be left behind."

As someone who transitioned from software development to data science at IBM, I've experienced that pressure firsthand. I was once tasked with helping IBMers transition to new roles, and I naively expected everyone to adapt at the same pace, regardless of their life circumstances. I now recognize this approach as fundamentally flawed—it assumes that external pressure is more effective than internal motivation.

This realization has profoundly shaped my developer advocacy work. Rather than creating fear around keeping up with the latest web scraping techniques or AI integrations, I focus on helping developers understand their unique strengths and how our tools can enhance their natural capabilities.

When I became a mother, this principle became even more important. When people ask if I'm concerned about what jobs will exist when Cyan grows up, especially with AI transforming the landscape, I share my belief that the most valuable skill isn't coding or any specific technical knowledge—it's self-knowledge and the ability to connect with one's authentic interests.

What's most important is to stay connected with yourself. This wisdom, which I developed through years of working with developers, now guides my parenting philosophy as well.

5. Joy is the ultimate productivity tool

In my webinars and workshops as a developer advocate, I deliberately incorporate storytelling and relatable analogies rather than dry technical explanations. Whether I'm explaining how baking cakes relates to machine learning or demonstrating how to build scrapers for 30 e-commerce sites in 30 minutes, I'm always looking for approaches that transform learning from obligation to delight.

This focus on joy as a catalyst for learning has been reinforced through my experiences with Cyan. One evening, we sat down to practice tracing letters—a task he typically found tedious. As his interest waned, I spontaneously began singing the letters to the tune of his favorite songs. His eyes lit up, and suddenly, what had been a chore transformed into play. We sang our way through the alphabet, and he completed the assignment with enthusiasm rather than resistance.

This moment crystallized something fundamental about learning that applies equally to children and adult developers alike: when joy is present, effort feels effortless. As Julia Cameron notes in "The Artist's Way," which has deeply influenced my journaling practice, creativity flows most freely when we approach it with playfulness rather than rigid discipline.

The tech industry often frames learning as a serious, competitive endeavor—master this framework, learn that language, or risk becoming obsolete. But my experiences both as a mother and as a developer advocate have convinced me that the most effective learning happens when joy and play are central.

What's fascinating is how our leadership at Zyte has influenced this approach. As I was building our developer community, my own manager, Marie, gave me that space, that open canvas to explore and experiment. She saw potential in me that I couldn't yet see in myself and trusted me with projects when I lacked self-confidence. This experience of being given freedom within a framework of accountability—deciding my own priorities and deadlines while maintaining responsibility for outcomes—transformed my professional approach.

The trust Marie placed in me became a model for the trust I now extend to Cyan, creating space for his authentic development while maintaining a gentle framework of guidance.

Joyful journeys, better outcomes

As I write this, Cyan is four years old and thriving in his nature-based school, while I'm expecting my second child.

Meanwhile, the developer community I've been nurturing at Zyte has grown from a handful of contributors to a vibrant ecosystem with 19,000+ members.

What surprises me most is how these insights cross-pollinate. My experiences as a developer advocate gave me frameworks for supporting Cyan's growth without imposing rigid expectations, while motherhood has deepened my empathy and responsiveness in professional contexts.

In both roles, I've learned that the most powerful approach isn't to prescribe paths but to create conditions where growth can happen organically. Whether I'm curating experiences on Cyan's iPad or designing collaborative spaces for developers, my goal is the same: to provide enough structure for safety while leaving ample room for exploration and discovery.

This balanced approach yields not just better outcomes but more joyful journeys. And ultimately, isn't that what we want for both our children and our professional communities—not just achievement, but fulfilment in the process of growth itself?

This isn't science fiction; it's the quiet, pervasive new reality of the digital age. According to research from the Pew Research Center, a staggering 25% of web pages existing between 2013 and 2023 have simply vanished. The web, our sprawling, vibrant repository of knowledge and culture, is built on shifting sands.

Yet, as the digital tide washes away swathes of our collective memory, a determined movement is fighting back. They are the self-preservationists: a diverse coalition of archivists, librarians, technologists, researchers, business compliance officers, and concerned individuals refusing to accept the ephemerality of the digital world.

From the monumental server racks of the Internet Archive to the focused efforts of Harvard Law School's Library Innovation Lab, from grassroots community projects capturing local histories to individuals taking steps to save content in durable formats, a race is underway to save the web from itself.

"More and more of our intellectual endeavors... exist only in a digital environment," explains Mark Graham, director of the Internet Archive's indispensable Wayback Machine. "That environment is inherently fragile."

The story of web preservation in 2025 is one of resilience, innovation, and a growing understanding that if something digital is worth keeping, it's worth saving—securely and accessibly—before it vanishes forever.

The Digital Erosion Crisis

The web is vanishing beneath our feet. What appears stable is, in reality, built on shifting digital sands. The statistics paint a sobering picture: according to Pew Research Center, 38% of web pages from 2013 are now inaccessible. Even recent content isn't immune; 8% of pages published in 2023 vanished within the same year.

Source: Pew Research Center, When Content Disappears, May 2024

This digital erosion affects critical information infrastructure. One in five government websites contains broken links, while Pew Research Center found over half of Wikipedia articles have broken links in their references. In other words, the foundation of evidence supporting online information is slowly disintegrating. "The risks are manifold... institutions fail, or companies go out of business," notes Mark Graham.

The causes are both technical and institutional. Link rot—the decay of hyperlinks—is endemic. Websites restructure, domains expire, companies fold, platforms pivot. Hardware and software obsolescence renders content unreadable.

Institutional decisions accelerate this loss. Cost-cutting, rebranding, legal concerns, or political expediency lead to content removal. The 2025 US presidential transition saw thousands of datasets disappear from Data.gov almost overnight. Walled gardens and proprietary platforms create digital silos where content is held captive, prone to disappearing when the platform changes course.

This digital impermanence is undermining accountability, hinders research, and fragments cultural memory.

The Preservation Vanguard: Major Players and Initiatives

At the forefront of the battle against digital amnesia stands the Internet Archive, a non-profit digital library founded in 1996. By 2025, it had preserved an astounding 866 billion web pages, 44 million books, and countless other digital artifacts. Its Wayback Machine relentlessly crawls the web, capturing snapshots of websites over time, making them available free of charge. "When we then went and looked at how many of those URLs were available in the Wayback Machine, we found that two-thirds of those were available," explains Graham, highlighting its success in preserving otherwise lost content.

Mark Graham, Internet Archive - Source: Wikimedia Commons, Creative Commons Attribution 2.0

A key initiative is the End of Term Web Archive, which snapshots U.S. government websites, many of whose pages are deleted during presidential transitions. The 2024/2025 crawl collected over 500 terabytes, including more than 100 million unique web pages, further secured on decentralized networks like Filecoin as part of its Democracy's Library project.

Academic institutions contribute crucial expertise. Harvard Law School's Library Innovation Lab complements the Internet Archive by creating a "data vault" for federal datasets. Following the 2025 transition, it released a 16-terabyte collection of over 311,000 datasets from Data.gov, providing open-source tools for replication. Elsewhere, national libraries like the U.S. Library of Congress and the UK Web Archive also play vital roles, archiving government sites, news, and national domains.

Collaborative efforts amplify these strengths. The Internet Archive's Community Webs program empowers over 200 smaller cultural heritage organizations to preserve their own, local digital histories, diversifying the archival record. As one participant, Raíces Cultural Center, notes, the program fosters "community identity... and sharing untold or undertold stories." Meanwhile, coalitions like the Data Rescue Project, run by university librarians, track data takedowns and coordinate preservation responses. Individuals also contribute; as Gary Price explained in Online Searcher, anyone can add websites to the Internet Archive, democratizing preservation.

Motivations: Why Save the Web?

You don’t have to be a hoarder to want to preserve digital data. The race to save the web is driven by a complex interplay of motivations spanning commercial interests, cultural imperatives, academic needs, and legal requirements.

For businesses, data archival is a strategic necessity. Regulatory frameworks like GDPR, CCPA, and the Sarbanes-Oxley Act mandate retention and accessibility, with non-compliance carrying severe penalties. Robust archiving is essential for risk management and business continuity across industries like healthcare, finance, and e-commerce. The enterprise information archiving market, projected to exceed $10.5 billion in 2025 according to The Business Research Company, underscores this commercial reality.

Cultural and historical imperatives provide a powerful motivation. "Historians of the future may struggle to understand fully how we lived our lives in the early 21st Century," a recent BBC News article warned. Institutions are working to ensure our digital present doesn't become a historical black hole, striving to capture diverse voices and recognizing that, without deliberate effort, the digital narratives of marginalized communities are particularly vulnerable.

Academia relies heavily on a stable digital record. Researchers need persistent access to cited sources to verify findings and build upon previous work. The disintegration of online references undermines scholarly integrity. Historians and social scientists require preserved digital artifacts—websites, social media data—to analyze contemporary society. Legal systems increasingly intersect with the digital realm. Court cases may hinge on archived emails or website content, while government accountability relies on preserved official records.

Underpinning these institutional drivers is a growing focus on personal digital agency—a desire to reclaim control in the face of ephemeral platforms. This reflects an unease with entrusting personal digital histories to transient third-party services. Individuals are exploring ways to save content in durable, accessible formats they control, such as using tools to capture web pages as Markdown files. A growing number of web user seems driven by the principle that content worth finding is often worth preserving.

Hurdles on the Path: Challenges and Controversies

Despite the growing momentum, the path to comprehensive web preservation is fraught with significant hurdles.

Legal battles cast a long shadow. Copyright law, designed for physical media, sometimes clashes with mass web crawling. Publishers have launched lawsuits against archives like the Internet Archive, arguing infringement. While archives invoke fair use defenses, the legal landscape remains uncertain and costly. The "right to be forgotten," established in some jurisdictions, introduces further complexity, pitting individual privacy against the historical record.

Wayback Machine, Internet Archive

Resource constraints perpetually challenge preservation. The scale of the web vastly outstrips available funding. Building and maintaining the technical infrastructure requires substantial, ongoing investment. Specialized expertise in web archiving and digital forensics remains scarce.

Selection bias inevitably shapes what gets saved. Who decides which websites or communities are worthy of preservation? Institutional archives may inadvertently perpetuate historical biases, underrepresenting content from marginalized communities or non-mainstream platforms. Ethical considerations, particularly around preserving sensitive personal information captured without explicit consent, remain paramount.

Technical difficulties abound. The modern web is dynamic, interactive, and often hidden behind logins. Capturing the full functionality of complex web applications is notoriously difficult. Format obsolescence remains a constant threat, requiring ongoing migration and emulation strategies.

Charting the Future: Innovation and Uncertainty

The future of web preservation is being actively shaped by technological innovation, evolving policies, and the shifting dynamics between institutional and individual efforts.

Decentralized technologies offer a paradigm shift. Projects leveraging networks like Filecoin to store archived data across multiple nodes promise greater resilience against single points of failure. Artificial intelligence is poised to revolutionize preservation workflows, assisting in identifying at-risk content, prioritizing crawling, generating metadata, and enhancing searchability.

Policy and legal frameworks remain critical battlegrounds. Calls for mandated archiving, particularly for government data, may gain traction. International collaboration could lead to more standardized practices. However, these efforts could continue to clash with commercial interests and privacy concerns. The role of dominant tech platforms remains uncertain, with tension between proprietary platform archives and independent, public-interest archiving likely to grow.

Finding sustainable funding models beyond non-profit donations and grants remains crucial. The future likely involves a deeper integration of institutional and personal preservation efforts, connecting large repositories with individual actions facilitated by user-controlled formats and tools, embodying a "file over app" philosophy for digital longevity.

Guardians of Digital Memory

The web, our sprawling, collective consciousness, is surprisingly ephemeral. As links decay and platforms vanish, the self-preservationists—institutional and individual—stand as guardians against a rising tide of digital amnesia. From the monumental efforts of the Internet Archive to the individual saving crucial articles in durable formats, the motivations are shared: a refusal to let our digital heritage dissolve.

The challenge is immense. Yet, the preservation movement is growing, driven by the understanding that access to information—past and present—is fundamental.

"Web archiving is more than just preserving history—it’s about ensuring access to information for future generations," observes Mark Graham. This task falls not only to dedicated archivists but to all users of the web. Supporting preservation organizations, advocating for responsible data policies, and adopting personal preservation habits are all part of the solution.

The race against digital forgetting requires collective vigilance. By embracing both institutional scale and individual agency, we can strive to ensure that the vibrant, complex, and essential record of our digital lives endures.

The web is worried. In 2025, website owners began noticing something different in their server logs – a flood of traffic from a new generation of crawler, scouring the web for AI training data

While new traffic is commonly seen as good and the new wave of chat bots and other AI products are ground-breaking, for some site operators, automated requests from AI services have also been heaping unprecedented demand on web infrastructure.

• One engineer reported over 70% of this traffic now comes from AI-linked user agents, with search bots now barely registering.

• Another site owner received 81,000 requests in just two and a half hours, 97% of which were reckoned to be AI bots.

• Wikimedia Foundation, one of the largest publishers in the world, is struggling under the weight. Between January 2024 and April 2025, its bandwidth used to serve images surged by 50%, as automated bots ingested its openly licensed content.

No wonder some site owners are beginning to feel “under attack”. Sudden traffic increases can overwhelm servers, leading to downtime, extended hosting bills, lower usage and lost business.

Ethical standards for the conventional gathering of web data are already in common use, but the infrastructure hasn’t seen anything on this scale before – and that is raising a range of questions about how businesses can build the next generation of AI services while also being good citizens of the web.

Understanding the new breed of crawlers

To answer those questions, we need to understand what scrapers for AI services are really doing.

Scraping of any kind involves three steps:

1. When you don’t have a defined list of target URLs, a crawler systematically explores one or more websites, in search of relevant pages. It follows links, collecting target URLs.

2. Fetching is a highly targeted practice. Fetching tools like curl or wget download specific pages, whether pre-defined or obtained by a crawler, when you tell them to.

3. Once obtained, extracting pulls sought-after information from those pages, for storage in a system.

While conventional scraping usually fetches a limited set of pages with a goal to extract only what’s relevant (be it search, real estate, product, job or news data), crawlers acting for Large Language Models (LLMs), by contrast, pursue exhaustiveness, fetching and extracting as much as possible.

According to the guiding assumption for training LLMs, the scaling hypothesis, more data produces better models. And LLMs can make meaning from any kind of content they discover, so their crawlers are keen to collect everything – structured or unstructured, polished or raw.

We have seen reports of crawlers that hit the same low-value pages – log files, diffs, redundant URLs – at high frequency, without any prioritization, or scheduling logic. Others run in loops, requesting the same pages over and over throughout the day.

The difference between scraping for AI services and standard web data acquisition, then, is one of scale.

While these patterns may not suggest deliberate abuse, they might suggest a need for nuance. Polite, efficient crawling requires engineering care and operational expertise.

The impact of overwhelm

Operating without that nuance could hurt the sustainability of the very web content we all know and depend upon.

• Some website owners have started to ban entire ranges of IPs linked to AI services.

• Some set honeypots, tarpits, and proof-of-work challenges to waste compute time and slow the crawlers down.

• Some employ aggressive tactics like content poisoning, where junk data is served to corrupt large-scale training sets.

• Micropayments, login walls, and CAPTCHAs are spreading, too – adding layers of friction that makes public content harder to access.

• Google itself has started shielding its search result pages behind JavaScript — a move some think reflects this new AI-driven pressure

• Even infrastructure providers are adapting. Cloudflare, for example, has started deploying AI traps that force the crawlers to ingest irrelevant AI-generated data - an AI arms race we call the “Turing tango”.

But these defenses don’t just hamper AI bots. They also hinder others: archivists, researchers, competitive analysts, journalists, academia, and real individual users.

The long-term risk is hard to ignore: the open web becomes less open..

All of us, then – AI developers, technology vendors, publishers, and the public – share an interest in keeping the web functional, fair, and open.

How Zyte crawls carefully

It is possible to build innovative, large-scale AI systems in a way that treats website owners gently and maximises data sustainability, by implementing appropriate technical measures.

This is a landscape Zyte knows well. Since 2014, Zyte has worked with organizations of all sizes – from startups to global enterprises – to gather web data for use cases including AI training in a careful, considered way.

We have explored and enabled those responsibilities at the technical level, long before the recent AI explosion put them in the spotlight. Here’s how we put this experience into action.

Principles of politeness

Step one in Zyte’s best practices for web scraping is: “Don’t be a burden.” Like the medical profession’s Hippocratic oath to “do no harm”, it urges restraint and care:

• “Limit the number of concurrent requests to the same website from a single IP.”

• “Respect the delay that crawlers should wait between requests by following the Crawl-delay directive outlined in the robots.txt file.”

• “If possible, it is more respectful if you can schedule your crawls to take place at the website’s off-peak hours.”

But having the right principles is only the start. You also need tools with the power to turn those values into action, without having to build every safeguard yourself.

Politeness as a product

Careful crawling is baked into Zyte’s products and policies.

Zyte API, our full-stack web scraping API, limits the rate at which websites can be accessed, to safeguard sites against overwhelm.

• Every website has its own unique tolerance for traffic. Zyte has a long-developed an intimate understanding of those tolerances and adheres to website-specific rate limits, ensuring websites’ capacity is respected in a personal way.

• Each Zyte API account also has its own rate limit for each website, so that no single account can flood any given website.

• As a back-stop, all Zyte API users are also limited to 500 requests per minute.

Behind the scenes, Zyte’s team monitors combined usage across all API entry points to ensure these limits are consistently enforced.

While these practices lighten the load on websites, they don’t limit our customers’ data ambitions. Not only does spreading the load across our network actually increase the chance our more demanding users get the results they need; in each case, users can also request an increase to their rate limit – something that is subject to a Zyte review process to assess the likely impact on target sites.

Together, these are expressions of fairness as a design principle — ensuring that no one player can dominate or exhaust the shared resource of public web data, while demanding users retain bandwidth to gather the data they need.

Servicing with sensitivity

While our principles and products help our software customers crawl carefully themselves, the experts at Zyte Data, our done-for-you data extraction service for large-scale data buyers, also go to considerable lengths to tread carefully on clients’ behalf.

For example, Zyte has recently been helping one scaled AI company gather a large amount of training data. Having seen the unfolding project docs and chats, I have been impressed by the diligence team members have shown in assessing the impact of their crawling.

I have seen colleagues estimating target sites’ traffic levels in order to consider the impression of their own footprint. I have seen project managers and developers reject temptations to exceed polite limits, deciding instead to reduce concurrency.

When the job demands greater frequency, the internal Zyte Data team states its case to increase a rate limit using the same request process as external Zyte API customers – a process laid out in a clearly documented workflow that includes an evaluation containing fine-grained consideration of the proposal and requiring senior-level approval.

Regular cross-team reviews, proactive limit-setting, and a culture of responsible web use underpin these practices, ensuring enterprise customers achieve optimum data while websites, large and small, stay resilient.

Hands-on controls

Zyte’s responsible approach extends to the wider ecosystem.

Scrapy, the open-source web scraping framework that Zyte’s founders kickstarted and another way in which many AI developers gather training data, builds care for capacity into its core.

It offers a wide set of adjustable levers to help you scale data collection with care.

1. Disclosing crawler identity helps servers recognize and, if they choose, manage a crawler’s access. In Scrapy, this is easy with the USER_AGENT and ROBOTSTXT_* settings. These work hand-in-hand with the robots.txt rules. The scrapy.downloadermiddlewares.robotstxt middleware reinforces this by automatically skipping forbidden URL paths.

2. Controlling speed and concurrency is where Scrapy shines. With eight different crawl speed-related settings and an adaptive AutoThrottle extension, data gatherers can fine-tune how fast their crawlers move to strike a balance between efficiency and caution.

3. Cache management is one of the most effective yet often overlooked ways to reduce server strain. Instead of repeatedly downloading unchanged data, only refresh when needed. Scrapy’s httpcache middleware stores and reuses HTTP responses based on RFC2616 standards. It honors cache directives (no-store, no-cache) and calculates freshness using Last-Modified, ETag, and Expires headers.

Rather than pushing everything to the limit, Scrapy helps businesses fine-tune their approach to AI training data, balancing speed and respect for the target servers, without building everything from the ground up.

A web worth sustaining

The web was designed to be open. That openness has made it resilient, generative, and surprising. It’s what made it possible for voices to be heard and systems to be built.

Letting that platform degrade—through neglect or overuse—means worse outputs, narrower perspectives, and less useful results. A resilient, open web is in everyone’s interest, especially those building on top of it.

The web deserves good citizens. We all have a part to play.