Why Web Scrapers Break When Websites Change in 2026

Web scrapers break when a site changes its layout, its code labels or how its content loads. The nine changes that cause it, how to catch a silent break, and your options once you're tired of rewriting scripts.

By Chirag Jakhariya, Co-founder & CEO · 24 September 2026 · 9 min read

Your web scraper stopped working because the website changed something the scraper was told to look for: the layout, the labels hidden in the page's code, or the way the content loads. You can't stop sites changing, and no scraper is immune. What you can change is how fast a break gets caught, how small each fix is, and who does the fixing.

Below are the nine changes that break scrapers most often, how to tell a scraper has broken when it hasn't told you, and what your options are once you're tired of rewriting scripts.

Why Do Web Scrapers Break When a Website Changes?

A scraper is a set of written directions, not a person reading the page. It says something like "go to this address, find the box labelled product-price, copy the number inside it".

When the site moves that box or renames it, the directions point at nothing. The scraper does exactly what it was told and comes back with a blank, or with the wrong thing.

What is a selector, and why does it matter?

A selector is the address of one piece of information on a page - the "box labelled product-price" part of the directions. Every field your scraper collects has one.

Selectors are written against how the page is built today. That's why a change you can't even see as a visitor, like a renamed label in the code, can stop a scraper cold.

What makes a production scraper different?

A production scraper is one that runs on a schedule and feeds something people rely on: a price report, a sales list, a dashboard. A break in a one-off script costs you an afternoon. A break in production costs you every day it goes unnoticed.

Which Website Changes Break Production Scrapers?

Nine kinds of change cause most of the breaks we see. Some are loud and some are silent:

Why does a redesign break a scraper?

A redesign is the break everyone expects. The page structure changes wholesale, most selectors stop matching, and the scraper either errors out or returns nothing.

It's the easiest kind to notice and the most work to fix, because every field needs new directions at once.

Why do renamed labels in the code break a scraper?

Many sites are built with tools that generate their own label names, like price_x7Qa2, and regenerate them each time the site is updated. The page looks identical to you. To the scraper, every label it relied on has vanished.

What if content starts loading after the page opens?

Some sites switch from sending a finished page to sending an empty frame that fills itself in using JavaScript, the code your browser runs to build the page as you watch. A scraper that only reads the first copy of the page now finds empty boxes. Our guide to scraping dynamic websites covers what changes when this happens.

Why is web scraper pagination not working?

Pagination is how a site splits a long list across pages. When a site swaps numbered pages for a "load more" button or endless scrolling, a scraper that clicks "next" finds no next. It collects page one, stops, and reports success.

Why do changed web addresses break a scraper?

Sites reorganise their sections, rename categories and retire old links. A scraper that starts from a saved list of addresses hits "page not found" or gets quietly sent to a home page with nothing on it.

Do pop-ups and cookie banners break scrapers?

Yes. A new consent banner, newsletter pop-up or "choose your country" screen can sit on top of the content. If the scraper's directions don't expect it, it collects the banner, or nothing.

What if the site starts refusing requests?

Sites set limits on how often they'll answer, and they tighten them. A scraper that used to run fine can start getting error pages instead of data. The right response is to slow down, spread the requests out and check the site's terms. If the terms now forbid collection, stop.

Can formatting changes break a scraper without an error?

This is the dangerous one. A few examples of changes that return a value that looks fine and isn't:

• Prices that switch from 1,299.00 to 1.299,00, so 1,299 becomes 1.299
• Dates that flip from month-first to day-first
• Currency that changes with the visitor's location
• "Was" and "now" prices that swap places on the page

Do A/B tests and regional versions break scrapers?

An A/B test is when a site shows different visitors different versions of a page to see which works better. Your scraper can get version A on Monday and version B on Tuesday, so it works one day and fails the next with no change on your side.

What Is the Difference Between a Loud and a Silent Break?

A loud break stops the run or throws an error. It's annoying, but somebody notices. A silent break finishes normally and hands you bad data:

• A column that's suddenly blank for every row
• Half the usual number of records, with no error
• The right field filled from the wrong part of the page
• Yesterday's data delivered again, because nothing new came in

Silent breaks do far more damage, because decisions get made on them. We wrote more about this in what happens when a broken scraper feeds an AI tool.

How Do You Know Your Scraper Is Not Working?

Check the data, not just the job. "The script ran" tells you almost nothing. Five checks after every run catch most breaks:

1. Row count. Is it close to the usual number? A sudden drop is the clearest sign.
2. Blank fields. Has any column gone mostly empty?
3. Sensible values. Are prices above zero, and dates in the past week, not 1970?
4. Freshness. Did anything actually change since the last run?
5. A human look. Open five random records next to the live page, once a week.

If a check fails, hold the data back rather than sending it on. A late delivery is better than a wrong one.

How Often Do Websites Change?

There's no reliable figure for this, and we'd be wary of anyone who quotes one. It depends on the site. In our experience, the pattern looks like this:

• Busy retail and travel sites ship small changes constantly and test page versions all the time
• Directories and listings sites change less often, but redesign in big jumps
• Small business and government sites can stay the same for years

If you collect from many sites, expect breaks to be a regular event, not a rare one. That's normal, not a sign the scraper was badly built.

How Can You Make a Scraper Break Less Often?

You can't make one unbreakable, but you can make it sturdier and quicker to fix:

• Use the page's own structured data first. Many pages carry a tidy, machine-readable copy of their key facts for search engines. It changes far less than the visible layout.
• Find fields by what they say, not how they're styled. "The number next to the word Price" survives a redesign better than a generated label name.
• Keep every selector in one place, so a fix is one edit, not a hunt through the code.
• Save a copy of the page when a run fails, so whoever fixes it can see what changed.
• Run the five checks above on every run, and alert a person when one fails.

Is an official API better than scraping?

Usually, yes, if one exists and covers what you need. An API is a doorway a site builds on purpose for software to request its data, and it's designed to stay stable.

The catch is that most sites don't offer one, or offer one with fewer fields, strict limits, or a price. When the API covers your fields, use it. When it doesn't, you're back to scraping.

What Does Maintaining a Scraper Really Cost You?

The fix itself is often small. The cost is everything around it:

• Interrupted work. Whoever built it drops what they're doing, often at short notice.
• Gaps in the data. Days between the break and the fix are usually gone for good.
• Bad decisions. A silent break can feed wrong numbers into reports for weeks.
• One person who knows how it works. When they leave, the scraper becomes a black box.

We can't put a number on this for you, and it varies too much for an average to mean anything. Track the hours you spend on fixes for a month; that figure is what you're comparing any alternative against.

Can AI Fix a Broken Scraper Automatically?

Partly. So-called self-healing scrapers use AI to re-find a field after the page changes, instead of relying on a fixed selector. They help with some breaks and not others:

• Good at: small layout changes, moved boxes, renamed labels
• Not good at: new pop-ups, changed page flows, data the site has removed
• Risky at: picking a plausible but wrong field and carrying on confidently

That last point is why self-healing doesn't replace the five checks. It turns some loud breaks into no break, and a few into silent ones.

What Are Your Options When You're Tired of Fixing Scrapers?

There are four, and they trade your time against your money differently:

1. Keep fixing it yourself, but add the checks so breaks are caught the same day
2. Move to a self-healing tool and keep a person checking its output
3. Switch to an official API or a data provider, if one covers your fields
4. Hand the scraper to a managed service that owns the fixes

When does fixing it yourself still make sense?

When you collect from one or two sites, the data isn't urgent, and someone on your team genuinely likes the work. A few fixes a year is a fair price for full control.

When does handing it off make sense?

When you collect from many sites, the data feeds something people act on daily, or the person who built it has moved on. Past that point, you're paying for maintenance either way. The question is only whose time it comes out of. We looked at this trade-off in more depth in running the scraper yourself versus buying the feed.

What Does a Managed Service Do When a Site Changes?

On a weekly or monthly retainer, we build, host and monitor the scrapers, and fixes for site changes are included, not billed extra. When a site changes, this is what happens:

1. The checks on that run flag the problem before the data reaches you
2. The engineer who runs your collection looks at what changed on the site
3. They update the directions and re-run the checks
4. Clean data goes out, and you get a note saying what changed

Fixed-price projects include an agreed support window after handover instead. One honest limit: nobody can guarantee continued access to a third-party site. If a site becomes unreachable, or changes its terms so that collecting would breach them, we tell you and stop.

Questions People Ask About Web Scrapers Breaking

Why is my web scraper not working?

Most likely the website changed something your scraper depends on: its layout, the label names in its code, how its content loads, or its web addresses. Open the page yourself and compare it with what the scraper expects. If the site now refuses requests, slow the scraper down and check the site's terms before trying again.

What are the challenges of web scraping?

Getting data once is rarely the hard part. The ongoing challenges are keeping scrapers working as sites change, catching silent breaks that return wrong data without an error, respecting each site's limits and terms, and cleaning what comes back into something consistent enough to use. Maintenance, not the first build, is where most of the effort goes.

What is a self healing web scraper?

It's a scraper that uses AI to find the field it needs even after the page layout changes, instead of relying on one fixed address in the page's code. It handles small changes well. It can't cope with removed data or new page flows, and it can pick the wrong field with confidence, so its output still needs checking.

What is web scraping vs API?

An API is a channel a site builds on purpose so software can request its data in a stable, agreed format. Web scraping reads the same pages a person sees and copies the information out. An API breaks less often but only covers what the site chooses to share. Scraping covers what's visible, and needs maintaining.

How much does web scraping cost?

It depends on how many sites, how often the data refreshes, how the sites are built and how much cleaning is needed. Ongoing maintenance is part of the price: sites change, so a scraper that runs for a year needs fixing along the way. Our pricing page sets out what moves the number.

Is web scraping legal?

Collecting publicly available information is often lawful, but it isn't automatic. A site's terms of use, database rights in the UK and EU, and data protection law for anything about people all apply regardless of whether the data is public. Check each site's terms, avoid personal data unless you have a lawful basis, and take advice for your case. This isn't legal advice.

If you'd rather someone else owned the fixes, send us the sites you collect from and we'll tell you what it would take to run them for you.