When the Scraper Breaks, Your AI Still Answers
A collection run can end in four states and most alerting covers one of them. When a model reads the data instead of a person, a break stops being an error and becomes a confident wrong answer. What to watch, what to require, and the part nobody can fix.
By Chirag Jakhariya, Co-founder & CEO · · 8 min read
A scraper that breaks used to be an incident. Now it's an answer.
When a person read the output, the break announced itself - the file didn't land, and by nine o'clock someone was asking where the data went. Put a model at the end of the pipe instead and that stops. Stale data in AI pipelines doesn't announce itself, because the model's job is to produce a fluent answer from whatever it's handed.
It has no way to know the price it's quoting was last confirmed six weeks ago. It'll say the number with exactly the same confidence either way, and your dashboard will be green the whole time.
That's the problem this post is about. Not "data quality" in the abstract - the specific case where the thing feeding your agent is a public website you don't own, don't control, and can't get a change notification from.
Your Monitoring Watches the Job, Not the Data
Your monitoring almost certainly watches whether collection ran. It probably doesn't watch whether the data it moved is still true. Those are different questions, and only the first one sets off an alarm.
A collection run that finishes on time, returns the expected number of well-formed records, and quietly fills half of them from a page that changed structure last Tuesday will pass every check you've got.
We hear the underlying request often enough to recognise it. It usually arrives as some version of this:
We want to feed this data to our AI.
A pipeline delivering live data into an agent. It's a reasonable thing to want. It also changes what counts as a failure, and most teams wire it up before noticing that.
A Scraper That Fails Loudly Is the Good Case
A collection run can end in four states. Most alerting covers one of them.
| State | What it looks like downstream | Alerts? |
|---|---|---|
| Failed | No file, no records, an error in the log | Yes |
| Partial | Fewer records than usual | Sometimes, if you baseline volume |
| Complete but wrong | Right shape, right count, wrong contents | No |
| Complete but stale | Right shape, right count, contents from last month | No |
The first one is the good case. It's loud, it's obvious, and somebody fixes it that morning. The two at the bottom are the ones that reach a customer.
This isn't a niche concern. F5's engineers measured 10.2% of HTTP requests as automated after mitigation across more than 200 billion transactions, and found that over half of all page requests for web content came from scrapers. A lot of collection is happening. Very little of it is watched for correctness rather than completion.
Credit where it's owed: the clearest write-up of the general idea is Junaid Effendi's piece for VentureBeat, arguing that teams monitor whether the job ran rather than whether the data is still true. He's writing about CRM records, document stores and internal pricing databases, and he says plainly that public web data is outside his scope. Fair enough. That's the bit we do, and it's harder for one reason: you own those internal systems. Nobody owns the public web on your behalf.
The Page Changed Shape and the Record Still Validates
The first failure mode is the one everyone expects and almost nobody catches properly.
A source changes its layout. Your selector doesn't error - it matches something else. The field that held a price now holds a delivery estimate, or an empty string, or the price of a different item further down the page. Schema validation passes, because a string is still a string and a number is still a number.
Null checks won't save you here. The record isn't missing a field. It has a field, the field has a plausible value in it, and the value is wrong.
What catches it is watching distributions rather than types. If a field's fill rate drops from 94% to 71% overnight, something moved. If a price column's median shifts by an order of magnitude, something moved. Baseline those per field, per source, and alert on the change rather than on the absence. It's more work than a null check. It's the only thing that finds this class of break before a user does.
Our own FAQ has said for a while that layout changes happen regularly and fixing them is part of what you're paying for. That's still true. What's changed is the cost of noticing late.
You Can't Tell "Gone" From "We Never Got It"
This one is genuinely hard, and we've not seen anyone else write it down.
A record was in yesterday's run and it's missing from today's. Two completely different things could have happened. The listing was removed at source - it sold, it expired, the company took it down. Or your collection hit a timeout, a rate limit, or a page that rendered slowly, and you didn't get it.
To a downstream index, those are identical. Both are an absence. And the two correct responses are opposites: one means delete the record, the other means keep it and try again.
Get it backwards in the first direction and you delete real data on a bad network day. Get it backwards in the second and you keep serving a listing that no longer exists - which is worse, because a deletion that never propagates leaves content retrievable and looking like evidence. The agent cites it. It reads as a fact. It hasn't been true for a month.
The fix isn't clever, it's just discipline: carry the collection outcome alongside the record, not instead of it. Every record needs a status - confirmed present, confirmed removed, or not reached this run - and "not reached" has to be its own state rather than collapsing into absence. Then downstream can decide what to do, because downstream can finally tell the difference.
Freshness Belongs to the Record, Not the Job
"The pipeline ran at 6am" is a statement about the pipeline. It says nothing about any particular record.
In a single run, some sources will have been re-fetched and confirmed. Some won't have changed and were skipped. Some will have fallen back to a cached copy after a failed request. All three land in the same table, with the same load timestamp, looking equally current.
So keep a last-verified timestamp per record - the moment that specific fact was last seen at source, not the moment the batch finished. Pass it through to the model. An agent that can say "as of 3 September" is a different product from one that can only say the number, and it costs you one column.
The second half of this is setting a freshness target per source instead of one number for everything. A statutory filing that changes twice a year and a price that moves hourly don't need the same schedule, and pretending they do means overpaying for one and being wrong about the other. That's the real question behind how often a source needs collecting, and it's worth answering deliberately rather than defaulting to nightly for the whole lot.
What to Put in the Arrangement
If someone else runs the collection - us, another supplier, or a team two floors down - four things belong in writing. This part tends to get skipped, because procurement asks about uptime and volume and nobody thinks to ask about truth.
- Per-record provenance and last-verified time. Not a batch timestamp. If you can't trace a number back to a source and a moment, you can't defend it in an audit.
- Notification when a source changes structure. Not a fix - a notification. You need to know the window exists before you can ask what happened during it.
- A freshness window per source, agreed up front. One global SLA is a number that's wrong for every source individually.
- A named answer to who fixes it, and how fast. Ours is that we monitor collection and usually spot a break before the client does, and selector fixes are included in retainers. Whatever your supplier's answer is, get it as a sentence rather than an implication.
Where the pages you depend on move often, collection can be built to read page structure rather than a fixed selector - that's what the AI-assisted extraction is for. It reduces how often a layout change breaks you. It doesn't remove the need for any of the four above.
A Check You Can Run This Week
Take ten facts your agent has cited to a real user recently. Open each source by hand and compare. Note not just whether the value matches, but when you'd have found out if it hadn't.
Ten records, an afternoon. Most teams find at least one, and the surprise is usually the detection lag rather than the error rate.
The Part That Has No Clean Fix
Everything above shortens the gap between a source changing and you noticing. None of it closes the gap, and we'd rather say that than sell you a number.
You don't own the source. It can change at any moment, for its own reasons, with no notice and no obligation to you. There's no webhook, no changelog, no deprecation period. A supplier who promises you stability on a third-party site is promising you something they don't control. What they can honestly offer is a short detection window and a fast fix, and you should read any stronger claim as marketing.
There's a second limit that's less comfortable, because it applies to us as much as anyone. Fresh, correct data still isn't sufficient. The FreshStack benchmark, published at NeurIPS in 2025, found that off-the-shelf retrieval models significantly underperformed oracle retrieval on all five of its topics, and that rerankers failed to improve first-stage accuracy on two of the five. On fast-moving technical content, retrieval isn't a solved problem. You can hand a model perfect data and still get a wrong answer, because it pulled the wrong three chunks.
One more thing, on numbers. Researching this post, we found figures repeated across a dozen articles with no traceable origin: a widely quoted claim that 80% of engineering time goes on maintenance, a 60% token saving from one document format over another, a market projection to 2030, an incident-detection rate at a large tech company. We cut all of them. PromptCloud's write-up on web data for AI agents cites a 2026 edition of the F5 bot report. It's the 2025 one. The figures on that page do check out - we followed them back to F5 and they're accurate - but we only know that because we went and looked. If a number in this space doesn't come with a link you can open, treat it as decoration.
So the honest position is this. The break is going to happen, you won't get told, and the model will keep answering while it's happening. What you get to decide is how long that goes on for, and whether anyone can tell afterwards what the agent was looking at.
If you're wiring web data into something that answers questions, tell us what you need collected and how fresh it has to be. We'll come back with a sample and a schedule, and say what's realistic before you commit to anything.