Service Intelligence

Web Data Mining

Deep relationship discovery across the massive expanse of the web.

We mine deep relationships between entities, uncovering hidden links in company structures, social networks, and research data across the entire internet.

Mining starts without a list. Rather than "collect these 400 URLs", the question is "find every company in this sector and tell me what connects them". Discovery first, structure second.

We crawl broadly, resolve the same entity appearing under different names across sources, and deliver a linked set rather than a pile of pages - with every attribute carrying the source it came from.

What Web Data Mining includes

Entity Linking

Connecting disparate data points into a unified knowledge graph.

Global Crawling

Discovery-based mining that finds new targets automatically.

Social Graphing

Mapping relationships and influence across social platforms.

Scientific Gathering

Systematic mining of academic and research repositories.

Technical Protocol

  • [1]Knowledge Graph Tech
  • [2]Bloom Filter Deduplication
  • [3]Recursive Discovery Engines
  • [4]Graph Database Exports

Common Use Cases

Anti-Fraud DetectionSupply Chain MappingAcademic ResearchOSINT Investigations

Enterprise Standard

99.9% Extraction Reliability

What you get

Every record carries these fields

The schema is yours to change — add fields, drop fields, rename them to match what your systems already read. This is the default.

FieldTypeWhat it is
entity_idstringResolved across every source it appeared in.
namestringCanonical, with variants kept.
entity_typeenumcompany | person | place | product.
attributesobjectWhatever was found, each with its own source.
linked_entitiesarrayThe relationships, which are usually the point.
sourcesarrayEvery URL the record draws on.
first_seentimestampWhen it entered the set.
last_seentimestampLast confirmed. Stale records are visible.
Sample output

One record, as it arrives

JSON — or a graph export where the relationships matter most.

Values here are illustrative. On a real sample these are your products, from your target sites — see below.

record.json
{"entity_id": "org_55120","name": "Example Plumbing Ltd","entity_type": "company","first_seen": "2026-03-02T11:00:00Z","last_seen": "2026-10-14T23:50:00Z",}
Who buys it

What people use it for

B2B and lead generation

Building a sector list from scratch, with the source behind every field.

Supply chain and risk

Mapping suppliers and their relationships across public sources.

Market research

Sizing a market where no published list of participants exists.

What it costs

Three ways to buy, and what moves the number

Fixed price

A defined one-off project with a known scope and a delivery date.

Retainer

Continuous collection, weekly or monthly. Selector fixes after a site changes are included.

Dedicated developer

A scraping engineer working inside your team, for roadmaps rather than deliverables.

What drives the quote for web data mining

  • How broad the discovery is - a defined sector costs far less than "everything adjacent".
  • How hard entity resolution is, which is where most of the effort goes.
  • Attribute depth per entity.
  • Refresh cadence, since a mined set goes stale faster than people expect.

How we price it →

Your options

Build it, buy a tool, or have it run

All three are reasonable. Which one fits depends on how much of this you want to own.

Build it in-house

Starting from a list is collection. Starting from a question - "every supplier in this sector" - is discovery, which means crawling broadly, deciding what counts, and resolving duplicates with no reference list to check against. Most in-house attempts stall at deduplication.

Buy a self-serve tool

Tools take URLs. Mining has no URLs at the start; producing them is the work. A tool can collect what you found, but it cannot tell you what you have not found yet.

Have it run for you

We run the discovery and the entity resolution, and deliver a linked set where every attribute carries the source it came from - so a disputed field can be checked rather than argued about.

Honestly

What goes wrong, and what happens then

Collection breaks. The question worth asking a supplier is not whether it breaks but whether you find out from them or from your own data.

The same organisation appears under a trading name, a legal name and a domain.

What it costs you

Your sector list has three entries for one company, and every count built on it is wrong.

What we do

Entities are resolved across sources with every variant and source kept, so a merge can be audited and undone.

Discovery drifts into adjacent sectors.

What it costs you

The set grows, looks impressive, and stops being the thing you asked for.

What we do

Inclusion criteria are agreed before the crawl and applied as a filter, with borderline entities surfaced rather than silently kept or dropped.

An entity stops being findable.

What it costs you

A set that only ever grows slowly fills with companies that no longer trade.

What we do

last_seen is on every record. A set that is not refreshed shows its own age instead of looking current.

Before you ask

What we will not do, for this service

  • Build people-level datasets - entities here are companies, places and products.
  • Collect contact details that are not published for that purpose.
  • Present an inferred relationship between entities as a confirmed one.

The full terms →

How the work runs

The same six steps on every engagement, whether it is one site or ten thousand.

  1. STEP 01

    Interaction

    Hand-shake connection has been established between potential customer and Business Team.

  2. STEP 02

    Consultation

    Potential customer exchange information and Business Team consult them with expert advice.

  3. STEP 03

    Evaluation

    Compliance & Technical specialist examines the identified business need of potential buyer.

  4. STEP 04

    Execution

    Potential Buyer confirms the compliance details to allow Engineering team to start Execution.

  5. STEP 05

    Disposition

    Delivery team dispose off the Exemplary Service to ensure the customer delight.

  6. STEP 06

    Exceed Expectation

    Build a strong long term relationship with a client by exceeding their expectations at every stage.

Questions people ask about web data mining

Is web scraping legal?

Collecting information that is published publicly is generally lawful, but it depends on the site, the data and where you operate. We only collect publicly available data, we don't gather personal or identity information, and we won't take on work that needs us to get around a site's access controls or break its terms of use. If a request falls the wrong side of that line, we'll tell you before we start rather than after.

Who owns the data you collect for me?

You do. The datasets we produce for you are yours. We do not resell them and we do not reuse them for other clients.

Do you sign NDAs?

Yes, routinely, and we are happy to work under your paperwork rather than ours. We can also sign a data processing agreement where your engagement involves us handling data on your behalf.

What formats can you deliver in?

JSON, CSV, Excel, XML, or straight into a database. We can drop files to S3, SFTP or Google Drive, push to a webhook, or expose the data through a REST API. Tell us what your systems already read and we will match it.

Do you work with small businesses, or only enterprises?

Both. We have built one-off datasets for solo founders and continuous pipelines running millions of pages a day for larger teams. The approach is the same, the scale is not.

Every question we are asked →

Before you commit

Request a free sample 100 rows within 24 hours

Send a target URL and the fields you want. We return 100 real extracted rows within 24 hours so you can check the structure and the quality yourself, before any money changes hands.

See pricing
  • Send one URL and the fields you want.
  • 100 real rows back within 24 hours.
  • No payment, no commitment, no contract first.
  • If it cannot be collected properly, we say so.
Related reading

More on this, at length

Other things we build