A new wave of jobs data vendors is making the same pitch: legacy providers are slow and expensive, AI-powered pipelines are faster and cheaper. If you run a job board with paying employer relationships, the question that actually matters is whether it works reliably every day, without you finding out from a customer that something broke. This is about how to evaluate vendors so a data problem doesn’t become a customer problem.
Two Different Problems
Job boards typically need jobs data for one of two reasons, and the failure modes are different.
Dedicated scraping: serves a paying employer or agency relationship. Jobs need to appear quickly, update throughout the day, and be filtered precisely, with nothing duplicated or stale. The main risk is that a broken source or a missed update reaches a paying advertiser directly.
Scale data is about breadth: jobs across sectors and geographies where you have no direct relationship. Quality still matters, but the risks are different: inflated counts, scam content, and unclear provenance rather than a single broken feed.
A vendor good at one isn’t necessarily competent at the other. A pitch that conflates them should put you on alert.
| Dedicated Scraping | Scale Data | |
|---|---|---|
| Who it's for | A paying employer or agency | Breadth, no direct relationship |
| Speed needs | Same-day, multiple updates a day | Consistent, less time-sensitive |
| Main risk | Broken source hits a paying advertiser | Inflated counts, scam content |
| "Good" looks like | Fast detection and fixes, no dead postings | High direct-source share, real dedup |
When You Have an Employer or Agency Relationship
When an employer pays to be on your platform, the scraping relationship is effectively part of your service contract, whether or not it’s written that way.
Collection frequency. A once-daily scrape isn’t enough for high-volume employers posting throughout the day. Ask what the collection frequency is for high-volume sources, and whether it’s configurable per client.
Broken source recovery. Every career site breaks eventually: an ATS update, a CDN change, a new login wall. Automated monitoring can flag the problem, but fixing it usually still needs a human. Ask about monitoring, typical resolution time once flagged, and who notifies you.
ATS migrations. Employers switch ATS providers (Workday to Greenhouse, Taleo to iCIMS) regularly across a client base, and a scraper built for the old system just stops working, often leaving stale postings live. Ask how migrations are detected and how long reconfiguration takes.
Evergreen jobs. Standing postings that get refreshed on a cycle can create duplicates or inflate counts if handled badly. Ask about deduplication logic for persistent postings.
Abandoned postings. Jobs get filled or pulled without notice. Ask how removals are detected, how fast the feed clears, and how a temporary server error is distinguished from a real removal.
When You Need Scale Data
Scale data is broader and harder to audit, since problems are less visible until they compound.
Source integrity. Data collected directly from employer sites is different from data pulled from other job boards and aggregators, already mixed, already duplicated, carrying someone else’s lag and errors. Ask what percentage of coverage comes directly from employer sites versus aggregators. A vague answer tells you something.
Duplication at scale. One job legitimately appearing on a career site, LinkedIn, Indeed, and several staffing agencies is still one job. Real samples from vendors claiming “enterprise-grade” dedup have shown a single role spread across 20+ sources and counted as 20 postings, or one employer under seven name variants. Inflated counts mean your platform looks richer than it is, until a user finds the same job twice.
Scam and spam content. At scale, quality becomes a moderation problem: work-from-home schemes, MLM listings, fake recruiter posts. Ask what scam detection looks like, whether it’s automated or human-reviewed, and for examples of what gets caught versus what slips through.
The aggregator-of-aggregators problem. Some vendors present themselves as primary sources when their pipeline is really built on other aggregators. Watch for implausibly high coverage claims for a given geography, and check whether the same posting shows different collection timestamps across “independent” sources, a sign of layered aggregation.
Why this matters beyond your own platform: Alexander Chukovski has tracked Google’s ongoing freeze on Indexing API quota increases, now in its third and longest stretch since October 2025 (Chukovski, 2026). One theory he raises is that Google clamped down after its approval pipeline was gamed by “indexing services” pushing non-job content through an API meant only for job postings, leaving it, in John Mueller’s words, “inundated by bloggers trying to act like legitimate sites.” Either way, the lesson holds: a well-structured, crawlable site with clean markup still gets discovered by Google Jobs on its own, and Chukovski’s own read is that when the freeze lifts, “document quality is exactly what decides who gets approved.” The same discipline that keeps a feed free of duplicates and fake postings is what determines whether Google trusts your jobs enough to show them at all.
Questions That Cut Across Both
1. Is collection consistent?
Not just frequent, but predictable enough to build reliable processes around.
2. Who’s accountable when something breaks?
Not a support ticket queue, a named person or team with a defined response time.
3. Can filtering be configured for your use case?
By category, location, employer type, posting date, and source.
4. What does their monitoring actually look like?
Vendors serious about reliability can tell you uptime per source and have alerts that fire before a client notices.
On the AI Pitch Specifically
Machine learning applied to classification, entity resolution, or parsing messy postings is genuinely useful, and vendors using it well can describe specifically what the model does and how errors get caught. Vendors using AI mainly as a marketing position will talk about “99.999% automated” pipelines that “audit every data point.” Test those claims against the operational questions above.
A packing job listed at $71,000 an hour isn’t a classification problem; it’s a missing sanity check. An employer under seven name variants isn’t a scraping problem; it’s an entity-resolution problem that needs a maintained reference dataset. An ATS migration that breaks a scrape doesn’t resolve itself because the system is labeled AI-powered. The hard parts of jobs data are hard because they require sustained attention, not because the technology is primitive.
Red flags in a vendor pitch
- “99.999% automated” with no mention of human escalation
- Coverage or job-count claims that seem implausibly high
- A vague or evasive answer on direct-source percentage
- No named person or team for when something breaks
- Job counts that don’t match across their website, docs, and sales deck
The Checklist
None of these are unreasonable questions. Vendors with the operational infrastructure they claim will answer specifically and confidently. Those who redirect to capability decks and case studies are telling you something worth knowing before you commit.
Aspen Tech Labs builds jobs data infrastructure for job boards, staffing firms, and labor market intelligence platforms, including JobsIndex, a job feed sourced directly from over 300,000 employer career sites and ATS systems, and custom scraping for dedicated employer and agency relationships. If you’re currently running scrapes or buying a jobs feed and want an independent look at data quality or coverage gaps, we’re offering readers a free audit: no sales process, just an honest assessment of what you have and where the gaps are.
Reference
Chukovski, A. (2026). Why Did Google Stop Approving Indexing API Requests and Quota Increases? alexanderchukovski.com.
See this data in action
Explore structured job data and labor market intelligence for your use case.
