Web pages, JSON-shaped. For training, RAG, and agents.
LLMs need clean, structured input. Raw HTML wastes tokens and pollutes embeddings. Convert any URL into structured JSON or LLM-optimised Markdown - built for fine-tuning, RAG, evaluation, and agentic browsing.
The LLM data pipeline challenge.
You can’t feed raw web pages to a model and expect great answers. Token waste + schema chaos = bad RAG, expensive inference, fragile fine-tunes.
Raw HTML is incompatible with LLM pipelines. Extensive preprocessing before any model can use it.
Nav menus, ads, boilerplate HTML burn precious context-window tokens with zero informational value.
Data varies wildly across sources. Impossible to build reliable structured datasets without constant reconciliation.
Millions of pages for training data = robust extraction that handles anti-bot and rate limits without breaking.
How AI teams use it.
Build knowledge bases that actually ground answers.
Extract clean, structured content from docs sites, knowledge bases, authoritative sources. Convert to token-efficient Markdown that preserves heading hierarchy and semantic structure. BM25 filtering extracts only domain-relevant passages - lower storage cost, higher retrieval precision. Feed Pinecone, Weaviate, Chroma.
- ~60% hallucination reduction with grounded knowledge bases
- ~70% token cost reduction vs raw HTML
- 100,000+ pages indexed with consistent structure
- Automated re-extraction keeps KBs current
Generate training datasets at scale.
Extract structured fields with extract_rules mapped to your training format - Q&A pairs, instruction-response examples, or domain entities. Validate the returned JSON against your own Pydantic or JSON Schema. Scrape API + Markdown API for diverse, well-formatted corpora.
- 50,000+ validated training examples from web sources
- Schema consistency via automatic JSON validation
- Dataset prep cut from weeks to hours
- Higher fine-tuned model accuracy from clean data
Build evaluation benchmarks from real sources.
Extract ground-truth data from authoritative web. Government databases, academic repos, reference sites with structured schemas capturing entities, relationships, attributes. Schema validation ensures benchmark quality. Auto-refresh keeps evaluations against current information.
- Domain-specific evaluation suites with verified ground truth
- Automated benchmark refresh for continuous evaluation
- Compare model performance across extraction complexity
- Detect model regressions early via structured tests
Give agents a usable data layer.
When autonomous agents browse the web, raw HTML is unusable. Provide clean Markdown for reading comprehension and structured JSON for data extraction. extract_rules pull exactly the fields the agent needs. Consistent output regardless of source layout - fits in context windows.
- Agents process web content ~10x faster than raw HTML
- ~75% reduction in agent token consumption
- JS-heavy sites + anti-bot handled transparently
- Consistent data structure regardless of source
Convert any web content into LLM-ready data.
Each source type maps to a recommended extraction strategy.
API references, tutorials, guides. Structured Markdown preserving code blocks, parameter tables, heading hierarchy for RAG.
Enterprise wikis, Confluence, KMS. Articles with metadata, categories, cross-references for LLM training.
Abstracts, methodologies, findings, citations from research repositories and preprint servers. Domain-specific scientific KBs.
Names, descriptions, specifications, pricing, reviews with validated JSON schemas. Product intelligence for e-commerce AI.
Articles with structured headlines, authors, dates, body text. Temporal knowledge bases for current events.
Regulations, public records, policy documents. Complex legal + regulatory content for compliance and legal AI.
Q&A from Stack Overflow, Reddit, community forums. Structured Q&A pairs for fine-tuning conversational + technical models.
Articles, code tutorials, engineering posts with preserved formatting. Clean Markdown with intact code snippets.
Three steps from web page to LLM-ready data.
Define your fields
Decide the fields you need. For RAG: Markdown API with BM25 filtering. For structured JSON: Scrape API with extract_rules - CSS selectors mapped to field names, with nested objects and arrays. Validate the response against your own JSON Schema or Pydantic model.
Extract and validate
Submit URLs. JS rendering, anti-bot, rate limits handled automatically. Scrape API returns JSON matching your extract_rules. Markdown API strips boilerplate; BM25 filtering scores passages against your relevance queries.
Feed your pipeline
JSON or Markdown via API response or webhook. Push to vector DB for RAG, format as JSONL for fine-tuning, pipe Markdown into context windows. Token counts, timing, validation warnings in metadata.
Drop a URL in.
See schema-validated structured data come back - ready for vector store, fine-tuning, or context window.
curl 'https://api.ujeebu.com/scrape' \
-H 'ApiKey: YOUR_API_KEY' \
-G \
--data-urlencode 'url=https://docs.python.org/3/tutorial/datastructures.html' \
--data-urlencode 'extract_rules={"title":"h1","sections":"h2","code_examples":"pre"}'Built for production LLM data pipelines.
Structured JSON output
extract_rules map CSS selectors to field names, returning predictable JSON with nested objects and arrays. Deterministic and repeatable - validate the response against your own JSON Schema or Pydantic model.
Token-optimised Markdown
Clean Markdown preserving semantic structure. Navs/ads/footers/boilerplate stripped. Heading hierarchy, code blocks, tables, lists preserved. ~70% token reduction vs raw HTML.
BM25 content filtering
Provide a query, receive only the passages most relevant. Dramatically reduces noise for RAG KBs. Pre-filter content before it hits your vector DB.
Batch processing
Thousands of URLs in parallel with consistent schema enforcement per page. Build large-scale datasets without infra complexity. Proxy rotation + CAPTCHA solving + JS rendering included.
Deterministic extraction
extract_rules return the same JSON every time, with no LLM variability. Define rules once, reuse across matching pages. Parse the response straight into your Pydantic models or dataclasses for local validation.
Multi-format output
Choose: schema-validated JSON for entity extraction, clean Markdown for text comprehension, Markdown with citations for research. Each format tuned for specific LLM use cases.
Frequently asked.
What output formats are available?
How do I control the output structure?
Can I match the output to my Pydantic model?
What’s BM25 content filtering and when to use it?
How many pages for training datasets?
How does this compare to traditional web scraping for LLMs?
Start building LLM data pipelines today.
Join AI teams using structured web extraction for RAG, fine-tuning, and agentic workflows.