Deepcrawl

An LLM can read a web page, but one page is not a website. Before an agent can research documentation, compare products, or answer a question, it needs to know what other pages exist and how they relate to one another. Without that map, it either guesses where to go next or spends tool calls opening pages blindly.

The raw material is messy. A page exposes a flat mix of relative links, duplicate URLs, fragments, tracking queries, subdomains, external sites, images, documents, and framework assets. A plain list loses the site's shape, while recursively opening every URL is slow, expensive, and unbounded.

At the time, I could not find a lightweight, open tool that returned a useful site map in one request without depending on a sitemap, an llms.txt file, a paid crawler, or a headless browser. That missing primitive became the center of Deepcrawl.

I designed Deepcrawl around a bounded high-fan-out discovery algorithm paired with a deterministic tree compiler. Instead of recursively crawling every URL it finds, the algorithm selects a small set of structurally useful vantage points—the target page, site root, ancestor paths, and a bounded batch of shallow descendants—then gathers the much larger set of links those pages expose.

This is the key tradeoff: discover broadly without reading everything. A site's root page and shallow section pages are usually link-dense. Fetching them together can reveal many deeper destinations, while the agent remains free to read only the pages that matter to its task.

I built this pipeline from scratch. The first public implementation already contained its full shape: direct extraction, normalized link sets, bounded parallel expansion, and path-based tree construction. Later work refined caching, platform scoping, typed contracts, and the product around it without changing that core idea.

The simplified excerpt below shows the core fan-out loop from the worker. Each batch is capped before it starts; every selected page passes through a request-local visited check; Promise.allSettled contains independent failures; and successful results are normalized and merged into shared Sets. Cache guards and error reporting are omitted only to keep the mechanism readable.

The fast path uses direct HTTP fetches and Cloudflare's HTMLRewriter instead of starting a browser. As links stream through the parser, Deepcrawl resolves relative paths, removes fragments and—by default—query parameters, filters framework resources, validates each URL, classifies internal, external, and media links, and deduplicates them in Sets.

The target page, the site root, and relevant ancestor paths start in parallel. Within the selected ancestor and descendant work, Deepcrawl fans out across bounded batches—currently at most 30 URLs per batch—concurrently. Every fetched page contributes its own links to the shared Sets, so one controlled batch can reveal far more URLs than it directly fetches. Promise.allSettled keeps one broken page from discarding the rest of the map.

In the demo below, Extract Links discovers 528 URLs from cloudflare.com in 2.42 seconds; the API response itself arrives in 637 milliseconds.

The tree is cached by site root. When a cached tree exists and full cleaned content is not requested, previously visited root and nearby pages can be skipped while genuinely new links are merged into the existing structure. A later request can therefore extend the map instead of paying to rebuild it from zero.

To test the fast path against an established crawler, I ran Deepcrawl and Firecrawl on the same public URL: https://nodejs.org. In that run, Firecrawl completed in 22.38 seconds and Deepcrawl in 86 milliseconds—a 260.20× difference.

One run is not a universal performance guarantee: cache state, network conditions, and requested output can change any comparison. But the result demonstrates the engineering tradeoff behind Deepcrawl—direct HTTP fetching, streaming extraction, and bounded discovery can return useful agent context without the compute and latency of starting a browser.

A deduplicated URL array is still not a map. I wrote the tree builder to treat each normalized URL as a path. It splits subdomains and path segments, creates missing intermediate nodes, and uses a Map keyed by full URL so existing branches are reused instead of duplicated.

For example, /docs/getting-started and /docs/api/auth become one docs branch with getting-started and api beneath it, even if /docs/api was never exposed as a standalone page. Folder-first ordering makes dense sections readable, while discovery order can be preserved to retain the source site's own emphasis.

Each node can carry its URL, metadata, visit time, outgoing links, errors, and children. The result is recursive, typed JSON rather than a visual-only sitemap, so an agent can traverse it directly as if it were looking through a project's file tree.

This changes the context-engineering workflow. The agent asks for the site tree first, sees the available sections and pages, chooses the smallest useful set, and only then calls the read API for clean Markdown. Navigation context and document context become two separate layers instead of one large HTML dump.

In the cached demo below, Read URL returns the Cloudflare Developer Docs as clean Markdown with a 3-millisecond API response.

The tree does not make the agent smarter by itself. It gives the agent better evidence about where information lives, while reducing the need to load irrelevant pages into its context window. That makes web research more deliberate, inspectable, and easier to control.

Deepcrawl is intentionally scoped to links exposed by ordinary public HTML. It does not claim to bypass anti-bot systems or guarantee an exhaustive inventory of client-only navigation. The goal is a fast, useful map of what the site actually reveals—not a false promise of seeing everything.

I built the API, TypeScript SDK, shared contracts, Cloudflare services, authentication, usage controls, logs, documentation, dashboard, and playground as one product. The same typed links contract runs from the extraction service through the SDK to the interface where developers inspect the tree.

Developers can configure content formats, caching, metrics, and other request options visually, with a TypeScript SDK example shown directly below the controls. The playground makes the path from exploration to integration explicit.

The documentation is part of the open-source implementation, not an afterthought. It covers quick starts, setup and authentication, endpoints, integrations, self-hosting, and API reference in a searchable site, with each page available as Markdown for developers and agents.

Keeping the whole stack open was part of the point. Developers can inspect the algorithm, self-host it, or use only the pieces they need instead of depending on a closed crawler.

Deepcrawl found an audience without paid promotion. An established open-source builder shared it, and the project picked up an unexpected wave of attention from developers who recognized the same problem.

That response showed me that a narrow infrastructure problem can travel when the product is easy to understand and useful on its first request.

Deepcrawl reflects how I like to build infrastructure: find the missing primitive, make the core algorithm explicit, keep the cost of each decision bounded, and return structured evidence that another system can reason over.