Community

Jevcrawl: AI-Focused Web Crawler for Efficient Information Retrieval

Jevcrawl is a specialized web crawler for AI agents that extracts only necessary information to answer specific questions, reducing cost and computational overhead.

AI-generated editorial illustration of the article topic; not a photograph, product screenshot or measured result.

Jevcrawl's Approach to Web Crawling

Jevcrawl is designed as a specialized web crawler for AI agents, focusing on extracting only the information necessary to answer a specific question. Unlike traditional crawlers that retrieve large volumes of data, Jevcrawl uses a small decision model to select relevant links and passages. This approach ensures that the agent's LLM processes only the content that directly addresses its query, reducing both cost and computational overhead. According to the project's description, this method allows for crawling at a cost of just a few cents per query. When a goal is defined, Jevcrawl prioritizes relevant passages; in the absence of a goal, it retrieves all content without invoking the Jev call. This targeted strategy aims to enhance efficiency and precision in AI-driven information retrieval tasks.

Benchmark Performance and Comparison

In benchmark tests, Jevcrawl demonstrated a 54% task completion rate compared to Claude's 33% when selecting links, and it performed twice as often as Crawl4AI's goal-directed crawl in fetching relevant pages. Every system began on a site's home page with the same 30-page budget and identical question, ensuring a fair comparison. Jevcrawl's BestFirstCrawlingStrategy, which uses a keyword scorer to prioritize pages likely to contain answers, enables it to navigate toward relevant content without relying on an LLM. This approach was evaluated against others to determine its effectiveness in fetching the page that holds the answer. The results highlight Jevcrawl's efficiency in achieving its goals with a non-LLM-based strategy, as noted in the benchmarks.

Tools and Usage Examples

Jevcrawl is a specialized web crawler designed for AI agents, with no large language model (LLM) involved in the crawling process itself. It provides tools like crawl(goal, start_url, max_pages=30, max_seconds=120, return_tokens=4000, max_decision_usd=0.05), which allows users to define a goal and starting URL, and automatically navigates up to 30 pages within 120 seconds, returning up to 4000 tokens of content. Another tool, map(start_url, goal, limit=50, max_pages=10), generates a ranked list of URLs based on relevance to the goal, helping users decide what to scrape next. These tools are also accessible via the command line, with commands like jevcrawl crawl and jevcrawl map, offering flexibility for developers and researchers. Jevcrawl operates by making decisions, not writing content, ensuring a clear separation between navigation and content generation.

Configuration and Customization

Jevcrawl’s configuration and customization options are designed to provide flexibility without compromising performance. The tool relies on environment variables for key settings, such as the TYPESAFE_API_KEY (required), JEVCRAWL_BASE_URL (default: https://api.typesafe.ai), and JEVCRAWL_MODEL (default: jev-1.13.0). Additional parameters like JEVCRAWL_CACHE_DIR allow users to specify where cached answers are stored. Beyond these, most settings—such as thresholds, links per page, and concurrency—are defined in jevcrawl/config.py with measured defaults. Python callers can override these defaults by passing their own Config object. A notable constraint is max_decision_usd, which acts as a hard limit: any request that would exceed this budget is not sent, and the crawl halts immediately with a budget error. This ensures cost control remains a priority during execution.

Cost and Legal Considerations

Jevcrawl, a web crawler designed for AI agents without an LLM in the crawl loop, charges users based on input tokens, at a rate of $0.042 per million for the jev-1.13.0 model. According to benchmark data, the average cost of a crawl was $0.011. However, users must be aware that they are solely responsible for ensuring compliance with the terms of service of the websites they crawl, as well as the legal requirements applicable to them, as noted in the project's documentation. The tool also includes type-checking and testing capabilities, with commands such as && uv run mypy --strict jevcrawl and uv run pytest allowing for offline testing with all HTTP requests mocked. These features help developers ensure the reliability and correctness of their implementations before deployment.