tavily-crawl

Crawl websites and extract content from multiple pages via the Tavily CLI. Use this skill when the user wants to crawl a site, download documentation, extract…

npx skills add https://github.com/tavily-ai/tavily-cursor-plugin --skill tavily-crawl

Tavily Crawl

Gather content from a site section, not from the open web.

When

  • You need many pages under one site (docs, API reference, a path prefix)
  • Step 4: search → extract → map → crawl → research

Do not use this skill for a single known URL (extract), for URL discovery only (map), or for a cited multi-source report across the web (research).

How

  • Map first when the target section is unclear.
  • Stay narrow: limit depth and page count; constrain to the paths you need.
  • For answering a question, prefer instruction-guided, chunked crawls over dumping every page into context.
  • For saving a docs tree onto disk, do that only in a terminal with a real filesystem. On a connector host, return the crawled content instead.
  • Cite the pages you used. Always cap the crawl so it cannot run away.

Execute

If tavily_crawl is callable (the name may be prefixed), call it. Do not install the CLI and do not run tvly for this. In a coding-agent terminal that is already using tvly, pass --client-name "cursor plugin" and take flags from tvly crawl --help. Never ask for an API key in chat.

More skills from tavily-ai

research
tavily-ai
Comprehensive research on any topic with automatic source gathering, analysis, and citations. Conducts multi-source web research with explicit citations, ideal for comparisons, current events, market analysis, and detailed reports Offers three model options: mini for targeted single-topic research (~30s), pro for comprehensive multi-angle analysis (~60-120s), and auto for API-driven complexity detection Authenticates via OAuth through Tavily MCP server with automatic browser-based login on...
search
tavily-ai
Web search with LLM-optimized results, relevance scoring, and flexible filtering. Supports four search depth modes (ultra-fast, fast, basic, advanced) with configurable latency and relevance tradeoffs Includes domain filtering, time range constraints, date ranges, country boosting, and raw content extraction Returns results with title, URL, content snippet, and relevance score; optional image results and favicons Automatic OAuth authentication via Tavily MCP server or API key configuration;...
tavily-best-practices
tavily-ai
Web search API for LLMs with real-time data access, content extraction, site crawling, and AI-powered research. Five core methods: search() for web results, extract() for URL content, crawl() for site-wide extraction, map() for URL discovery, and research() for end-to-end AI synthesis Supports Python and JavaScript SDKs with async clients for parallel queries and configurable search depth (ultra-fast/fast/basic/advanced) Crawl method accepts semantic instructions to focus extraction on...
tavily-cli
tavily-ai
Web search, content extraction, site crawling, and deep research via Tavily CLI. Five command modes covering search, extraction, URL discovery, bulk crawling, and multi-source research with citations All commands support JSON output and file saving for structured, agentic workflows Escalation pattern guides you from simple search through extraction, mapping, crawling, to comprehensive research based on your needs Requires tavily-cli installation and API key authentication via tvly login
tavily-crawl
tavily-ai
Multi-page website crawler with semantic filtering and markdown export. Crawl entire site sections with depth and breadth control; filter by path regex, domain, or natural language instructions to focus results Save each page as local markdown files via --output-dir , or return structured JSON for agentic processing Use semantic instructions with chunk extraction to prevent context bloat when feeding results to LLMs; use full-page extraction for offline documentation downloads Supports...
tavily-dynamic-search
tavily-ai
Search the web, filter results, and extract content so that raw search data never enters your context window . Only your curated print() output comes back.
tavily-extract
tavily-ai
Extract clean markdown or text from up to 20 URLs, with JavaScript rendering and query-focused chunking support. Handles JavaScript-rendered pages with configurable extraction depth (basic for simple pages, advanced for dynamic SPAs and tables) Supports query-focused extraction to return only relevant content chunks instead of full pages Returns LLM-optimized markdown by default, with options for plain text format and structured JSON output Processes up to 20 URLs in a single call;...
tavily-research
tavily-ai
Comprehensive AI-powered research with multi-source synthesis and citations. Produces structured reports grounded in web sources, taking 30-120 seconds depending on model selection (mini for targeted queries, pro for complex comparisons) Supports multiple output formats: markdown reports, JSON with custom schemas, and configurable citation styles (numbered, MLA, APA, Chicago) Includes async workflow for long-running research via --no-wait , status , and poll commands, plus real-time...