scanner
द्वारा google-gemini
एक वेबसाइट को गहराई से स्कैन करता है, HTML पृष्ठों को मार्कडाउन में बदलता है, robots.txt का सम्मान करता है, और स्नैपशॉट लॉग को अपडेट करता है।
npx skills add https://github.com/google-gemini/gemini-managed-agents-templates --skill scannerScanner Skill
Use this skill to scan and analyze all relevant pages under a target website domain to build or refresh a local customer support corpus.
Embedded Script
python skills/scanner/scripts/scan.py <URL> [--force]
Arguments
| Argument | Description |
|---|---|
<URL> | The start/seed URL of the website to analyze (e.g. https://example.com) |
--force | Force scanning and bypass the 24-hour cache check |
Features
- Robots.txt Compliance: Checks
robots.txtbefore parsing. If restricted, aborts scanning. Make sure to perform scanning only on sites that are allowed. - Domain-Locked Recursive Scanning: Only analyzes links within the same domain/subdomain to avoid leaking to other websites.
- HTML to Markdown: Converts HTML structure into clean, readable Markdown text suitable for LLM document matching.
- Caching & Snapshot Maintenance: Creates or updates
.agents/workspace/snapshots.jsonwith mapping and timestamp. - Corpus Directory Index: Automatically generates
.agents/workspace/pages/index.mdlisting and explaining all files in a structured, clickable table format so agents can locate matching topics immediately.