UseScraper
A server for web scraping using the UseScraper API.
UseScraper MCP Server
This is a TypeScript-based MCP server that provides web scraping capabilities using the UseScraper API. It exposes a single tool 'scrape' that can extract content from web pages in various formats.
Features
Tools
scrape
- Extract content from a webpage- Parameters:
url
(required): The URL of the webpage to scrapeformat
(optional): The format to save the content (text, html, markdown). Default: markdownadvanced_proxy
(optional): Use advanced proxy to circumvent bot detection. Default: falseextract_object
(optional): Object specifying data to extract
- Parameters:
Installation
Installing via Smithery
To install UseScraper for Claude Desktop automatically via Smithery:
npx -y @smithery/cli install usescraper-server --client claude
Manual Installation
-
Clone the repository:
git clone https://github.com/your-repo/usescraper-server.git cd usescraper-server
-
Install dependencies:
npm install
-
Build the server:
npm run build
Configuration
To use with Claude Desktop, add the server config:
On MacOS: ~/Library/Application Support/Claude/claude_desktop_config.json
On Windows: %APPDATA%/Claude/claude_desktop_config.json
{
"mcpServers": {
"usescraper-server": {
"command": "node",
"args": ["/path/to/usescraper-server/build/index.js"],
"env": {
"USESCRAPER_API_KEY": "your-api-key-here"
}
}
}
}
Replace /path/to/usescraper-server
with the actual path to the server and your-api-key-here
with your UseScraper API key.
Usage
Once configured, you can use the 'scrape' tool through the MCP interface. Example usage:
{
"name": "scrape",
"arguments": {
"url": "https://example.com",
"format": "markdown"
}
}
Development
For development with auto-rebuild:
npm run watch
Debugging
Since MCP servers communicate over stdio, debugging can be challenging. We recommend using the MCP Inspector, which is available as a package script:
npm run inspector
The Inspector will provide a URL to access debugging tools in your browser.
Related Servers
Playwright Server
Automate web browsers and perform web scraping tasks using the Playwright framework.
Web Browser MCP Server
Provides advanced web browsing capabilities for AI applications.
Agentic Deep Researcher
A deep research agent powered by Crew AI and the LinkUp API.
Cloudflare Playwright
Control a browser for web automation tasks using Playwright on Cloudflare Workers.
Playwright Record MCP
Browser automation using Playwright with video recording. Enables LLMs to interact with web pages through structured accessibility snapshots.
MCP YouTube Extract
Extracts information from YouTube videos and channels using the YouTube Data API.
WebDriverIO
Automate web browsers using WebDriverIO. Supports actions like clicking, filling forms, and taking screenshots.
Bilibili Comments
Fetch Bilibili video comments in bulk, including nested replies. Requires a Bilibili cookie for authentication.
HTML to Markdown MCP
Fetch web pages and convert HTML to clean, formatted Markdown. Handles large pages with automatic file saving to bypass token limits.
Web Scout
An MCP server for web search and content extraction using DuckDuckGo.