Scrapfly

官方

Scrapfly MCP Server 為 AI 代理提供一個簡單、統一的方式來抓取即時網頁資料,並內建反機器人處理功能。

文件

Getting Started with Scrapfly MCP Server

The Model Context Protocol (MCP) is an open standard that allows AI models to interact with external tools and data sources. The Scrapfly MCP Server transforms your AI from being limited by training data to having real-time access to any website-giving your AI the power to scrape live web data on demand.

MCP Cloud or Self-Hosted

The Scrapfly MCP server runs two ways, with the same tools. Pick one:

AspectMCP Cloud (managed)Self-hosted
Where it runsHosted by Scrapfly at https://mcp.scrapfly.io/mcpYour machine or servers, from the open-source server
InstallNothing: add the URL to your AI clientBuild the Go binary or the Docker image
AuthenticationOAuth2 sign-in from the client, or your API keyYour API key
UpdatesAlways the latest versionYou pull and rebuild
Best forMost users: Claude, Cursor, VS Code, ChatGPT and other clientsCustom tools, air-gapped or audited environments

MCP Cloud

Add the server URL to your AI client. Clients that support OAuth2 open a Scrapfly sign-in on first use; others take your API key (see Authentication).

{
  "mcpServers": {
    "scrapfly-cloud-mcp": {
      "type": "http",
      "url": "https://mcp.scrapfly.io/mcp"
    }
  }
}

A client that only starts a local process connects through mcp-remote:

{
  "mcpServers": {
    "scrapfly": {
      "command": "npx",
      "args": ["mcp-remote", "https://mcp.scrapfly.io/mcp"]
    }
  }
}

Each client has its own steps: see the setup guides (Claude Code, Claude Desktop, Cursor, VS Code, ChatGPT and more).

Self-Hosted

Build the open-source server, written in Go:

git clone https://github.com/scrapfly/scrapfly-mcp
cd scrapfly-mcp
go build -o scrapfly-mcp ./cmd/scrapfly-mcp

Local client (stdio): the client starts the binary with your API key in its environment.

{
  "mcpServers": {
    "scrapfly": {
      "command": "/path/to/scrapfly-mcp",
      "env": { "SCRAPFLY_API_KEY": "" }
    }
  }
}

HTTP server: started without a key, each caller passes their own, as Authorization: Bearer <key> or ?key= on http://<host>:8080/mcp.

./scrapfly-mcp -http :8080

# or with Docker
docker build -t scrapfly-mcp .
docker run -p 8080:8080 scrapfly-mcp

A key set in SCRAPFLY_API_KEY together with -http serves every caller with that key and no authentication, so the server binds 127.0.0.1 unless you pass -allow-remote. Every option is in the README.

What This Unlocks

Instead of your AI answering "I don't have current information," it can now:

Scrape Live Data

Pull current prices, listings, news, or any webpage content in real-time

Bypass Anti-Bot Systems

Automatically handle CAPTCHAs, proxies, JavaScript rendering, and rate limits

Extract Structured Data

Parse complex websites into clean JSON using AI-powered extraction

Capture Screenshots

Take visual snapshots of pages or specific elements for analysis

Why Scrapfly MCP Server?

Built on battle-tested infrastructure used by thousands of developers:

Always Reliable

Your AI never gets blocked

Fast Setup

Running in under 60 seconds

Secure

OAuth2 + enterprise auth

The Power of MCP Connectors

MCP is a universal connector that turns any AI client into a web scraping powerhouse. Instead of building custom integrations for each AI platform, MCP provides a single standard interface.

Works With Your Favorite AI Tools:

  • Claude Desktop, Cursor, Cline, Windsurf
  • LangChain, LlamaIndex, CrewAI
  • OpenAI function calling, custom agents
  • n8n, Make, Zapier workflows

One Integration, Infinite Possibilities:

Configure once, use everywhere. The same MCP server works across all platforms-no need to rebuild integrations for each AI tool you adopt.

How to Use Scrapfly MCP

Getting started is simple-choose your AI client and follow the setup guide:

[### Setup for Your AI Client

15+ integrations: Claude Desktop, Cursor, Cline, Windsurf, LangChain, n8n, and more

](https://scrapfly.io/docs/mcp/integrations) [### Available Tools

9 tools: scraping, screenshots, Cloud Browser with dynamic WebMCP tool discovery

](https://scrapfly.io/docs/mcp/tools) [### Authentication Options

API key, OAuth2, or header-based auth-choose what fits your needs

](https://scrapfly.io/docs/mcp/authentication)

Next Steps

See Real-World Examples - Job aggregation, price monitoring, data extraction, and more Client-Specific Setup Guides - Cursor, Cline, LangChain, n8n, and all supported platforms Tools & API Specification - Complete parameter documentation and usage guides Set Up OAuth2 Authentication - Secure, enterprise-grade authentication for production Read the FAQ - Common questions and troubleshooting

Verify Your Setup

After configuration, verify everything works correctly:

  1. Check Tool Availability

    Look for the hammer icon (🔨) in your AI client or check tool listings to confirm Scrapfly tools are loaded.

  2. Test a Simple Request

    Try asking: "What are the top posts on Hacker News right now?"

    Your AI should automatically use Scrapfly tools to fetch and display current results.

  3. Check Account Usage

    Ask: "How many Scrapfly credits do I have left?"

    The AI will use info_account to show your current usage and quota.

Troubleshooting

If you encounter issues, here are common problems and solutions:

Tools Not Detected

  • Verify JSON config - Ensure your configuration has no syntax errors (missing commas, brackets)
  • Restart your AI client - Many clients require a full restart to load new MCP servers
  • Check npx availability - Run npx --version in your terminal to verify it's installed
  • Validate API key - Ensure your Scrapfly API key is correct and has available credits

Authentication Errors

  • Invalid API key - Double-check your key at Dashboard → Project Settings
  • Network unauthorized / Access restricted by your project - The project only accepts specific IP addresses and the key is fine. See Troubleshooting
  • OAuth2 issues - Ensure you've completed the OAuth flow and tokens are refreshed
  • Expired credentials - Re-authenticate if your session has expired

Need More Help?

Check out our detailed troubleshooting resources: