Tokenectomy Razor

Fast, lightweight Rust-based MCP server designed to optimize prompt payloads and reduce token overhead for LLMs.

Documentation

Tokenectomy Razor

Fast, deterministic log surgery and secret redaction for AI coding agents — purge 90%+ framework noise, redact credentials with O(N) ReDoS immunity, sub-millisecond latency. Written in safe Rust.

Tokenectomy (noun): token + -ectomy (surgical removal) — the precise excision of wasteful tokens from LLM context windows.

High-Performance Log Surgery & Secret Redaction Engine for AI Coding Agents

What It Does

Tokenectomy Razor is an autonomous, machine-to-machine (M2M) Model Context Protocol (MCP) server and stream processing engine written in safe Rust. It intercepts error logs from AI agents, strips 90%+ of framework noise, automatically redacts secrets (JWTs, API keys, database credentials), and caches sanitized contexts with a 24-hour TTL—all without sending raw data to external services.

In 30 Seconds

The Problem:

AI agents waste tokens on framework noise (node_modules, site-packages, .cargo/registry)
Sensitive credentials accidentally leak into LLM logs (AWS keys, database URLs, API tokens)
Repeated identical errors cost money for every retry

The Solution:

Raw Error Log (38K tokens + secrets)
    ↓
[Redact secrets locally] → [Filter framework frames] → [Extract user code]
    ↓
Sanitized Context (2K tokens, no secrets) → Safe to send to LLM

Real Example

Before:

$ cat error.log | head -20
Error in /home/user/.cargo/registry/src-xxx/tokio-1.35/src/runtime/mod.rs:12345
  at /home/user/.cargo/registry/src-yyy/serde/src/lib.rs:456
  Database connection failed: postgresql://admin:secretpass@db.example.com:5432/mydb
  JWT Auth token: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIn0...
  [... 500+ more framework frames ...]

After:

$ cat error.log | razor --scrub
Error in /home/user/src/main.rs:42
  at /home/user/src/utils.rs:18
  Database connection failed: [CONNECTION_STRING_REDACTED]
  JWT Auth token: [JWT_REDACTED]

Benefits:

✅ 95% smaller context (2K vs 38K tokens) → Save money on LLM API calls
✅ Zero secrets in logs → Sleep better at night
✅ Identical errors cached → Second retry costs $0

Technical Highlights

                  ┌──────────────────────────────────────────────┐
   Agent Error     │              TOKENECTOMY RAZOR               │    Sanitized Context
   Dump (38K toks) │  - Polyglot Stack Frame Filter               │ ──►  (2K toks) ──► LLM
  ────────────────►│  - Deterministic Secret Redactor (O(N))      │
                   │  - SHA-256 Idempotency Cache (24h TTL)       │
                   └──────────────────────────────────────────────┘

Polyglot Trace Surgery: In-memory parsing across Rust, Python, TypeScript/JavaScript, and Go. Filters noisy dependency frames and isolates user-written code only. (Java, C/C++, PHP support coming in v1.2)
AI Gateway Reverse Proxy (--proxy): Transparently intercepts prompt streams on 127.0.0.1:8080, performing real-time token excision and credential sanitization before upstream forwarding to OpenAI, Anthropic, or Ollama.
Zero-Knowledge Secret Redaction: Linear-time deterministic regex engine strips JWTs, API tokens, cloud access keys, connection strings, and private keys prior to network transmission. All processing happens locally.
SHA-256 Idempotency Cache: Stores deterministic responses with a 24-hour TTL. Repeated CI/CD or agent loop failures incur zero upstream API cost.
Path Traversal Containment: All MCP filesystem access is canonicalized and locked to the workspace root boundary (CWD). No ../ escapes or symlink breakouts.
M2M Protocol Compliance: Native JSON-RPC 2.0 stdio server compliant with the official Model Context Protocol specification.

Verifiable Benchmarks

Performance metrics are hardware-grounded and reproducible via standalone benchmark suites:

Benchmark Target	Workload Under Test	Verified Measurement	Result
High-Volume Log Redaction	250,000 lines (24.44 MB) enterprise dump containing API keys and connection URIs	333.49 ms (73.3 MB/sec, 749,652 lines/sec)	Pass
ReDoS Resistance	50,000-character pathological backtracking string	1.44 ms (Linear $O(N)$ evaluation)	Pass
Thread Concurrency	100 concurrent OS threads executing simultaneous redaction and extraction	100/100 completed in 27.35 ms (7,312 ops/sec)	Pass
Kernel Memory Footprint	Peak Resident Memory during 250,000-line continuous stress test	76.24 MB VmRSS via `/proc/self/status`	Pass

Understanding the Benchmarks

Metric	Why It Matters	What To Expect
73.3 MB/sec redaction throughput	Most logs are <5MB; you'll redact them in milliseconds	<10ms for typical CI logs
1.44ms ReDoS immunity	Prevents malicious log payloads from DoS'ing your system	Safe to use in production with untrusted input
76.24 MB peak memory	Suitable for constrained CI/CD runners (GitHub Actions, GitLab)	Fits within 256MB limits comfortably
7,312 ops/sec concurrent	Multiple AI agents querying simultaneously	100 concurrent requests handled safely

Reproduce locally:

cargo test --release --test stress_benchmark -- --nocapture

Installation

Method 1: Instant via npx (Recommended for MCP Clients)

No Rust toolchain, native compilation, or manual path setup required:

npx -y tokenectomy-razor --mcp

Or install globally via npm:

npm install -g tokenectomy-razor

Method 2: Cargo (crates.io)

cargo install tokenectomy

Method 3: Build from Source

git clone https://github.com/Tokenectomy-Labs/Tokenectomy.git
cd Tokenectomy
cargo build --release
sudo cp target/release/razor /usr/local/bin/razor

Method 4: Multi-Arch Container (GHCR)

docker pull ghcr.io/tokenectomy-labs/razor:latest
docker run -it ghcr.io/tokenectomy-labs/razor:latest --help

Model Context Protocol (MCP) Integration

Configure Tokenectomy Razor as an autonomous background server across major AI agent environments:

Claude Desktop

Add to ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows):

{
  "mcpServers": {
    "tokenectomy": {
      "command": "npx",
      "args": ["-y", "tokenectomy-razor", "--mcp"]
    }
  }
}

Cursor

Add to .cursor/mcp.json in your project root:

{
  "mcpServers": {
    "tokenectomy": {
      "command": "npx",
      "args": ["-y", "tokenectomy-razor", "--mcp"]
    }
  }
}

Cline / Roo Code / Windsurf / VS Code

Add to your client configuration (cline_mcp_settings.json or settings.json):

{
  "mcpServers": {
    "tokenectomy": {
      "command": "npx",
      "args": ["-y", "tokenectomy-razor", "--mcp"]
    }
  }
}

Google Antigravity CLI

agy mcp add tokenectomy-razor -- npx -y tokenectomy-razor --mcp

Exposed MCP Tools

Tool Name	Capability Description
`get_error_context`	Performs trace surgery on error dumps, removes framework noise, redacts credentials, and extracts relevant local source context bounded to the workspace.
`search_stack_overflow`	Queries Stack Exchange API for relevant error signatures using sanitized search terms.
`apply_code_patch`	Applies atomic file modifications with post-write language syntax verification (`cargo check`, `py_compile`, `node --check`) and automated rollback on validation failure.

Quick Start

I use Claude Desktop

# 1. Add to claude_desktop_config.json (see MCP Integration section above)
# 2. When Claude encounters errors, it automatically uses "get_error_context" tool
# 3. Errors stay sanitized without any additional setup

I use GitHub Actions

# Add to your workflow (.github/workflows/build.yml)
- name: Sanitize Build Failure Log
  if: failure()
  uses: Tokenectomy-Labs/Tokenectomy@v1
  with:
    log-file: 'build.log'
    output-file: 'sanitized.log'

# Now you can safely share sanitized.log without leak concerns

Parameters:

Parameter	Type	Default	Description
`log-file`	String	`''`	Path to raw error log file to process
`log-content`	String	`''`	Direct string content if file is not specified
`output-file`	String	`tokenectomy-sanitized.log`	Path for scrubbed output file
`version`	String	`v1.1.3`	Binary release target version

I want max privacy (air-gapped environment)

# All redaction happens locally—no network calls except to your LLM
cargo install tokenectomy

# Process logs without any cloud services
echo $ERROR_LOG | razor --scrub --local-only

# Or from a file:
razor --scrub --file /var/log/app/error.log > sanitized.log

Advanced Usage

Standalone CLI

In addition to M2M agent mode, Razor provides CLI commands for terminal piping and local shell scripting:

# Scrub framework frames and output clean log
npm test 2>&1 | razor --scrub > sanitized.log

# Sanitize a specific log file
razor --scrub --file /var/log/app/error.log > sanitized.log

# CLI diagnosis with specific AI provider
razor --file error.log --provider openai
razor --file error.log --provider anthropic
razor --file error.log --local-only

AI Gateway Reverse Proxy Mode

Tokenectomy Razor can operate as a transparent local HTTP reverse proxy. It sits between client applications and upstream LLM providers (OpenAI, Anthropic, Ollama, OpenRouter), performing real-time token excision and credential sanitization before upstream forwarding.

Local Development (Default Loopback):

# Forward to OpenAI
razor --proxy --proxy-bind 127.0.0.1:8080 --upstream-url https://api.openai.com/v1

# Forward to local Ollama instance
razor --proxy --proxy-bind 127.0.0.1:8080 --upstream-url http://127.0.0.1:11434/v1

Point any standard SDK or IDE client to the local proxy:

export OPENAI_BASE_URL="http://127.0.0.1:8080/v1"
# Now all API calls are automatically sanitized

Production Proxy Hardening

Binding to external interfaces (0.0.0.0) requires explicit token authorization:

razor --proxy --proxy-bind 0.0.0.0:8080 --upstream-url https://api.openai.com/v1 --allow-remote --proxy-token "YOUR_SECURE_TOKEN"

Resource limits enforced: MAX_HEADER_SIZE (64 KB), MAX_BODY_SIZE (10 MB), client/upstream timeouts (30s / 60s), and a 128-connection concurrency cap.

Configuration

Configuration values can be set via ~/.tokenectomy.toml:

default_provider = "openai"  # openai | anthropic | ollama | mock
openai_api_key = "sk-..."
anthropic_api_key = "sk-ant-..."
ollama_base_url = "http://localhost:11434"
context_lines = 10
max_context_chars = 10000

Supported Ecosystems

Language	Primary Frameworks	Excluded Framework Paths
Rust	Tokio, Actix-web, Axum	`.cargo/registry`, `.rustup`, `target/debug/build`
Python	Django, FastAPI, Flask, PyTorch	`site-packages`, `dist-packages`, `venv`, `__pycache__`
TypeScript / JavaScript	Next.js, Express, NestJS, Vite	`node_modules`, `.next`, `dist`, webpack internals
Golang	Gin, Fiber, Stdlib panics	`go/src` (stdlib), `go/pkg/mod`, `vendor`
Java / Kotlin	Spring Boot, Quarkus, Gradle	`.m2/repository`, `.gradle/caches`, framework internals
C / C++	GDB Backtraces, AddressSanitizer	`/usr/include`, `/usr/lib`, `vcpkg_installed`
PHP	Laravel, Symfony	`vendor/composer`, `vendor/symfony`, `vendor/laravel`

Note: Currently shipped with robust extractors for Rust, Python, TypeScript/JavaScript, and Go. Java/Kotlin, C/C++, and PHP support is coming in v1.2. See #1 for progress tracking.

How Tokenectomy Compares

Feature	Tokenectomy	Splunk Log Obfuscation	Datadog Logs	git-secrets
Instant setup (no agent install)	✅	❌	❌	✅
Works with AI agents (MCP)	✅	❌	❌	❌
Local-only processing	✅	❌	❌	✅
Polyglot stack traces	✅	✅	✅	❌
Redaction caching (cost savings)	✅	❌	✅	❌
Open source (MIT)	✅	❌	❌	✅
Price	Free OSS	$$$ /mo	$$$ /mo	Free

When to Use Tokenectomy:

✅ You use AI coding agents (Claude, Cursor, Cline, etc.)
✅ You care about privacy & local-first processing
✅ You want to reduce LLM token costs
✅ You're worried about secret leakage in logs

When to Use Something Else:

❌ You only need static secret scanning → use truffleHog, detect-secrets
❌ You need real-time monitoring dashboards → use Datadog, New Relic, Splunk
❌ Your error logs are naturally <100 tokens → overhead not worth it
❌ You're fully air-gapped → Actually Tokenectomy is perfect! (100% local processing)

Frequently Asked Questions

Q: Does Tokenectomy send my logs to external servers?

A: No. All redaction, parsing, and filtering happens locally on your machine. The only network call is to your chosen LLM (OpenAI, Anthropic, Ollama) after sanitization is complete. See SECURITY.md for the zero-knowledge guarantee.

Q: What secrets does Tokenectomy redact?

A: GitHub PATs, AWS keys, OpenAI/Anthropic API keys, JWTs, database connection strings (PostgreSQL, MySQL, MongoDB, Redis), private SSH keys, Slack/Discord webhooks, and more. Full list in src/redact.rs.

Q: What if my secret doesn't match the redaction patterns?

A: File an issue with an example (sanitized). We'll add the pattern. For now, you can add custom patterns in ~/.tokenectomy.toml (feature coming in v1.3).

Q: Is Tokenectomy safe for production?

A: Yes. Written in safe Rust (zero unsafe code in security paths), ReDoS-immune, and audited via RustSec. See SECURITY.md for full details.

Q: Can I use Tokenectomy offline?

A: Yes—except Stack Overflow search. Use --local-only flag to disable all network access (except your LLM).

Q: How do I remove Tokenectomy?

A: Simply uninstall:

npm uninstall -g tokenectomy-razor
# OR
cargo uninstall tokenectomy

Zero config cleanup needed—no files left behind.

Q: Can I use Tokenectomy in my CI/CD pipeline?

A: Yes! Use the GitHub Marketplace action (see Quick Start section) or the Docker container. Works with GitHub Actions, GitLab CI, Jenkins, etc.

Q: What's the difference between Razor (OSS) and Sentinel (Commercial)?

A: Razor is the free, community version with all essential features. Sentinel adds advanced capabilities like tree-sitter AST healing, anti-hallucination guards, and time-machine undo. See Edition Comparison below.

Edition Comparison

Capability	Razor (Community OSS)	Sentinel (Commercial Tier)
Framework Log Filtering	Yes	Yes
Polyglot Trace Extraction (4 Languages)	Yes	Yes (7 Languages)
AI Reverse Proxy Gateway (`--proxy`)	Yes	Yes
Stack Overflow Integration	Yes	Yes
SHA-256 Idempotency Cache	Yes	Yes
ReDoS-Safe Secret Redaction	Yes	Yes
MCP Protocol Server (JSON-RPC)	Yes	Yes
Bundled Agent Skills	2 Skills (Spec TDD & Fuzzer)	Full 4 Skills Suite
Tree-sitter AST Syntax Healing	No	Yes
Anti-Hallucination Scope Guard	No	Yes
Automated Test Rollback Loop	No	Yes
Multi-File Atomic Transactions	No	Yes
Time Machine Undo Engine (`--undo`)	No	Yes
True Ectomy Deep Surgery Engine	No	Yes
Live DB Port & Docker Diagnostics	No	Yes

Interested in Sentinel? View pricing & features

Roadmap

Feature	Status	Target Version
Java/Kotlin extractor	🔄 In Progress	v1.2
Go extractor improvements	🔄 In Progress	v1.2
Custom redaction rules (TOML config)	📋 Planned	v1.3
VS Code extension	📋 Planned	v1.4
Tree-sitter AST healing	✅ Sentinel (Paid)	Now
Multi-file atomic transactions	✅ Sentinel (Paid)	Now
Time machine undo engine	✅ Sentinel (Paid)	Now

Security & Reliability Invariants

Zero-Knowledge Processing: All scanning and redaction occurs on local hardware before data leaves the system boundary.
ReDoS Immunity: All pattern matchers utilize finite automaton evaluation with linear time guarantees. Verified in benchmarks.
Path Traversal Isolation: File operations are strictly locked within workspace boundaries via WorkspaceBoundary security module.
Memory Safety: Implemented in safe Rust with bounded stream readers (.take()) preventing resource exhaustion attacks.
Audit Verification: Continuous dependency auditing maintained via RustSec advisory databases.

Vulnerability Disclosure: See SECURITY.md for responsible disclosure procedures.

Contributing

Found a bug? Have a feature request? Want to add support for a new language?

Issues: github.com/Tokenectomy-Labs/Tokenectomy/issues
Pull Requests: Fork, create a feature branch, and submit a PR with tests
Security: See SECURITY.md for private vulnerability disclosure

See CONTRIBUTING.md for detailed contribution guidelines.

Resources

📖 Documentation — Full guides, API reference, and integration tutorials
📋 Changelog — Release history and notable changes
🤝 Contributing — How to contribute, development workflow, and testing
🔒 Security Policy — Vulnerability disclosure and audit details
🏗️ Architecture — Internal design and system architecture

License

MIT License. See LICENSE for full terms.

Made with ❤️ by @daffa2555

Questions? Open an issue or start a discussion on GitHub.