RAGStack-Lambda

Serverless document and media processing with AI chat.

GitHub

Serverless document and media processing with AI chat. Scale-to-zero architecture — no vector database fees, no idle costs. Upload documents, images, video, and audio — extract text with OCR or transcription — query using Amazon Bedrock or your AI assistant via MCP.

Features

  • ☁️ Fully serverless architecture (Lambda, Step Functions, S3, DynamoDB)
  • 🧠 NEW Amazon Nova multimodal embeddings for text and image vectorization
  • 📄 Document processing & vectorization (PDF, images, Office docs, HTML, CSV, JSON, XML, EML, EPUB) → stored in managed knowledge base
  • 🎬 NEW Video/audio processing - transcribe speech with AWS Transcribe, searchable by timestamp
  • 💬 AI chat with retrieval-augmented context and source attribution
  • 📎 Collapsible source citations with optional document downloads
  • ⏱️ NEW Media sources with timestamp links - click to play at exact position
  • 🔍 Metadata filtering - auto-discover document metadata and filter search results
  • 🎯 Relevancy boost for filtered results - prioritize matches from metadata filters
  • 🔄 Knowledge Base reindex - regenerate metadata for existing documents with updated settings
  • 🗑️ Document management - reprocess, reindex, or delete documents from the dashboard
  • 🌐 Web component for any framework (React, Vue, Angular, Svelte)
  • 🚀 One-click deploy
  • 💰 $7-10/month (1000 docs, Textract + Haiku)

Live Demo

EnvironmentURLCredentials
Base Pipelinedhrmkxyt1t9pb.cloudfront.net[email protected] / Guest@123
Project Showcaseshowcase-htt.hatstack.funLogin as guest

Base Pipeline: The core document processing tool - upload, OCR, and query documents.

Project Showcase: See RAGStack powering a real application.

Quick Start

Option 1: One-Click Deploy (AWS Marketplace)

REPO IS IN ACTIVE DEVELOPMENT AND WILL CHANGE OFTEN

Deploy directly from the AWS Console - no local setup required:

  1. Subscribe to RAGStack on AWS Marketplace (free, not required - If subscribed Lambda roles auto-accept Bedrock model agreements on first invocation)
  2. Click here to deploy
  3. Enter a stack name (lowercase only, e.g., "my-docs") and your admin email
  4. Click Create Stack (deployment takes ~10 minutes)

After deployment:

  • Check your email for the temporary password (from Cognito)
  • Go to CloudFormation → your stack → Outputs tab to find the Dashboard URL (UIUrl)

Option 2: Deploy from Source

For customization or development:

Prerequisites:

  • AWS Account with admin access
  • Python 3.13+, Node.js 24+
  • AWS CLI, SAM CLI (configured)
  • Docker (for Lambda layer builds)
git clone https://github.com/HatmanStack/RAGStack-Lambda.git
cd RAGStack-Lambda

# Create virtual environment and install dependencies
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install -r requirements.txt

# Deploy (defaults to us-east-1 for Nova Multimodal Embeddings)
python publish.py \
  --project-name my-docs \
  --admin-email [email protected]

Option 3: Nested Stack Deployment

Deploy RAGStack as part of a larger CloudFormation stack. See Nested Stack Deployment Guide for details.

Quick example:

Resources:
  RAGStack:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: https://ragstack-quicklaunch-public.s3.us-east-1.amazonaws.com/ragstack-template.yaml
      Parameters:
        StackPrefix: 'my-app-ragstack'  # Required: lowercase prefix
        AdminEmail: [email protected]

Web Component Integration

See RAGSTACK_CHAT.md for web component integration guide.

API Access

Server-side integrations use API key authentication. Get your key from Dashboard → Settings.

curl -X POST 'YOUR_GRAPHQL_ENDPOINT' \
  -H 'x-api-key: YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{"query": "query { searchKnowledgeBase(query: \"...\") { results { content } } }"}'

Web component uses IAM auth (no API key needed - handled automatically).

Each UI tab shows server-side API examples in an expandable section.

MCP Server (AI Assistant Integration)

Use your knowledge base directly in Claude Desktop, Cursor, VS Code, Amazon Q CLI, and other MCP-compatible tools.

# Install (or use uvx for zero-install)
pip install ragstack-mcp

Add to your AI assistant's MCP config:

{
  "ragstack-kb": {
    "command": "uvx",
    "args": ["ragstack-mcp"],
    "env": {
      "RAGSTACK_GRAPHQL_ENDPOINT": "YOUR_ENDPOINT",
      "RAGSTACK_API_KEY": "YOUR_API_KEY"
    }
  }
}

Then ask naturally: "Search my knowledge base for authentication docs"

See MCP Server docs for full setup instructions.

Architecture

Upload → OCR → Embeddings → Bedrock KB
                                ↓
 Web UI (Dashboard + Chat) ←→ GraphQL API
                                ↓
 Web Component ←→ AI Chat with Sources

Usage

Documents

Upload documents in various formats. Auto-detection routes to optimal processor:

TypeFormatsProcessing
TextHTML, TXT, CSV, JSON, XML, EML, EPUB, DOCX, XLSXDirect extraction with smart analysis
OCRPDF, JPG, PNG, TIFF, GIF, BMP, WebP, AVIFTextract or Bedrock vision OCR (WebP/AVIF require Bedrock)
MediaMP4, WebM, MP3, WAV, M4A, OGG, FLACAWS Transcribe → 30s segments → searchable with timestamps
PassthroughMarkdown (.md)Direct copy

Processing time: UPLOADED → PROCESSING → INDEXED (typically 1-5 min for text, 2-15 min for OCR, 5-20 min for media)

Images

Upload JPG, PNG, GIF, WebP with captions. Both visual content and caption text are searchable.

Web Scraping

Scrape websites into the knowledge base. See Web Scraping.

Video & Audio

Upload MP4, WebM, MP3, WAV, M4A, OGG, or FLAC files. Speech is transcribed using AWS Transcribe and segmented into 30-second chunks for search. Sources include timestamps (e.g., "1:30-2:00") with clickable links that play at the exact position.

Features:

  • Speaker diarization (identify who said what)
  • Configurable language (30+ languages supported)
  • Timestamp-linked sources in chat responses

See Configuration for language and speaker settings.

Chat

Ask questions about your content. Sources show where answers came from.

Documentation

Development

npm run check  # Lint + test all (backend + frontend)

Deployment Options

Direct Deployment

# Full deployment (defaults to us-east-1)
python publish.py --project-name myapp --admin-email [email protected]

# Skip dashboard build (still builds web component)
python publish.py --project-name myapp --admin-email [email protected] --skip-ui

# Skip ALL UI builds (dashboard and web component)
python publish.py --project-name myapp --admin-email [email protected] --skip-ui-all

# Enable demo mode (rate limits: 5 uploads/day, 30 chats/day; disables reindex/reprocess/delete)
python publish.py --project-name myapp --admin-email [email protected] --demo-mode

Publish to AWS Marketplace (Maintainers)

To update the one-click deploy template:

python publish.py --publish-marketplace

This packages the application and uploads to S3 for one-click deployment.

Note: Currently requires us-east-1 (Nova Multimodal Embeddings). When available in other regions, use --region <region>.

Acknowledgments

This project was inspired by:

Related Servers