Version: 0.7.28 Status: Production Deployed Deployment: luminarimud.com:8003 Repository: https://github.com/LuminariMUD/sage
ORIGINAL AUTHOR: Jamie McLaughlin - This is a fork of his private project published with his permission.
Luminari Sage transforms the vast lore of LuminariMUD into an intelligent, queryable knowledge system that understands context, relationships, and narrative possibilities. It serves as both a reference tool and a creative assistant for players, dungeon masters, and content creators.
- Instant Lore Access: Ask questions in natural language and receive comprehensive, contextual answers
- Story Exploration: Discover connections between characters, events, and locations
- Quest Discovery: Get personalized quest recommendations based on interests
- Campaign Planning: Generate quests and story arcs grounded in canonical lore
- Creative Assistance: Develop new stories while maintaining consistency
- Narrative Generation: Create atmospheric prose for game sessions
- Lore Validation: Ensure new content aligns with established canon
- Relationship Mapping: Understand complex interconnections in the world
- Collaborative Development: Build on existing lore with AI assistance
Luminari Sage is a graph-based lore management system with AI-powered search and validation capabilities for the LuminariMUD world. It combines Neo4j's native graph database with PostgreSQL's pgvector extension for embeddings, orchestrated by PydanticAI agents and Graphiti knowledge graph management, to provide hybrid RAG (Retrieval-Augmented Generation) search through an MCP (Model Context Protocol) interface.
Hybrid Knowledge System - Uniquely combines:
- Vector Embeddings: For semantic similarity search
- Graph Database: For relationship traversal and entity connections
- Knowledge Graphs: For structured fact extraction and validation
Intelligent Agent Orchestration - Multiple specialized AI agents that can:
- Work independently for simple tasks
- Collaborate on complex, multi-step operations
- Pass context between operations for coherent workflows
Real-time Streaming - Advanced streaming architecture provides:
- Token-by-token response streaming
- Progress updates for long operations
- Immediate user feedback
- Graph-First Architecture: Entities and relationships modeled in Neo4j knowledge graph
- Hybrid RAG Search: Combines PostgreSQL pgvector, full-text search, and graph traversal
- Semantic Chunking: Intelligent document segmentation with Graphiti integration
- Entity & Relationship Extraction: Automatic knowledge graph construction from markdown
- Bounded Graphiti Routing: Durable extraction uses explicit candidate retries/fallback, post-validation request accounting, and degraded fallback outcomes
- Canonical Relationship Policy: Versioned edge vocabulary, safe alias normalization, endpoint validation, append-only quality evidence, and separate read-only reporting
- Policy-Aligned Extraction Benchmark: Staged entity/relationship extraction, JSON-schema validation, explicit outcome expectations, and content-free relationship-quality scoring
- Lore Validation System: Comprehensive validation with finding storage and review workflow
- Correction System: Batch corrections with rollback capabilities
- RESTful API: 35+ endpoints for search, retrieval, validation, and corrections
- LangChain Agents: Multi-agent system with ReAct workflow (quest planning, story development)
- MCP Server: Model Context Protocol interface for Claude Desktop integration
- Authentication: Multi-tier API key system with middleware protection
- Production Deployment: Running on luminarimud.com with Docker orchestration
- Advanced Entity Resolution: Handle aliases, variants, and disambiguation
- Temporal Modeling: Track both in-world and real-world timelines
- Performance Optimization: Query caching and index tuning
- Discord Bot: Interactive queries and notifications
- Web Interface: Browser-based lore exploration and management
- Public API: Rate-limited public access for community tools
- Neo4j Graph Database: Native graph storage for entities and relationships
- PostgreSQL + pgvector: Document storage with integrated vector embeddings
- PydanticAI: Type-safe AI agent framework for entity extraction and validation
- Graphiti: Knowledge graph construction and management
- Data Pipeline: ETL for markdown → knowledge graph + vectors
- API Server: REST/GraphQL endpoints for queries
- MCP Server: AI agent interface with specialized tools
- Validation Engine: Lore consistency checker for builders
📚 Complete Documentation: See docs/DOCUMENTATION.md for the complete documentation guide.
Getting Started:
- Quickstart Guide - Get up and running in 10 minutes
- User Guide - Complete user documentation
- FAQ - Frequently asked questions
Deployment:
- Deployment Guide - Production deployment
- WSL2 Ubuntu Deployment - WSL2-specific guide
Technical Reference:
- API Reference - Complete API documentation (35+ endpoints)
- Architecture - System design and data flow
- Database Schemas - PostgreSQL and Neo4j schemas
Development:
- Developer Guide - Development environment and patterns
- Contributing Guide - How to contribute
- Testing Guide - Testing patterns and best practices
- CLAUDE.md - AI assistant guidance for development
Systems Deep Dives:
- Agent System - LangChain and PydanticAI agents
- Validation System - Lore consistency validation
- Correction System - Automated corrections
- Pipeline System - Data ingestion and processing
Project Meta:
# Clone the repository
git clone https://github.com/LuminariMUD/sage.git
cd sage
# Copy and configure environment
cp .env.example .env
chmod 600 .env
# Edit .env with database/auth secrets and only the credentials required by
# your selected text and embedding providers
# Start only the services required by the selected provider capabilities
make dev
# Check service health
docker compose logs -f api
# Inspect migrations and embedding storage before any vector operation
make db-migrate-status
make embedding-preflight
# After the backup-gated migrations, initialize and activate the vector-empty
# episode space at the configured provider's output width
make embedding-space-initialize-empty \
CONFIRM_EMBEDDING_SPACE=INITIALIZE_EMPTY_EMBEDDING_SPACE
# Run individual pipeline steps only after preflight reports READY
# make load-canon
# make create-episodes
# make generate-embeddings
# make sync-to-graphiti CONFIRM_GRAPH_SYNC=RUN_DURABLE_GRAPH_SYNC
# Test the API
curl http://localhost:8003/ping
curl http://localhost:8003/api/v1/healthSee docs/guides/QUICKSTART.md for detailed instructions.
- 16 Canon Source Files: Markdown lore in
lore_docs/canon/ - 99 Draft Source Files: Markdown source material in
lore_docs/drafts/ - 14 Preserved Local Documents: The current database retains source documents but has zero episodes after the rejected corpus was deleted
- Empty Current Graph: The current local Neo4j store has zero nodes and relationships
- No Active Episode Vectors: The pending target is 1024 dimensions; legacy 384-dimensional chunk search is retired
- 35+ API Endpoints: Health, search, RAG, validation, corrections, chat
- 6 Validation Types: Structural, semantic, type-specific, cross-ref, temporal, canonical
- 2 Correction Types: Deduplication, semantic standardization
- 4 Major Agent Types: Direct answer, quest planner, story developer, validator
- 3 Authentication Tiers: Backend API, MCP operations, MCP backend access
- Entity Search: 100-300ms average
- Lore Search: 200-500ms average
- RAG Query: 1-3 seconds average
- Agent Response: 3-10 seconds (streaming)
- Pipeline Processing: 30-60 minutes (initial load)
- ✅ Development environment setup
- ✅ Database schemas (PostgreSQL + Neo4j)
- ✅ Entity extraction pipeline with Graphiti
- ✅ Docker configuration
- ✅ Semantic chunking with episode system
- ✅ FastAPI API with 35+ endpoints
- ✅ Entity and relationship extraction
- ✅ Hybrid RAG (vector + graph + FTS)
- ✅ LangChain agent system with ReAct workflow
- ✅ Validation system with finding management
- ✅ Correction system with batch operations
- ✅ MCP server for Claude Desktop
- ✅ Authentication middleware
- ✅ Conversation storage
- ✅ Unit tests for core components
- ✅ Integration testing suite
- ✅ API integration tests
- ✅ Data-dependent test markers
- ✅ Server deployment (luminarimud.com:8003)
- ✅ Docker orchestration with compose
- ✅ Health checks and monitoring
- ✅ API authentication system
- ✅ Database persistence with volumes
- ✅ Data pipeline automation with Makefile
- 📋 Performance optimization (caching, indexes)
- 📋 Discord bot integration
- 📋 Web-based administration interface
- 📋 Public API with rate limiting
- 📋 Advanced analytics and metrics
See docs/meta/TODO.md for detailed task tracking.
The API runs on http://localhost:8003 by default (production: luminarimud.com:8003).
- Health checks and system status
- Entity search and retrieval
- Lore document search
- Hybrid RAG queries (vector + graph + FTS)
- Content validation
- Relationship validation
- Finding management and review
- Validation history and statistics
- Batch corrections with rollback
- Correction history tracking
- Preview before applying
- Streaming chat with LangChain ReAct
- Legacy PydanticAI agent support
- Conversation history storage
Interactive API docs: http://localhost:8003/docs Complete reference: docs/reference/API_REFERENCE.md
The data pipeline processes markdown lore documents into a queryable knowledge graph:
# Complete pipeline (recommended)
make semantic-pipeline
# Individual steps (run in order)
make load-canon # Load canonical lore into PostgreSQL
make create-episodes # Create semantic chunks (200-500 tokens)
make embedding-preflight # Must report READY before vector reads/writes
make generate-embeddings # Generate vector embeddings
make sync-to-graphiti CONFIRM_GRAPH_SYNC=RUN_DURABLE_GRAPH_SYNC
# Monitoring and status
make status # System health and statisticsPipeline Features:
- Idempotent operations (safe to re-run)
- Processing flags track completion
- Resume capability after interruption
- Resource-intensive (run separately from deployment)
See docs/systems/PIPELINE_SYSTEM.md for detailed documentation.
The application embedding profile is persisted separately from the physical pgvector column and index. API startup, RAG, validation, and episode embedding generation compare the configured fingerprint with that metadata, the physical dimension, the index method/operator class, and aggregate row counts before any provider request. A mismatch leaves embedding-dependent endpoints unavailable.
# Strictly read-only; emits no source text or vector values
make embedding-preflight
make embedding-preflight-jsonMigration 0004_embedding_index_profiles records the physical spaces but leaves
existing episode vectors unverified. After applying it through the verified
backup gate, a vector-empty space can be initialized at the configured profile
width and activated without a provider call:
make embedding-space-initialize-empty \
CONFIRM_EMBEDDING_SPACE=INITIALIZE_EMPTY_EMBEDDING_SPACEThis operation refuses any space containing stored vectors or carrying an existing profile identity. Adopt a populated Nomic space only when its provenance has been independently confirmed:
make embedding-profile-activate \
ADOPT_EXISTING=1 \
CONFIRM_EMBEDDING_PROFILE=ADOPT_EXISTING_EMBEDDING_PROFILEMetadata activation and empty-space initialization never call a provider. Matching dimensions alone are not proof of provenance. Changes to a populated provider or dimension must use a shadow vector space and pass retrieval gates before cutover. See the migration rules for both empty-space and existing-space procedures.
The checked-in retrieval corpus contains 12 lore questions, 33 graded episode judgments, and 39 expected entity aliases. Every judgment is tied to a portable document stable ID, episode index, and exact source fingerprint; the complete 611-episode source snapshot is fingerprinted separately.
# Strictly read-only; validates source drift and entity grounding without a model
make retrieval-corpus-check
make retrieval-corpus-check-jsonThe active-index benchmark computes macro Recall@5, Recall@10, MRR@10, and nDCG@10. It is a real embedding-provider operation and therefore refuses unless the active embedding preflight is ready and the exact confirmation token is provided:
make benchmark-retrieval \
CONFIRM_RETRIEVAL_BENCHMARK=RUN_RETRIEVAL_BENCHMARKTransport retries are disabled for this benchmark and the default provider-call ceiling is one batched request. Output excludes query text, ranked episode identities, vectors, source text, and credentials. A completed run is evidence, not automatic approval: acceptance thresholds and manual comparison remain rollout decisions.
Candidate embedding profiles use isolated storage and never overwrite the active Nomic column. Read-only status requires no provider configuration:
make embedding-shadow-statusRegistration, bounded backfill, index construction, and side-by-side retrieval benchmarking each require a distinct exact confirmation token. See the migration runbook. Backfill defaults to one provider request with transport retries disabled; every request is reserved durably before inference and every stored vector is fenced to the source revision it embedded.
Whole-graph rebuilds preserve the durable attempt ledger and require a fresh, verified PostgreSQL/Neo4j backup. Inspect readiness without changing either store:
make graph-rebuild-plan
make graph-rebuild-statusThe workflow uses separate exact confirmations for the destructive preparation, provider-backed durable sync, and final profile activation:
make graph-rebuild-prepare \
BACKUP_REFERENCE=backups/provider-upgrade-<timestamp> \
CONFIRM_GRAPH_REBUILD=PREPARE_DURABLE_GRAPH_REBUILD
make sync-to-graphiti CONFIRM_GRAPH_SYNC=RUN_DURABLE_GRAPH_SYNC
make graph-rebuild-finalize \
CONFIRM_GRAPH_REBUILD_FINALIZE=FINALIZE_DURABLE_GRAPH_REBUILDPreparation records both target profiles, requeues jobs through the authoritative lifecycle, and commits that state before clearing Neo4j. Repeating the same command resumes an interrupted clear. Finalization records the active profiles only after all jobs are verified and the exact-profile cross-store audit is clean. The legacy direct graph clear and Boolean sync-reset paths intentionally refuse execution.
# Run all tests
pytest
# Run unit tests only (fast)
pytest -m unit
# Run integration tests (requires running services)
pytest -m integration
# Run data-dependent tests (requires loaded data)
pytest -m data_dependent
# Run with coverage
pytest --cov=src tests/ --cov-report=htmlSee docs/development/TESTING.md for complete testing guide.
PostgreSQL connection failed
- Check if PostgreSQL is running:
docker ps - Verify credentials in
.env - Ensure pgvector extension is installed
Neo4j connection failed
- Check Neo4j browser at http://localhost:7474
- Verify Neo4j credentials
- Check if indexes were created
Entity extraction not working
- Ensure documents are loaded first:
make load-canon - Check logs:
docker compose logs api - Verify Graphiti is properly initialized
See docs/guides/TROUBLESHOOTING.md for comprehensive troubleshooting.
Luminari Sage follows industry best practices for security and credential management.
The repository received a full credential and runtime hardening review on 2026-07-30. See SECURITY.md for findings and response guidance.
✅ Runtime-managed credentials - Secret values are not tracked in Git ✅ Automated secret scanning - Pre-commit and full-history CI scans ✅ Credential validation - Missing database credentials are rejected before connecting ✅ Credential redaction - Public errors and logs redact credential-shaped values ✅ Validated dynamic identifiers - SQL/Cypher identifiers are allowlisted ✅ Non-root containers - Docker runs as unprivileged user
All secrets must be configured in .env (never commit this file):
Always-required production secrets:
SAGE_API_KEY- Backend API authentication (64-character hex)POSTGRES_PASSWORD- PostgreSQL password (16+ characters)NEO4J_PASSWORD- Neo4j password (16+ characters)SAGE_MCP_KEY- MCP operations authenticationSAGE_MCP_BACKEND_KEY- MCP backend access authentication
Provider-dependent and optional secrets:
OPENAI_API_KEYorOPENAI_API_KEY_FILE- required only when direct OpenAI is selectedOPENROUTER_API_KEYorOPENROUTER_API_KEY_FILE- required only when OpenRouter is selected;OPENROUTER_KEYis a deprecated compatibility aliasLANGSMITH_API_KEY- LangSmith tracing (optional)
# Generate 64-character hex API key
python3 -c "import secrets; print(secrets.token_hex(32))"
# Generate 32-character URL-safe token
python3 -c "import secrets; print(secrets.token_urlsafe(32))".env files, API keys, or passwords to version control.
Automated Scanning:
- Pre-commit hooks with Gitleaks (blocks commits with secrets)
- GitHub Actions security workflow (secrets, dependencies, Docker)
- pip-audit for Python dependency vulnerabilities
- Bandit for Python security linting
- Trivy for Docker image scanning
Access Control:
- Multi-tier API key authentication (backend, MCP operations, MCP backend)
- Path-based authorization (different endpoints require different keys)
- Configurable auth bypass for local development (
DISABLE_AUTH=true)
Data Protection:
- Query values are parameterized and dynamic identifiers are validated
- Pydantic models validate all API inputs
- Credential-shaped values are redacted from public errors and logs
- Restrictive CORS, host validation, and browser security headers
The latest audit was completed on 2026-07-30. The point-in-time findings and credential-response requirements are documented in SECURITY.md.
If you discover a security vulnerability:
- Email: security@luminarimud.com
- Response time: Within 48 hours
- Disclosure: Coordinated disclosure (90-day policy)
Please do not publicly disclose vulnerabilities until they have been addressed.
For complete security documentation, see SECURITY.md.
We welcome contributions! Please see CONTRIBUTING.md for:
- Development environment setup
- Security requirements (REQUIRED)
- Code style guidelines
- Testing requirements
- Pull request process
- Commit message conventions
Security Note: All contributors must install pre-commit hooks (pre-commit install) before making commits. This ensures automated security scanning and prevents accidental credential commits.
- Documentation Hub: docs/DOCUMENTATION.md
- LuminariMUD: https://www.luminarimud.com
- Issues: https://github.com/LuminariMUD/sage/issues
[License details to be determined]
Last Updated: 2026-08-07 Version: 0.7.28 Status: Production Ready