Nexus

One API endpoint. Any backend. Zero configuration.

I am the creator and principal maintainer of Nexus, a distributed LLM orchestrator written in Rust.

Nexus solves a critical pain point in the rapidly fragmenting AI landscape: it unifies heterogeneous inference backends (such as Ollama, LM Studio, vLLM, llama.cpp, Exo, and cloud providers) behind a single, intelligent API gateway. It prioritizes local-first execution while seamlessly falling back to the cloud when necessary.

Architectural Highlights

• Zero-Config Auto-Discovery: Automatically finds and connects to local LLM backends on the network via mDNS without requiring any static configuration files.
• Intelligent Routing & Failover: Dynamically routes requests based on real-time model capabilities, current load, and latency, offering transparent automatic failover if a node drops.
• Fleet Intelligence & Lifecycle Management: Features advanced orchestrator logic to dynamically load, unload, and migrate models across the backend fleet based on usage pattern analysis.
• Privacy & Governance: Enforces structural “Privacy Zones” preventing sensitive local data from ever reaching cloud backends, paired with token-aware budget management and cost tracking.
• High Performance: Implemented in Rust, the orchestrator provides blazing-fast, concurrent request queuing, load balancing, and an OpenAI-compatible API with minimal overhead.

This project showcases my ability to architect complex, high-performance distributed systems, specifically focusing on the lifecycle orchestration and resource management of Large Foundational Models.