EXO

Managing Distributed AI Clusters

I contributed massive architectural updates to EXO, a popular open-source AI workflow tool designed for managing and executing distributed local AI clusters via Event Sourcing.

Expanding Hardware Compatibility

Originally, EXO’s inference runner was tightly coupled to Apple’s MLX framework, effectively limiting its hardware footprint to Apple Silicon. My goal was to run a distributed EXO cluster on a Raspberry Pi equipped with an attached NVIDIA GPU.

To achieve this, I completely refactored the core execution pipeline and submitted the changes upstream:

• Engine Abstraction Layer: I introduced a generic BaseEngine interface, cleanly decoupling the Event Sourcing architecture from the underlying inference mechanics.
• PyTorch Backend: I designed and implemented the PytorchEngine utilizing HuggingFace Transformers. This brought full, native NVIDIA GPU support to the ecosystem.
• Hardware Metrics & Parity: I ensured the PyTorch backend maintained strict feature parity with MLX, implementing support for streaming text generation, tool calling, and multi-GPU nvidia-smi hardware metrics extraction for Linux environments.
• Dashboard Integration: I updated the UI dashboard to support model-engine compatibility filtering, ensuring a seamless user experience regardless of the underlying hardware.

By participating in this open-source community and introducing the multi-backend paradigm, I significantly lowered the barrier of entry for engineers looking to deploy powerful AI models locally on diverse, heterogeneous hardware architectures.