Community

Enki LLM: Pure Rust Implementation with Live Brain Migration

The Enki LLM project demonstrates a pure Rust implementation of a Transformer-based language model, achieving high-performance inference with minimal code and supporting live brain migration between GPU VRAM and CPU RAM.

AI-generated editorial illustration of the article topic; not a photograph, product screenshot or measured result.

Pure Rust Implementation and Performance

The Enki LLM project demonstrates a pure Rust implementation of a Transformer-based language model, achieving high-performance inference with minimal code. According to the project's documentation, it runs the Llama-3.2 1B model at approximately 40 tokens per second on an Nvidia Tesla T4 GPU, without relying on CUDA dependencies. The implementation includes essential components such as a Safetensors parser, tokenizer, Split-K GEMV, RoPE, RMSNorm, and SwiGLU, all built from scratch in Rust. The project highlights its minimalist approach, with the entire system spanning around 2,000 lines of code. Notably, Enki supports live brain migration, allowing the model to transition between GPU VRAM and CPU RAM during generation. This capability is attributed to the project's design and is not presented as a universal guarantee. The performance metrics are attributed to the project's documentation and are not independently verified.

Live Brain Migration and Cross-Silicon Portability

The Enki LLM demonstrates a unique capability in live brain migration, allowing seamless hot-swapping of active execution between GPU VRAM and CPU RAM mid-sentence, as described in Hacker News Show. This process maintains context continuity by mirroring the key-value (KV) cache during migration. Furthermore, the system supports cross-silicon portability, enabling the exact same binary to run on discrete GPUs (Nvidia), integrated graphics (Intel UHD), and multi-core CPUs without requiring code modifications, as noted in Hacker News Show. When migrated to CPU, the same kernels execute natively in parallel across all CPU cores using Rayon, as stated in Hacker News Show. These features highlight Enki's flexibility and adaptability across different hardware environments.

Compilation and Execution via Enki Runtime

The Enki LLM runtime enables seamless compilation and execution of Rust code across both CPU and GPU environments. Unlike traditional Rust GPU tools that rely on wrapping CUDA/C++ or transpiling syntax into WGSL, Enki uses its own compiler, Parsu, to directly convert LLVM bitcode generated by rustc into SPIR-V compute shaders at runtime. This approach eliminates the need for intermediate representations and allows for more efficient execution on GPU hardware. Execution is further optimized through Vulkan 1.3, which supports 64-bit Buffer Device Addresses (BDA), improving memory ergonomics and performance on the GPU. This architecture enables live migration of workloads between CPU and GPU, offering flexibility and efficiency in deployment. The system is built with a focus on leveraging modern hardware capabilities without requiring significant changes to the source code.

Interactive Controls and Usage

Interactive controls allow users to manage the Enki LLM engine in real time during generation. The /gpu command migrates active execution and KV-cache to GPU silicon, enabling hardware-specific optimizations. Additionally, the /dual command activates ping-pong mode, which alternates token generation between GPU and CPU live, offering flexibility in resource utilization. For users seeking a hands-on experience, a one-click Google Colab environment is available via the badge above or through this direct link: Open in Google Colab (Free T4 GPU). Once opened, users can simply click Run all to begin interacting with the model. These features are demonstrated in the console and are accessible through the provided Colab notebook.

Local Setup and Requirements

To set up Enki LLM locally, users must first install Vulkan 1.3 drivers and a standard Rust toolchain, as outlined in the project's documentation (Hacker News Show). The process begins with cloning the repository using git clone https://github.com/enkiruntime/enki_llm.git, followed by navigating into the project directory. To run the model, users can execute cargo run --release HuggingFaceTB/SmolLM2-135M-Instruct for the SmolLM2-135-Instruct model or cargo run --release unsloth/Llama-3.2-1B-Instruct for the Llama-3.2-1B-Instruct model. The project's core runtime is described as the Enki Heterogeneous Compute Platform, which enables live migration of the model's brain between GPU VRAM and CPU RAM during inference, as noted in (Hacker News Show).