GitHub - google-ai-edge/ml-drift: GPU-Accelerated AI/ML Inference
ML Drift is a high-performance, cross-platform GPU-accelerated inference engine designed for on-device machine learning. It offers significant performance improvements through optimized kernels and runtime, and supports multiple GPU APIs including OpenCL, Metal, and WebGPU.
Overview and Key Features
ML Drift is a GPU-accelerated inference engine developed by Google AI Edge, aimed at enabling efficient execution of AI/ML workloads on a variety of devices, including mobile phones, web browsers, and servers. The engine is designed to minimize latency and power consumption, making it suitable for real-time applications on edge devices. It supports multiple GPU APIs, including OpenCL, Metal, WebGPU (via Dawn), and OpenGL ES 3.1+, ensuring broad compatibility across platforms.
One of the key features of ML Drift is its Unified Compute Language (UCL), which allows developers to write GPU kernels once and compile them dynamically for different backends. This abstraction layer over device-specific shading languages (GLSL, MSL, WGSL) simplifies development and deployment across various hardware configurations.
ML Drift also includes advanced memory management techniques, such as efficient reuse of GPU memory, which is particularly useful for running large models on devices with limited resources. Additionally, it supports quantization techniques like FP16, INT8, and INT4, which help reduce memory usage and improve inference speed.
Architecture and Optimization
The architecture of ML Drift is centered around a backend-agnostic in-memory representation of a machine learning model's compute graph, known as GpuModel. This abstraction allows the engine to be platform-agnostic and facilitates the integration of various GPU APIs.
ML Drift employs several runtime optimizations, including graph transformations, operator fusion, and layout adjustments, which help reduce kernel launch overhead and improve performance. The engine also supports dynamic code generation, where shaders are generated at runtime based on the specific model, inputs, and GPU, ensuring maximum efficiency.
For large language models (LLMs), ML Drift includes specialized optimizations such as efficient key-value (KV) cache management and stage-aware execution (prefill vs. decode). These features are crucial for handling the computational demands of generative AI models on resource-constrained devices.
Supported Backends and Use Cases
ML Drift supports a range of GPU backends, including OpenCL (primary for Android), Metal (for Apple devices), WebGPU (via Dawn for web browsers), and OpenGL ES 3.1+. This broad support ensures that developers can deploy ML models across a wide range of devices and platforms.
Use cases for ML Drift include on-device inference for mobile applications, in-browser ML acceleration using WebGPU, and running large language models efficiently on edge devices. The engine is particularly well-suited for applications that require real-time intelligence, user privacy, and offline functionality.
ML Drift is actively developed and serves as the successor to the GPU delegate in TensorFlow Lite. Developers interested in contributing to the project can find more information on the contributing page.
Source: github.com. Analysis based on publicly available information; no hands-on testing is claimed.
