Cloud

Multi-node Inference Architecture Using Ray LLM DLC on Amazon EKS

The multi-node inference architecture using Ray LLM Deep Learning Containers enables deployment of large language models across multiple GPU-powered nodes with efficient communication via EFA and Ray Serve.

AI-generated editorial illustration of the article topic; not a photograph, product screenshot or measured result.

Multi-node Inference Architecture and Components

The multi-node inference architecture using Ray LLM Deep Learning Containers (DLC) enables the deployment of large language models across multiple GPU-powered nodes. This setup leverages the Ray Serve LLM, NVIDIA Inference Transfer Library (NIXL), and the Elastic Fabric Adapter (EFA) stack to facilitate efficient communication and coordination between nodes. The architecture is designed to operate behind a single endpoint, providing an OpenAI-compatible API without requiring application code. The vLLM inference engine is integrated for performance, while Ray Serve’s HTTP layer manages the serving component. The EFA stack, including libfabric and aws-ofi-nccl, ensures high-bandwidth, low-latency communication between nodes, which is essential for maintaining performance in distributed inference workloads. This approach is particularly useful for deploying large models across multiple nodes on Amazon EKS, as described in the implementation walkthrough.

Deployment Setup on Amazon EKS

Deployment Setup on Amazon EKS involves creating an EKS cluster within a virtual private cloud (VPC) to host multi-node inference workloads using Ray LLM Deep Learning Containers. According to AWS Machine Learning, the architecture includes a CPU system node (m7i.xlarge) that runs system workloads, the KubeRay operator, and the Ray head pod, which is CPU-only. The Ray head is responsible for managing the distributed computation, while the prefill and decode phases are executed on two GPU worker nodes. To begin the deployment, users are instructed to set a base directory and load variables from the project root, as outlined in AWS Machine Learning. The first step is to create the EKS cluster using the script ./deploy_cluster.sh, which provisions the EKS cluster, VPC, OIDC provider, and core add-ons, along with the CPU system node group.

Implementation Steps and Cluster Configuration

Implementation Steps and Cluster Configuration involve several key actions to set up a multi-node inference environment using Ray LLM Deep Learning Containers. Step 2 involves adding GPU worker nodes by executing ./deploy_node_group.sh, which provisions two g6.8xlarge instances with EFA enabled, pinned to a single availability zone and labeled with role=gpu-worker to ensure Ray workers target them correctly. Step 3 requires installing the KubeRay operator via ./install_kuberay.sh, which uses Helm to deploy the operator while keeping the GPU nodes available for inference tasks. Finally, Step 4 deploys the RayService by running ./deploy_ray_service.sh, which generates and applies the manifest/rayservice.yaml file with the specified image, model, and naming variables using envsubst. These steps are essential for configuring the cluster to support distributed inference workloads efficiently.

Verification and Validation of Deployment

Verification and validation of a multi-node Ray LLM Deep Learning Container deployment involves checking logs for evidence of successful backend instantiation. By grepping worker logs for entries like 'Backend (LIBFABRIC|UCX) was instantiated,' administrators can confirm that the necessary communication layers are correctly configured across nodes. The script deploy_node_group.sh also provides visibility into the number of EFA interfaces per node, which is crucial for performance validation. According to AWS Machine Learning, the tested inference stack was delivered via the multi-node Ray LLM DLC, with KubeRay managing the Ray cluster and NIXL over EFA facilitating KV cache transfers between nodes. AWS Machine Learning highlights that common multi-node inference strategies include tensor and pipeline parallelism, data parallelism, and prefill/decode disaggregation, each designed to optimize model execution across distributed resources.