Deploying Kimi K3 on Amazon SageMaker HyperPod and Amazon EKS

Open weight fashions have change into highly effective sufficient to deal with complicated duties equivalent to multi-step agentic workflows, superior reasoning, and long-horizon coding. Nonetheless, as these fashions develop in functionality, additionally they develop in dimension and internet hosting multi-trillion parameter architectures requires purpose-built infrastructure, high-end GPU compute, and optimized serving frameworks. On July 27, 2026, Moonshot AI launched Kimi K3, a 2.8 trillion parameter Combination of Specialists (MoE) mannequin that represents the first open-weight system to succeed in the three trillion parameter class. Kimi K3 delivers frontier-level intelligence whereas making its weights publicly obtainable, in order that organizations can self-host one of the crucial succesful fashions in existence on their very own infrastructure.

This publish walks via deploying Kimi K3 on AWS utilizing two approaches: Amazon SageMaker HyperPod, and Amazon Elastic Kubernetes Service (Amazon EKS) cluster.

Kimi K3 is constructed on a differentiated structure that includes Kimi Delta Consideration (KDA), Gated Multi Head Latent Consideration (MLA), and a Steady LatentMoE framework. The mannequin distributes its 2.8 trillion parameters throughout 896 specialist specialists, activating solely 16 per token. This implies roughly 104 billion parameters are energetic throughout any single ahead go, yielding a 2.5x enchancment in scaling effectivity over its predecessor, Kimi K2.

Kimi K3 excels at long-horizon coding duties, agentic workflows, and sophisticated reasoning. It helps native software calling, structured output, and an always-on considering mode for multi-step drawback fixing.

The open weights for Kimi K3 can be found on Hugging Face beneath the mannequin identifier moonshotai/Kimi-K3. The weights are distributed in MXFP4 (Microscaling Floating Level 4-bit) format, which supplies an efficient stability between mannequin high quality and reminiscence effectivity for large-scale inference deployments.

Given the mannequin’s structure and dimension, serving Kimi K3 requires a vLLM day-0 inference container for Kimi K3. On the time of writing, vllm commits for Kimi K3 are in vllm/vllm-openai:kimi-k3. We count on these to be merged to the principle vllm container within the upcoming releases. vLLM supplies native help for MoE architectures, tensor parallelism, and the MXFP4 quantization format, making it the advisable serving engine for this mannequin.

Deploying a mannequin of this scale requires substantial GPU compute. Kimi K3 requires a p6-b300 occasion (ml.p6-b300.48xlarge), which supplies 8 NVIDIA B300 Blackwell Extremely GPUs with high-bandwidth interconnects essential for environment friendly tensor-parallel inference throughout the complete knowledgeable pool.

AWS presents two major mechanisms to obtain this capability:

1. Versatile Coaching Plans (for SageMaker HyperPod): Present dedicated capability reservations that may be allotted to your HyperPod cluster, in order that GPU sources can be found for sustained inference workloads.
2. Capability Blocks: Permit you to reserve EC2 GPU situations for an outlined interval, offering assured entry to p6-b300 capability with out long-term commitments. Amazon EKS workloads eat these reservations by concentrating on the reserved capability.

Amazon SageMaker HyperPod with the Inference Operator supplies the best path to deploying Kimi K3. The Inference Operator is put in robotically as a part of cluster creation, and it abstracts away the complexity of container orchestration, mannequin loading, and endpoint administration.

Stipulations

Earlier than deploying the mannequin, full the next two prerequisite steps to arrange your infrastructure.

Step 1: Create a SageMaker HyperPod cluster with EKS orchestration

Earlier than deploying the mannequin, you should provision a HyperPod cluster. Navigate to the Amazon SageMaker AI console and comply with the cluster creation workflow:

1. Open the SageMaker AI console and choose HyperPod Clusters > Cluster Administration > Create HyperPod cluster.
2. Select Orchestrated by Amazon EKS from the checklist.
3. Choose Fast setup to provision a cluster with default networking, storage, and IAM sources, or select Customized setup to combine with present VPC, subnets, and safety teams.
4. Below Orchestration, both create a brand new EKS cluster or connect an present one. Confirm that the Use default Helm charts and add-ons choice is chosen in order that the Inference Operator and different required operators are put in robotically.
5. Below Occasion teams, add a employee group configured with the ml.p6-b300.48xlarge occasion sort.
6. Overview the configuration and select Submit to start provisioning.

For the entire walkthrough, seek advice from the Making a SageMaker HyperPod cluster with Amazon EKS orchestration documentation.

Step 2: Procure capability utilizing a Versatile Coaching Plan

The ml.p6-b300.48xlarge occasion sort requires reserved capability. A Versatile Coaching Plan supplies a dedicated capability reservation in your Blackwell GPU nodes, guaranteeing that p6-b300 situations can be found to your cluster with out competition from the final on-demand pool. Go to SageMaker Console, choose a FTP block primarily based in your timeline and occasion depend. To create or connect a coaching plan:

1. Within the occasion group configuration, select Coaching plan because the capability supply.
2. Choose an present plan that covers ml.p6-b300.48xlarge capability, or create a brand new reservation specifying the occasion depend and length that you really want.
3. Set the Goal Availability Zone to match the zone the place your coaching plan capability is allotted.

As soon as the cluster reaches an Lively state with wholesome p6-b300 nodes, you might be able to deploy the mannequin.

Deploying the mannequin

To deploy Kimi K3 on HyperPod, apply the next InferenceEndpointConfig manifest to your cluster:

apiVersion: inference.sagemaker.aws.amazon.com/v1
sort: InferenceEndpointConfig
metadata:
  identify: kimik3
spec:
  modelName: Kimi-K3
  instanceType: ml.p6-b300.48xlarge
  invocationEndpoint: v1/chat/completions
  replicas: 1
  modelSourceConfig:
    huggingFaceModel:
      modelId: moonshotai/Kimi-K3
    modelSourceType: huggingface
  employee:
    picture: vllm/vllm-openai:kimi-k3
    modelInvocationPort:
      containerPort: 8000
      identify: http
    modelVolumeMount:
      mountPath: /choose/ml/mannequin
      identify: model-weights
    sources:
      limits:
        nvidia.com/gpu: 8
      requests:
        nvidia.com/gpu: 8
    args:
      - "--model"
      - "moonshotai/Kimi-K3"
      - "--trust-remote-code"
      - "--load-format"
      - "fastsafetensors"
      - "--enable-prefix-caching"
      - "--enable-auto-tool-choice"
      - "--tool-call-parser"
      - "kimi_k3"
      - "--reasoning-parser"
      - "kimi_k3"
      - "--served-model-name"
      - "Kimi-K3"
      - "--moe-backend"
      - "auto"
      - "--tensor-parallel-size"
      - "8"
      - "--no-enable-flashinfer-autotune"
    environmentVariables:
      - identify: "VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION"
        worth: "1"

This yaml can also be supplied within the GitHub repository. Apply this configuration with:kubectl apply -f kimi-k3.yaml

The Inference Operator handles mannequin obtain from Hugging Face, container scheduling, well being checks, and endpoint readiness. As soon as the endpoint transitions to a prepared state, it exposes an OpenAI suitable API on the configured invocation path.

For groups that choose to handle their very own Kubernetes infrastructure, you’ll be able to deploy Kimi K3 on a standalone Amazon EKS cluster and procure GPU capability via EC2 Capability Blocks, which let you reserve p6-b300 situations for an outlined length with out long-term commitments.

Key deployment steps

The AI on EKS challenge supplies an inference-ready cluster recipe that automates the end-to-end provisioning. At a excessive stage, the deployment entails the next phases:

1. Provision the EKS cluster

Use the supplied Terraform modules to create a GPU-optimized EKS cluster. This consists of VPC networking, managed node teams, and the mandatory IAM roles and insurance policies for GPU workloads.

2. Reserve GPU capability with Capability Blocks

Create a Capability Block reservation for p6-b300.48xlarge situations in your goal Availability Zone. Capability Blocks assure that the requested GPU nodes shall be obtainable for the reserved time window. As soon as the reservation turns into energetic, the situations be a part of your EKS cluster as employee nodes.

3. Set up GPU drivers and machine plugin

The recipe installs the NVIDIA machine plugin and GPU drivers on the node group, in order that Kubernetes can uncover and schedule in opposition to the obtainable GPUs.

4. Deploy the vLLM inference server

A Helm chart or Kubernetes manifest deploys the vLLM container with Kimi K3 particular arguments, together with tensor-parallel dimension of 8, the MXFP4 load format, MoE backend configuration. The mannequin identifier factors to the Hugging Face repository, or instead, you’ll be able to sync the mannequin weights to Amazon Easy Storage Service (Amazon S3) for sooner mannequin loading. The serving arguments mirror these proven within the HyperPod configuration above.

5. Expose the inference endpoint

A Kubernetes Service (sort LoadBalancer or through an Ingress controller) exposes the vLLM server on port 8000, offering the OpenAI suitable /v1/chat/completions endpoint to your purposes.

6. Validate

Verify the deployment by sending a check request to the endpoint and verifying a profitable mannequin response.

For the complete deployment walkthrough, together with Terraform modules, Helm values, and step-by-step directions, seek advice from the AI on EKS Kimi K3 recipe.

As soon as deployed, the Kimi K3 endpoint exposes an OpenAI suitable chat completions API. You may invoke it utilizing the OpenAI Python SDK or a easy curl command.

Utilizing the OpenAI Python SDK

from openai import OpenAI
shopper = OpenAI(
    base_url="http://:8000/v1",
    api_key="not-needed"
)

response = shopper.chat.completions.create(
    mannequin="Kimi-K3",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain the benefits of mixture of experts architectures."}
    ],
    temperature=0.7,
    max_tokens=1024
)
print(response.decisions[0].message.content material)

Utilizing curl

curl -X POST http://:8000/v1/chat/completions 
  -H "Content material-Sort: software/json" 
  -d '{
    "mannequin": "Kimi-K3",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Explain the benefits of mixture of experts architectures."}
    ],
    "temperature": 0.7,
    "max_tokens": 1024
  }'

Exchange with the service endpoint uncovered by your HyperPod Inference Operator or EKS ingress configuration.

To keep away from ongoing costs, delete the sources you created throughout this walkthrough while you now not want them.For SageMaker HyperPod deployments:

1. Delete the InferenceEndpointConfig by working kubectl delete -f kimi-k3.yaml.
2. Within the SageMaker AI console, navigate to HyperPod Clusters, choose your cluster, and select Delete.
3. Launch or cancel your Versatile Coaching Plan reservation whether it is now not wanted.

For Amazon EKS deployments:

4. Delete the vLLM deployment and related Kubernetes companies.
5. Terminate the GPU node group or delete the EKS cluster utilizing Terraform (terraform destroy).
6. Launch your Capability Block reservation if it has not but expired.
7. For pricing particulars on p6-b300 situations and Capability Block reservations, seek advice from the Amazon EC2 pricing web page.

Kimi K3 represents a brand new frontier in open-weight mannequin capabilities, and AWS supplies the infrastructure and managed companies to deploy it at scale. Whether or not you select the streamlined HyperPod Inference Operator path or the pliability of a self-managed EKS cluster, the mixture of p6-b300 GPU situations, vLLM serving, and MXFP4 quantized weights delivers a deployment with built-in well being checks, auto-recovery, and endpoint readiness verification for the world’s largest open mannequin.To get began, listed below are some hyperlinks

1. SageMaker HyperPod Kimi K3 Instance
2. AI on EKS Kimi K3 Recipe
3. Making a SageMaker HyperPod Cluster (AWS Documentation)
4. Kimi K3 on Hugging Face
5. vLLM Documentation

In regards to the authors