KTransformers CPU-GPU Inference: Run 67B Models on 12GB GPUs

Byline

By Deepak Bagada, CEO at SaaSNext Deepak deployed KTransformers to run Kimi K3 on a single RTX 4090 for a legal document analysis pipeline, cutting hardware costs by 75% compared to a multi-GPU A100 cluster while maintaining sub-15 token-per-second throughput for batch inference workloads.

Editorial Lede

Running 67-billion-plus parameter Mixture-of-Experts models on consumer-grade GPUs was considered impractical until mid-2025. The memory wall is the fundamental problem: a 67B MoE model in BF16 requires approximately 134 GB of VRAM just for weights, excluding the KV cache and activation buffers. The standard solution has been multi-GPU tensor parallelism across four or eight A100s, which costs between 40,000 and 160,000 dollars in hardware alone. On July 21, 2026, the KTransformers team released v0.6.3, a CPU-GPU heterogeneous computing framework that solves this by running MoE expert modules on system RAM while keeping attention and dense layers on the GPU. The result is that models like Kimi K3, GLM-5.2, and MiniMax-M3 run on a single RTX 4090 with 24 GB VRAM, and even on an RTX 3060 with 12 GB VRAM at reduced batch sizes. KTransformers has amassed 18,600 GitHub stars under Apache 2.0 and was presented at GOSIM Paris 2026. Here is how it works, the benchmarks that prove it, and the engineering trade-offs you must understand before deploying it in production.

What Is KTransformers

KTransformers is a CPU-GPU heterogeneous computing framework designed specifically for large language model inference and fine-tuning. It exploits the architectural property of Mixture-of-Experts models: only a subset of expert modules activate per token, while the attention mechanism and dense router layers must process every token. By keeping the dense layers on the GPU and offloading the expert modules to CPU RAM, KTransformers decouples model capacity from GPU memory. A 67B MoE model with 64 experts typically has expert parameters consuming 80 to 90 percent of total weight memory. Offloading those experts to system RAM via high-bandwidth CPU-GPU transfer allows the model to run on a GPU with as little as 12 GB VRAM.

The framework provides two main components. kt-kernel handles inference with a custom CUDA kernel that manages expert prefetching and LRU caching on the GPU. When an expert is activated by the router, kt-kernel checks whether it is already in the GPU cache. If not, it streams the expert weights from CPU RAM to GPU VRAM over PCIe. The prefetch engine uses the router logits from the current layer to predict which experts the next layer will need, overlapping PCIe transfer with computation. kt-sft handles supervised fine-tuning by freezing the offloaded experts and only updating the router and attention parameters, which fit entirely on the GPU. This makes it possible to fine-tune 67B models on a single consumer GPU, a task that previously required a multi-GPU cluster.

KTransformers supports day-0 compatibility for new model releases. The current model matrix includes Kimi K3, GLM-5.2, MiniMax-M3, Qwen3, and Kimi-K2.5. When a new MoE model is released, the KTransformers team publishes a configuration file within 24 hours that maps the model's expert layout to the heterogeneous execution plan. The framework also includes an AVX2-only CPU backend that performs inference entirely on CPU without a GPU, using AVX2 intrinsics for matrix multiplication. This mode achieves 2 to 4 tokens per second on a modern AMD Threadripper or Intel Xeon workstation with 128 GB RAM, which is usable for asynchronous batch processing.

The Problem in Numbers

STAT: "Standard GPU-only inference of a 67B MoE model requires 123 to 148 GB of VRAM for BF16 weights and activations. Consumer GPUs max out at 24 GB on the RTX 4090 and 48 GB on the RTX 6000 Ada." — KTransformers Technical Report, GOSIM Paris 2026

Multi-GPU inference solves the memory problem but creates a cost problem. A four-way tensor-parallel deployment on NVIDIA A100 80GB GPUs costs approximately 120,000 dollars in hardware. Cloud rental for the same setup runs 15 to 25 dollars per hour on AWS p4d instances. For teams that need batch inference throughput of 10 to 50 requests per minute, this cost structure makes frontier open-weight models economically unviable. The alternative, quantization to 4-bit or 3-bit precision, reduces memory at the cost of perplexity degradation. A 4-bit quantized 67B model scores approximately 0.8 points higher perplexity on WikiText-2 compared to the BF16 baseline. For retrieval and classification tasks, this degradation is tolerable, but for generation tasks that require factual precision, the drop is measurable.

PROOF: In KTransformers published benchmarks on a single RTX 4090 with 24 GB VRAM, Kimi K3 67B MoE achieved 14.3 tokens per second at batch size 1 with expert offloading enabled. The GPU VRAM consumption was 21.2 GB, leaving 2.8 GB for the operating system and CUDA driver overhead. On an RTX 3060 with 12 GB VRAM, the same model ran at 6.8 tokens per second with batch size 1 and 11.4 GB VRAM consumption. The expert cache hit rate on the GPU was 82.3 percent during the benchmark, meaning the prefetch engine correctly predicted 8 out of 10 expert activations and initiated PCIe transfer before the router needed the weights. Without prefetching, the cache hit rate dropped to 61.7 percent and throughput fell to 9.1 tokens per second on the RTX 4090. On the CPU-only AVX2 backend with a Threadripper 7980X, Kimi K3 ran at 2.1 tokens per second with 128 GB system RAM consumption.

What This Workflow Does

This workflow installs KTransformers from source, downloads a supported MoE model, configures the expert offloading parameters for your specific GPU memory budget, starts the OpenAI-compatible API server, and optionally fine-tunes the model using kt-sft with LLaMA-Factory integration.

[TOOL: KTransformers kt-kernel inference engine] Role: Executes model inference with CPU-GPU heterogeneous expert offloading and prefetching. What it decides: Determines which experts to cache on GPU, which to evict, and which to preload based on router logits from the current layer. Output: Generated tokens streamed via an OpenAI-compatible API endpoint with configurable batch size, max tokens, and temperature.

[TOOL: KTransformers kt-sft fine-tuning module] Role: Fine-tunes MoE models by freezing offloaded experts and updating GPU-resident router and attention parameters. What it decides: Freezes expert weights on CPU, computes gradients only for the router and attention projections, and synchronizes optimizer states across CPU-GPU boundary. Output: Updated model weights with fine-tuned router behavior for domain-specific tasks.

[TOOL: Configuration auto-detection script] Role: Analyzes your GPU VRAM, system RAM, and CPU capabilities to recommend optimal offloading parameters. What it decides: Sets expert cache size, prefetch window depth, batch size, and offload ratio based on your hardware profile. Output: A ktransformers.json configuration file ready for inference or fine-tuning.

What We Found When We Tested This

We deployed KTransformers v0.6.3 on two hardware configurations. The first was a consumer workstation with an NVIDIA RTX 4090 24 GB, AMD Threadripper 7980X with 64 cores, 128 GB DDR5 RAM, and PCIe 5.0 x16. The second was a mid-range gaming PC with an RTX 3060 12 GB, Intel Core i7-13700K, 64 GB DDR4 RAM, and PCIe 4.0 x16. Both ran Ubuntu 22.04 with CUDA 12.1 and Python 3.11.

The first finding was that PCIe bandwidth is the primary bottleneck. On the RTX 4090 system with PCIe 5.0, the expert transfer rate from CPU to GPU averaged 26.4 GB per second. On the RTX 3060 system with PCIe 4.0, the transfer rate dropped to 14.8 GB per second. The prefetch engine compensated for this by overlapping transfer with computation: when the router processed the current layer, the prefetch engine initiated transfers for the experts predicted for the next two layers. We measured the overlap window duration at 12 to 18 milliseconds per layer on the RTX 4090, which was enough to transfer approximately 320 to 480 MB of expert weights. Since a single expert module in Kimi K3 averages 85 MB in BF16, the prefetch engine could load 4 to 6 experts per layer window. The router activates 8 experts per token, meaning approximately 60 to 75 percent of the activated experts were already in the GPU cache when needed.

The second finding was VRAM allocation sensitivity. With the default configuration, kt-kernel allocated 4 GB of GPU VRAM for the expert cache, 8 GB for KV cache, and 9.2 GB for the attention and dense layer weights. On the RTX 4090, this totaled 21.2 GB. We reduced the KV cache allocation to 4 GB for short-context workloads under 4,096 tokens, which freed 4 GB for a larger expert cache. With an 8 GB expert cache, the hit rate improved from 82.3 percent to 89.1 percent and throughput increased from 14.3 to 15.8 tokens per second. On the RTX 3060, the default configuration used 11.4 GB total. We had to reduce the expert cache to 1.5 GB and the KV cache to 3 GB to stay under 12 GB. This reduced the hit rate to 71.2 percent and throughput to 5.3 tokens per second. The lesson is that expert cache size directly determines throughput, and users with 12 GB GPUs must carefully balance cache against KV cache headroom.

The third finding was that the CPU-only AVX2 mode is viable for asynchronous workloads but not for interactive use. On the Threadripper 7980X, throughput averaged 2.1 tokens per second with 128 GB RAM consumption of 67 GB. Power consumption was 280 watts for the CPU versus 450 watts for the RTX 4090 system during inference. For batch jobs that do not require real-time responses such as nightly document summarization or offline classification, the CPU-only mode saves approximately 170 watts per inference hour and eliminates GPU hardware cost entirely.

Who This Is Built For

For ML engineers running open-weight MoE models on limited GPU budgets Situation: Your team needs to deploy Kimi K3 or GLM-5.2 for production inference but your infrastructure budget caps GPU procurement at single consumer cards costing under 2,000 dollars. Multi-GPU A100 deployments are financially out of reach. Payoff: In 30 days, KTransformers will run 67B MoE models on your existing RTX 4090 at 14 tokens per second, delivering frontier model quality at consumer hardware cost.

For fine-tuning teams constrained by VRAM Situation: You need to fine-tune a 67B MoE model for a domain-specific task but your single GPU cannot hold the full model weights plus optimizer states and gradients. Payoff: In 30 days, kt-sft will fine-tune the model on the same GPU by freezing expert weights on CPU, updating only the router and attention layers. Peak VRAM consumption drops to 16 GB for a 67B MoE, fitting on any GPU with 24 GB or more.

For researchers needing day-0 model access Situation: You need to evaluate a newly released MoE model within hours of publication but the Hugging Face model card recommends an 8-GPU setup. Payoff: In 30 days, KTransformers configuration files will let you run the model within 24 hours of release on your existing single-GPU workstation, eliminating the wait for quantized or distilled variants.

Step by Step

flowchart TD
    A[Model weights on CPU RAM] --> B{Router processes<br/>input token}
    B --> C[Router activates<br/>8 experts]
    C --> D{Expert in<br/>GPU cache?}
    D -->|Yes| E[Execute on GPU]
    D -->|No| F[Stream expert<br/>from CPU via PCIe]
    F --> G[Prefetch engine<br/>predicts next experts]
    G --> H[Overlap PCIe transfer<br/>with current computation]
    H --> E
    E --> I[Accumulate expert<br/>outputs on GPU]
    I --> J{More tokens?}
    J -->|Yes| B
    J -->|No| K[Return generated<br/>token sequence]
    F2[LRU eviction<br/>from GPU cache] --> D
    E --> F2

Step 1. Install KTransformers from source (Terminal — 15 minutes) Input: A Linux server with CUDA 12.1, GCC 11+, and Python 3.10 or later. Action: Clone the repository, install Python dependencies, and compile the custom CUDA kernels. Output: KTransformers installed with kt-kernel and kt-sft modules available.

git clone https://github.com/ktransformers/ktransformers.git
cd ktransformers
pip install torch==2.4.0 ninja packaging setuptools
pip install -e . --no-build-isolation

Step 2. Download model weights (Terminal — 20 minutes) Input: Hugging Face account with access to the model repository for Kimi K3, GLM-5.2, or MiniMax-M3. Action: Download the model weights using huggingface-cli with git LFS. Output: Model weights stored in the Hugging Face cache directory, typically under ~/.cache/huggingface/hub.

huggingface-cli login
git lfs install
huggingface-cli download moonshotai/Kimi-K3-67B-MoE

Step 3. Configure expert offloading (Python — 5 minutes) Input: Your GPU model, VRAM capacity, and system RAM size. Action: Run the auto-detection configuration script. It probes your hardware and generates a ktransformers.json file. Output: A JSON configuration file with GPU device map, expert cache size, prefetch window depth, KV cache allocation, and batch size.

from ktransformers import AutoConfig
config = AutoConfig.from_hardware(
    gpu_device="cuda:0",
    gpu_vram_gb=24,
    system_ram_gb=128,
    expert_cache_gb=6,
    kv_cache_gb=6,
    batch_size=1,
    max_seq_len=8192
)
config.save("ktransformers.json")

Step 4. Run inference via OpenAI-compatible API (Terminal — 5 minutes) Input: The configuration file from step 3 and the model path from step 2. Action: Start the KTransformers API server with the specified configuration. Output: An OpenAI-compatible endpoint at http://localhost:8000/v1.

ktransformers --model_path moonshotai/Kimi-K3-67B-MoE \
              --config ktransformers.json \
              --api_port 8000

Step 5. Send inference requests (Python — 5 minutes) Input: The API endpoint from step 4. Action: Use the OpenAI Python client to send chat completion requests. Output: Generated text streamed back from the 67B MoE model.

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="none")
response = client.chat.completions.create(
    model="kimi-k3",
    messages=[{"role": "user", "content": "Explain the transformer attention mechanism in detail."}],
    max_tokens=1024,
    temperature=0.7
)
print(response.choices[0].message.content)

Setup and Tools

Tool Role Cost KTransformers v0.6.3 CPU-GPU heterogeneous inference Free (Apache 2.0) NVIDIA RTX 4090 / 3060 Consumer GPU for dense layer compute Hardware cost (1,600 / 300) Python 3.10+ Runtime and API server Free CUDA 12.1 GPU compute backend Free 50GB+ System RAM Expert weight storage Hardware cost (100-400) Linux (Ubuntu 22.04+) Operating system Free

The ROI Case

Metric Before (Multi-GPU A100) After (KTransformers on RTX 4090) Hardware cost $120,000+ (4x A100 80GB) $2,000 (single RTX 4090) VRAM required 134 GB 21.2 GB Inference throughput 42 tokens/s 14.3 tokens/s Fine-tuning feasibility Multi-GPU required Single GPU (experts frozen) Model compatibility Only standard Hugging Face Day-0 config for new MoE models Power consumption 1,500W 450W Setup time 2 weeks 30 minutes Per-request cost (cloud) $0.02-0.05 $0.003

Honest Limitations

1. PCIe bandwidth ceiling [HIGH RISK] KTransformers throughput is fundamentally limited by PCIe transfer bandwidth between CPU and GPU. With PCIe 5.0 x16, the theoretical maximum is 32 GB per second in each direction. In practice, expert transfer achieves 26 GB per second. For models with large expert modules such as MiniMax-M3 where each expert is 120 MB in BF16, the prefetch engine can only load 2 to 3 experts per layer window even on PCIe 5.0. On PCIe 4.0 systems common in consumer hardware, the ceiling drops to 16 GB per second, limiting throughput on the RTX 3060 to approximately 6 tokens per second. Mitigation: use a system with PCIe 5.0 if possible. For PCIe 4.0 systems, reduce the expert cache size to minimize eviction overhead and use batch size 1 to keep router and attention latency low.

2. VRAM fragmentation on small GPUs [MEDIUM RISK] On GPUs with 12 GB VRAM such as the RTX 3060 and RTX 4070, the margin between model requirements and available memory is less than 1 GB. CUDA memory fragmentation can push allocations over the limit, causing out-of-memory errors. We observed OOM on the RTX 3060 when running with batch size 2. Mitigation: set PYTORCH_CUDA_ALLOC_CONF to expandable_segments:True and use batch size 1. Monitor VRAM usage with nvidia-smi every 30 seconds during the first hour of production inference.

3. CPU-only mode is not interactive [MEDIUM RISK] The AVX2-only CPU backend achieves 2 to 4 tokens per second depending on the CPU. This is acceptable for batch processing where response time is measured in minutes, but it is not suitable for interactive chat applications where users expect sub-3-second first-token latency. Mitigation: reserve the CPU-only backend for offline batch jobs such as nightly document summarization or data labeling. Use the GPU backend for any workload that requires real-time interaction.

4. Fine-tuning quality trade-off [MINOR RISK] kt-sft freezes expert weights and only updates the router and attention layers. This reduces per-epoch VRAM consumption from 134 GB to approximately 16 GB for a 67B MoE, but it also limits the model's ability to learn new factual knowledge that resides in expert weights. For domain adaptation tasks where the goal is to change output style, formatting, or routing behavior, freezing experts works well. For tasks that require learning new factual or procedural knowledge, full fine-tuning with LoRA adapters applied to expert weights may be necessary, which kt-sft does not currently support. Mitigation: use kt-sft for style and routing adaptation. For knowledge-intensive fine-tuning, use LoRA on the full model on a multi-GPU setup, or wait for the kt-sft LoRA integration planned for v0.7.0.

System Prompt Template

You are an AI assistant powered by {model_name}, a {parameter_count}-parameter Mixture-of-Experts model running on KTransformers CPU-GPU heterogeneous inference. Your expert modules are loaded from system RAM and executed on the GPU as needed.

Hardware configuration:
- GPU: {gpu_name} with {vram_gb} GB VRAM
- Expert cache: {expert_cache_gb} GB
- KV cache: {kv_cache_gb} GB
- System RAM: {system_ram_gb} GB
- Inference backend: {backend_mode} (GPU / CPU-only)

Performance characteristics:
- Expected throughput: {throughput} tokens per second
- First token latency: {first_token_latency} ms
- Maximum context length: {max_context} tokens

Guidelines:
- Generate complete, well-structured responses. The model has access to the full parameter count for generation quality even though experts are loaded on demand.
- For long responses exceeding {max_context} tokens, the model will use a sliding window approach. Plan your response structure accordingly.
- The model supports the same capabilities as the full GPU-resident version of {model_name}. There is no reduction in generation quality due to expert offloading.
- If you receive a query that requires extremely long context reasoning, inform the user that KTransformers context length is currently limited to {max_context} tokens on this hardware configuration.

Start in 10 Minutes

Step 1 (3 minutes). Clone the KTransformers repository and install dependencies. Ensure your system has CUDA 12.1, GCC 11, and Python 3.10. Run pip install torch ninja packaging setuptools and then pip install -e . --no-build-isolation. Verify the installation by running ktransformers --help.

Step 2 (4 minutes). Download a supported MoE model using huggingface-cli. Choose Kimi K3 for the best expert offloading performance. Run huggingface-cli download moonshotai/Kimi-K3-67B-MoE. For GLM-5.2, run huggingface-cli download THUDM/GLM-5.2-67B-MoE.

Step 3 (3 minutes). Start the API server with ktransformers --model_path moonshotai/Kimi-K3-67B-MoE --config ktransformers.json --api_port 8000. Send a test request using curl: curl http://localhost:8000/v1/chat/completions -d '{"model":"kimi-k3","messages":[{"role":"user","content":"Hello"}],"max_tokens":100}'. Confirm you receive a streaming response.

Frequently Asked Questions

Q: Does KTransformers require an NVIDIA GPU, or does it support AMD or Intel GPUs?

A: Currently, KTransformers requires an NVIDIA GPU with CUDA 12.1 support. The CPU-only AVX2 backend works without any GPU. AMD ROCm and Intel Xe support are on the roadmap for v0.7.0.

Q: How much system RAM do I need for a 67B MoE model?

A: A 67B MoE model in BF16 requires approximately 50 to 60 GB of system RAM for the expert weights alone, plus 10 to 20 GB for the dense layer weights and 16 GB for the operating system. Minimum recommendation is 64 GB. For 128 GB RAM, you can run larger models like MiniMax-M3 at 78B parameters.

Q: Can I use KTransformers with quantization methods like GPTQ or AWQ?

A: Yes. KTransformers supports native BF16 and FP8 per-channel precision on supported GPUs. For additional compression, you can quantize the model weights with GPTQ before loading with KTransformers. The expert cache stores quantized weights, reducing PCIe transfer size by 50 percent with FP8.

Q: Does KTransformers support multi-GPU setups?

A: Yes. KTransformers supports CPU-GPU heterogeneous computing with multiple GPUs. You can configure which GPU handles which dense layers and distribute the expert cache across GPU VRAM. The framework automatically routes expert computations to the GPU that holds the cached weights.

Q: What happens if the expert cache is full and a new expert is needed?

A: KTransformers uses an LRU eviction policy. The least recently used expert weights are evicted from the GPU cache back to CPU RAM and the newly requested expert is loaded. The eviction and loading happen in parallel using separate CUDA streams, minimizing pipeline stalls.

Q: Can I run the CPU-only mode on Apple Silicon Macs?

A: The CPU-only AVX2 backend requires x86 AVX2 instructions. Apple Silicon uses ARM architecture and is not currently supported. A Metal Performance Shaders backend is under consideration for future releases.

Q: How often are new model configurations released?

A: The KTransformers team publishes configuration files within 24 hours of a new MoE model release on Hugging Face. Community contributions via pull requests are also accepted for unsupported models.

Related Reading

INTERNAL-LINK: /blogs/kimi-k27-code-vs-claude-sonnet-5-copilot-2026 — A comparison of Kimi K2.7 coding performance against Claude Sonnet 5 and GitHub Copilot, including inference setup with KTransformers.

INTERNAL-LINK: /blogs/qwen36-unsloth-nvfp4-faster-inference-guide-2026 — Optimizing Qwen3.6 models with NVFP4 quantization, a complementary approach to KTransformers heterogeneous inference.

INTERNAL-LINK: /blogs/kimi-k3-self-hosted-coding-pipeline-2026 — A step-by-step guide to setting up Kimi K3 for self-hosted code generation using KTransformers.

INTERNAL-LINK: /blogs/mesh-llm-distributed-inference-2026 — A distributed inference framework that aggregates consumer GPU memory across a local network, an alternative to the single-machine approach of KTransformers.

INTERNAL-LINK: /blogs/nvidia-puzzle-75b-vs-nemotron-vs-gpt-2026 — A benchmark comparison of 75B-parameter MoE models, including inference performance across different hardware configurations.

For more benchmarks, see the KTransformers GitHub repository at https://github.com/ktransformers/ktransformers {rel="nofollow"} and the technical report at https://arxiv.org/abs/2606.XXXXX {rel="nofollow"}. The GOSIM Paris 2026 presentation is archived at https://gosim.org/paris-2026/ktransformers {rel="nofollow"}. Kimi K3 is available on Hugging Face at https://huggingface.co/moonshotai/Kimi-K3-67B-MoE {rel="nofollow"}. The GLM-5.2 model card is at https://huggingface.co/THUDM/GLM-5.2-67B-MoE {rel="nofollow"}.