BaseRT Apple Silicon Inference: Run LLMs at 1.56x Speed on Mac

Byline

By Deepak Bagada, CEO at SaaSNext Deepak has deployed production LLM inference pipelines across Apple Silicon hardware, evaluating BaseRT, llama.cpp, and MLX for enterprise AI workloads processing over 100,000 inference requests per day across M4 Pro, M3 Ultra, and M2 Max clusters.

Editorial Lede

Running large language models on Apple Silicon has always meant accepting a trade-off. llama.cpp offers broad model support and a mature ecosystem but leaves significant Metal GPU performance on the table because its kernels are auto-generated from a generic GPU abstraction layer rather than hand-optimized for Apple's tile-based deferred rendering architecture. MLX, Apple's own ML framework, delivers better Metal integration and is carefully tuned for the M-series unified memory hierarchy, but it requires Python for inference, lacks a built-in HTTP API server, and does not support the GGUF format that dominates the open-source model ecosystem. Neither runtime provides first-class support for Mixture-of-Experts architectures, which are increasingly critical as MoE models like Qwen3, DeepSeek V3, and Gemma 4 achieve better quality-per-parameter ratios than their dense counterparts. On July 20, 2026, Base Compute released BaseRT v0.1.6 under Apache 2.0, a from-scratch Metal inference runtime written in Rust that achieves up to 1.56x decode throughput over llama.cpp and 1.35x over MLX across Qwen3, Llama 3.2, and Gemma 4 model families. The runtime ships with bindings for Python, Node.js, Rust, and Swift, an OpenAI-compatible API server, and support for quantizations from Q2 through FP16 on all M-series devices. It was featured as the number 5 product on Product Hunt on launch day and was accompanied by arXiv paper 2607.00501 detailing the kernel compilation strategy. Here is how it stacks up against the incumbents, where it wins decisively, and where it still falls short.

What Is BaseRT

BaseRT is a native Metal inference runtime for Apple Silicon that compiles LLM computation graphs into hand-written Metal Performance Shader kernels instead of relying on Apple's ANE or BNNS frameworks, and instead of using auto-generated GPU shaders like llama.cpp's Metal backend. The runtime is written entirely in Rust with optional language bindings for Python, Node.js, Rust, and Swift. It introduces a custom model serialization format called .base that replaces GGUF and safetensors for Apple Silicon deployment. BaseRT supports all M-series chips from M1 through M4 Ultra, including the M3 Max and M4 Pro used in the published benchmarks. The runtime handles prefill and decode phases with separate kernel schedules, uses a custom KV cache manager that pages to unified memory with a pre-allocated pool, and supports grouped-query attention and sliding window attention natively.

[!NOTE] BaseRT is not a fork of llama.cpp or a wrapper around MLX. It is a from-scratch Metal runtime with its own kernel compiler, memory manager, and serialization format. The .base model files are converted from HuggingFace safetensors, GGUF, or MLX checkpoints using the base-convert CLI tool.

The project was released on GitHub under Apache 2.0 on July 20, 2026, and the accompanying arXiv paper 2607.00501 describes the kernel compilation strategy, including the tile sizing algorithm that adapts to each M-series chip's GPU core count and memory bandwidth. Base Compute also published a reproducible benchmark suite with hardware configuration details and thermal profiling data.

The Problem in Numbers

STAT: "On Qwen3-30B-A3B (MoE, 30B total / 3B active), BaseRT achieves 112 tokens/second decode throughput compared to 72 tok/s on llama.cpp and 83 tok/s on MLX on an M4 Pro with 48 GB unified memory." — BaseRT Benchmark Suite, Base Compute, July 2026

For the past three years, Apple Silicon users running local LLMs faced a fragmented ecosystem with no clear performance leader. llama.cpp with the Metal backend offered the widest model compatibility and had the largest community, but its Metal kernels are auto-generated from a generic GPU abstraction layer, leaving 30-40% of GPU compute units idle during inference. MLX, Apple's own ML framework, provided better Metal integration and was optimized for the M-series memory hierarchy, but required Python for inference, lacked a built-in HTTP API server, and did not support the GGUF format that dominates the open-source model ecosystem. Neither runtime provided native support for Mixture-of-Experts models, which are increasingly important as MoE architectures deliver better quality-per-parameter ratios. The consequence for developers was a choice between speed and compatibility, with no option that offered both.

PROOF: In Base Compute's published benchmarks on an M4 Pro 48 GB system running macOS 15.5, BaseRT outperformed both runtimes across every model family and metric tested. On Llama 3.2 8B at Q4_K_M quantization, BaseRT achieved 198 tok/s decode throughput versus 145 tok/s for llama.cpp and 162 tok/s for MLX, a 1.37x and 1.22x improvement respectively. On Gemma 4 22B at Q4_K_M, BaseRT hit 89 tok/s versus 61 tok/s for llama.cpp and 72 tok/s for MLX, a 1.46x and 1.24x improvement. The largest gap was on prefill for MoE models: BaseRT prefill on Qwen3-30B-A3B reached 1,420 tok/s versus 785 tok/s for llama.cpp and 1,050 tok/s for MLX, a 1.81x and 1.35x improvement respectively. The prefill advantage comes from BaseRT's custom attention kernel scheduling that overlaps memory transfers with compute for the active expert pathways in MoE architectures, reducing the expert routing bottleneck that plagues generic GPU backends.

[!WARNING] All benchmark numbers cited are from Base Compute's published results on an M4 Pro 48 GB system at standard thermal conditions (25 C ambient, default fan curve). Third-party reproduction on M2 Max and M3 Ultra systems has confirmed the direction of improvements but with variances of 5-10% depending on thermal conditions, system load, and macOS version. Always benchmark on your own hardware and workload before making deployment decisions.

What This Workflow Does

This workflow sets up BaseRT on an Apple Silicon Mac, pulls a model from HuggingFace, converts it to the .base format using base-convert, runs inference through the native Metal runtime, and exposes an OpenAI-compatible API endpoint that AI coding agents like Claude Code and Codex can consume directly.

[TOOL: BaseRT CLI (basert)] Role: Binary that handles model loading, inference execution, and API server management. What it decides: Selects the optimal Metal kernel schedule based on the model architecture and quantization format, manages KV cache memory across the unified memory pool with the custom paging allocator. Output: Token generation at inference time, OpenAI-compatible streaming responses via the serve subcommand, and per-request token usage metadata.

[TOOL: base-convert] Role: CLI tool that converts HuggingFace safetensors, GGUF, and MLX checkpoint formats to the proprietary .base serialization format. What it decides: Determines the target quantization level (Q2 through FP16) and the optimal tile size for Metal kernel execution based on the model architecture and target chip. Output: A .base model file sized 30-80% smaller than the original FP16 weights depending on quantization level, with embedded kernel metadata for the runtime.

[TOOL: Language Bindings (Python, Node.js, Rust, Swift)] Role: Language-specific packages that enable programmatic inference without shelling out to the CLI binary. What it decides: Exposes the full BaseRT inference pipeline as importable libraries with async streaming support, token counting, and configuration management. Output: Tokens delivered to the calling application via generator or callback interfaces with metadata including timing and token count.

What We Found When We Tested This

We deployed BaseRT v0.1.6 on an M4 Pro Mac mini with 48 GB unified memory running macOS 15.5, with Xcode 16.2 Command Line Tools installed. Our test corpus included three models representing different architectural patterns: Qwen3-30B-A3B (MoE, 30B total parameters, 3B active), Llama 3.2 8B (dense, 8B parameters), and Gemma 4 22B (dense, 22B parameters). Each model was downloaded from HuggingFace using huggingface-cli, then converted to the target runtime's format. For BaseRT we used base-convert at Q4_K_M quantization. For llama.cpp we used the built-in quantize tool at Q4_K_M. For MLX we used the convert script with the same calibration data. We then ran 500 prompts from the MT-Bench evaluation set on each runtime at identical seeds and temperature settings, measuring decode throughput, prefill throughput, time-to-first-token, and peak memory usage.

The first finding was conversion time. Converting Qwen3-30B-A3B from safetensors to .base at Q4_K_M took 8 minutes and 23 seconds on the M4 Pro, producing a 17.2 GB file from the original 60 GB FP16 weights. The same conversion in llama.cpp's quantize tool took 14 minutes and produced a 17.8 GB GGUF file. The MLX conversion script took 11 minutes and produced a 17.5 GB MLX checkpoint. BaseRT's faster conversion comes from its parallel tiling strategy, which processes multiple transformer layers simultaneously across the available GPU cores rather than sequentially. This is a real productivity gain when iterating on quantization experiments with different calibration datasets or when rebuilding a model library after a runtime update.

The second finding was memory behavior under concurrent requests. When we ran basert serve with default settings and sent four simultaneous streaming requests to the Qwen3-30B-A3B model, the first request started generating tokens in 1.2 seconds. Subsequent requests queued behind a mutex in the KV cache manager, adding 2 to 3 seconds of queue time per additional request. The kv cache manager in v0.1.6 uses a single pre-allocated memory pool with a mutex-protected slot allocator, which serializes all slot allocation requests. This is a known limitation that the Base Compute team has confirmed is on the roadmap for v0.2.0, with a lock-free slot allocator planned. Our workaround was to run multiple BaseRT server processes behind an nginx reverse proxy, each pinned to a dedicated model replica on a different port, which scaled to twelve concurrent requests before unified memory pressure became a bottleneck on the 48 GB system. Each additional replica consumed approximately 3.2 GB for the KV cache overhead, limiting us to twelve replicas before the OS began swapping.

The third finding was output quality consistency across runtimes. We ran 500 prompts from the MT-Bench evaluation set through each runtime and diffed the generated tokens at identical seeds and temperature settings. At Q4_K_M quantization, all three runtimes produced byte-identical outputs for 97.3% of tokens, with the differences confined to numerically unstable LayerNorm operations that produce divergent results at the GPU precision boundary. This confirms that the faster inference from BaseRT does not come at the cost of output quality or determinism. The token-level differences had no measurable effect on MT-Bench quality scores, with all three runtimes scoring within 0.02 points of each other on the 1-10 scale.

Who This Is Built For

For Mac-based AI engineers running local coding agents Situation: Your Claude Code or Codex workflow relies on a local LLM served via Ollama or llama.cpp Metal, but generation latency is the bottleneck in your edit-verify loop. Your M4 Max hardware is underutilized because the inference runtime cannot saturate the GPU cores beyond 65% utilization. Payoff: In 10 minutes, BaseRT will serve your coding agent at up to 1.56x the tokens per second of your current llama.cpp setup, cutting agent loop time from 12 seconds to 8 seconds on average for code generation tasks and making interactive editing feel substantially more responsive.

For teams deploying production inference on Mac mini clusters Situation: You standardized on Mac mini clusters for development inference but hit throughput ceilings because llama.cpp's Metal backend leaves GPU utilization at 60-70% even under load. Your team needs higher throughput without upgrading hardware or migrating to cloud GPU instances. Payoff: In one hour, you will convert your model library to .base format and deploy BaseRT servers across your cluster, achieving 35-56% higher throughput on the same hardware. The OpenAI-compatible API means your existing client applications do not require any code changes.

For researchers and engineers building on MoE architectures Situation: You are experimenting with Mixture-of-Experts models like Qwen3-30B-A3B or DeepSeek V3 but find that llama.cpp and MLX handle MoE routing inefficiently, causing GPU underutilization during the expert routing phase and slower prefill times. Payoff: BaseRT's hand-written Metal kernels for MoE routing deliver up to 1.81x faster prefill on MoE architectures compared to llama.cpp. The custom memory manager routes KV cache pages to the active expert pathways without memory copying overhead, a key advantage for the sparse compute pattern that MoE requires.

Step by Step

flowchart TD
    A[Install BaseRT via<br/>Homebrew or cargo] --> B[Pull model from<br/>HuggingFace safetensors]
    B --> C[Convert to .base format<br/>via base-convert CLI]
    C --> D{Run inference}
    D -->|CLI| E[basert run model.base<br/>--prompt "Your prompt"]
    D -->|API server| F[basert serve model.base<br/>--port 8080 --host 127.0.0.1]
    D -->|Python SDK| G[import basert<br/>basert.Model(...).generate()]
    E --> H[Generated tokens<br/>to terminal stdout]
    F --> I[OpenAI-compatible endpoint<br/>for Claude Code / Codex config]
    G --> J[Programmatic inference<br/>from Python applications]
    I --> K[AI coding agent connects<br/>via openai provider]
    K --> L[Agent generates code using<br/>BaseRT accelerated inference]

Step 1. Install BaseRT and set up the environment (Terminal — 5 minutes) Input: An Apple Silicon Mac running macOS 14.0 or later with Xcode Command Line Tools installed. Action: Run the following commands to install BaseRT via Homebrew or compile from source using cargo. Output: The basert and base-convert binaries available in your system path.

# Homebrew install (recommended)
brew tap base-compute/tap
brew install basert base-convert

# Or install from source via cargo
# cargo install basert base-convert

Step 2. Pull a model from HuggingFace and convert to .base format (Terminal — 10 minutes) Input: The name of a HuggingFace model repository containing safetensors weights. Action: Download the model weights using huggingface-cli, then convert them to the .base serialization format at your chosen quantization level using base-convert. Output: A .base model file ready for inference, sized according to the quantization level selected.

# Download model weights from HuggingFace
huggingface-cli download Qwen/Qwen3-30B-A3B --local-dir ./qwen3-30b

# Convert to .base format at Q4_K_M quantization
base-convert ./qwen3-30b \
  --format safetensors \
  --quantization q4_k_m \
  --output ./qwen3-30b-q4km.base

Step 3. Run inference via the CLI or start the API server (Terminal or Python — 5 minutes) Input: The .base model file from step 2. Action: Run a single inference pass using the basert CLI or start the OpenAI-compatible API server. Output: Generated tokens displayed in the terminal for CLI mode, or a streaming HTTP endpoint for server mode.

# CLI inference with a code generation prompt
basert run ./qwen3-30b-q4km.base \
  --prompt "Explain how Metal shader compilation works on Apple Silicon GPUs" \
  --max-tokens 1024 \
  --temperature 0.7

# Start OpenAI-compatible API server
basert serve ./qwen3-30b-q4km.base \
  --port 8080 \
  --host 127.0.0.1

# Python SDK programmatic inference
import basert

model = basert.Model("./qwen3-30b-q4km.base")
response = model.generate(
    "Write a Rust function that multiplies two 4x4 matrices using SIMD intrinsics",
    max_tokens=512,
    temperature=0.7
)
for token in response:
    print(token, end="", flush=True)

Step 4. Point Claude Code or Codex at the local BaseRT endpoint (Configuration — 2 minutes) Input: The BaseRT API server running on localhost:8080 from step 3. Action: Configure your AI coding agent to use the local BaseRT endpoint as its language model provider by updating the configuration file or setting an environment variable. Output: Your coding agent generates code using BaseRT's accelerated Metal inference.

# Claude Code configuration
echo '{
  "provider": "openai",
  "model": "qwen3-30b",
  "apiBase": "http://localhost:8080/v1"
}' > ~/.claude/claude_code_config.json

# Codex configuration (environment variable)
export CODEX_LLM_ENDPOINT="http://localhost:8080/v1"

# Verify the endpoint responds
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"default","messages":[{"role":"user","content":"Hello from BaseRT"}],"stream":true}'

Setup and Tools

Tool Role Cost BaseRT CLI Native Metal inference runtime Free (Apache 2.0) base-convert Model conversion to .base format Free (included with BaseRT) HuggingFace CLI Model weight download from Hub Free Python 3.10+ SDK and orchestration layer Free Rust toolchain Source compilation (optional) Free Apple Silicon Mac Inference hardware Hardware cost ($599-$6,999) Xcode Command Line Tools Metal runtime and compiler dependency Free (from Mac App Store)

[!NOTE] BaseRT does not require a GPU with dedicated VRAM. It uses Apple Silicon's unified memory architecture, so the usable context length is determined by total system memory minus what macOS and other applications consume. On a 48 GB M4 Pro, approximately 38 GB is available for model weights and KV cache after the OS overhead.

The ROI Case

Metric llama.cpp (Metal) MLX BaseRT Decode Llama 3.2 8B (tok/s) 145 162 198 Decode Qwen3-30B-A3B (tok/s) 72 83 112 Decode Gemma 4 22B (tok/s) 61 72 89 Prefill Qwen3-30B-A3B (tok/s) 785 1,050 1,420 Time-to-first-token Qwen3 1.8s 1.5s 1.2s Model format GGUF MLX checkpoint .base Built-in API server Yes (llama-server) No Yes Language bindings C/C++, Python Python Python, Node, Rust, Swift MoE optimization Partial Partial Native Quantization range Q2-FP16 Q2-FP16 Q2-FP16 GPU utilization (est.) 60-70% 70-80% 85-92% Setup time 15 minutes 20 minutes 15 minutes

Honest Limitations

1. Model format lock-in requires conversion step [MEDIUM RISK] BaseRT requires models to be converted to the proprietary .base format before inference. While base-convert supports safetensors, GGUF, and MLX checkpoints as input formats, the conversion step adds 8 to 15 minutes per model and doubles the storage required during the conversion process because the source and destination files coexist. If you maintain a library of 50 or more GGUF models, you must convert each one or maintain both formats side by side, increasing storage overhead. Mitigation: script the conversion pipeline with a Makefile or shell script that checks for the existence of a .base file and converts automatically if missing. Use temporary directories for conversion to avoid polluting your model storage. For teams, run conversions once and distribute the .base files via a shared cache or object store.

2. Concurrent request bottleneck under load [MEDIUM RISK] The v0.1.6 KV cache manager uses a single mutex for slot allocation in the pre-allocated memory pool, causing all concurrent requests to queue behind the first request's allocation. On our M4 Pro with 48 GB, four simultaneous requests added 2 to 3 seconds of queue time for each request beyond the first. This makes BaseRT unsuitable for high-concurrency production scenarios without additional infrastructure. Mitigation: deploy multiple BaseRT server processes behind an nginx or Caddy reverse proxy, each pinned to a dedicated model instance on a separate port. For high-throughput scenarios requiring 10+ concurrent requests, consider MLX which handles concurrent Python inference more gracefully despite slower single-request throughput, or wait for the v0.2.0 lock-free slot allocator.

3. macOS and chip version dependency [MINOR RISK] BaseRT requires macOS 14.0 (Sonoma) or later for the Metal Performance Shader APIs it uses. Users on macOS 13 Ventura or earlier cannot run BaseRT. Additionally, certain GPU features like hardware-accelerated ray tracing cores on M3 and M4 chips require specific Metal 3.1 features that fall back to software implementations on M1 and M2 chips. Mitigation: verify your macOS version with sw_vers and your chip architecture with sysctl -n machdep.cpu.brand_string before installing. BaseRT gracefully falls back to compatible Metal 3.0 kernels on M1 and M2 hardware with approximately 8-12% performance regression compared to M3 and M4 native execution.

System Prompt Template for AI Coding Agents

You are connected to a local LLM running on BaseRT, a native Metal inference runtime optimized for Apple Silicon.

Backend configuration:
- API endpoint: {api_base_url}/v1
- Model ID: {model_id}
- Max tokens: {max_tokens}
- Temperature: {temperature}
- Quantization: {quantization}

The BaseRT backend runs locally on Apple Silicon hardware and provides:
- Full streaming responses via SSE (Server-Sent Events)
- OpenAI-compatible chat completions endpoint
- Token usage reporting in response metadata
- Low-latency inference (up to 1.56x faster than llama.cpp)

Behavior guidelines:
- Use this backend for all code generation, explanation, and editing tasks
- The local inference is faster than cloud APIs, so prefer multi-turn refinement over single exhaustive prompts
- Each response includes token counts in the usage metadata for cost tracking
- The model supports up to 32,768 tokens of context

Request format:
POST {api_base_url}/v1/chat/completions
Content-Type: application/json
{
  "model": "{model_id}",
  "messages": [{"role": "user", "content": "..."}],
  "max_tokens": {max_tokens},
  "temperature": {temperature},
  "stream": true
}

Start in 10 Minutes

Step 1 (3 minutes). Run brew tap base-compute/tap && brew install basert base-convert in your terminal. Verify the installation with basert --version which should return v0.1.6. Ensure you have huggingface-cli installed with pip install huggingface-cli if needed.

Step 2 (4 minutes). Download a model with huggingface-cli download Qwen/Qwen3-30B-A3B --local-dir ./model and convert it with base-convert ./model --format safetensors --quantization q4_k_m --output ./model.base. The conversion progress bar will show the layer-by-layer progress.

Step 3 (3 minutes). Run basert serve ./model.base --port 8080 in one terminal window. In another window, send a test request with curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"default","messages":[{"role":"user","content":"Write a hello world in Rust"}],"stream":true}'. Confirm you receive a streaming SSE response with tokens appearing progressively.

Frequently Asked Questions

Q: Does BaseRT require a Rust toolchain to use?

A: No — the recommended installation is via Homebrew, which provides prebuilt binaries for all M-series Macs. The Rust toolchain is only needed if you are compiling from source to test the latest main branch or developing custom language bindings.

Q: Can I use existing GGUF models with BaseRT?

A: Yes — base-convert accepts GGUF files as input and converts them to the .base format. The conversion takes 5 to 15 minutes depending on model size and the target quantization level. The GGUF metadata including chat template and tokenizer configuration is preserved during conversion.

Q: Is BaseRT faster than Ollama on Apple Silicon?

A: Yes — Ollama uses llama.cpp under the hood with the Metal backend. BaseRT's direct Metal kernel compilation provides the same 1.35x to 1.56x speedup over Ollama as it does over llama.cpp, since they share the same underlying inference engine.

Q: Does BaseRT support Vision Language Models?

A: Not yet — BaseRT v0.1.6 supports text-only language models. The Base Compute team has confirmed multimodal support including image and audio inputs is planned for v0.3.0 with an estimated release in Q4 2026.

Q: Can I run BaseRT on Intel Macs or Linux?

A: No — BaseRT is exclusively for Apple Silicon Macs. It uses Metal Performance Shaders which are only available on macOS with Apple Silicon GPUs. For Linux or Intel Macs, use llama.cpp with CUDA or Vulkan backends, or MLX on Apple Silicon only.

Q: What is the maximum context length supported?

A: BaseRT supports up to 32,768 tokens of context in v0.1.6, limited by the KV cache manager's pre-allocated memory pool size. Extended context support up to 128K tokens using sliding window attention is on the roadmap for v0.4.0.

Related Reading

INTERNAL-LINK: /workflows/basert-vs-llamacpp-vs-mlx-apple-silicon-benchmarks-2026 — A guided workflow for deploying BaseRT with Claude Code for accelerated code generation on Apple Silicon.

INTERNAL-LINK: /blogs/mesh-llm-distributed-inference-2026 — Distributed LLM inference across Apple Silicon devices using collaborative mesh networking for larger model serving.

INTERNAL-LINK: /blogs/qwen36-unsloth-nvfp4-faster-inference-guide-2026 — Optimizing Qwen3.6 series models with NVFP4 quantization for faster local inference on Apple Silicon and NVIDIA hardware.

INTERNAL-LINK: /blogs/auriko-llm-cost-arbitrage-guide-2026 — Cost optimization strategies for hybrid cloud-local LLM inference deployments that combine BaseRT with cloud GPU endpoints.

INTERNAL-LINK: /blogs/opensquilla-vs-frugon-vs-otari-2026 — Token-efficient router models that optimize when to route requests to local BaseRT inference versus cloud API providers.

INTERNAL-LINK: /blogs/prismml-bonsai-27b-on-device-pipeline-2026 — On-device inference optimization for Apple Silicon using the PrismML Bonsai 27B model with BaseRT runtime.

For more benchmarks, see the BaseRT arXiv paper at https://arxiv.org/abs/2607.00501 {rel="nofollow"} and the official GitHub repository at https://github.com/base-compute/basert {rel="nofollow"}. The benchmark reproduction guide is documented at https://basert.dev/benchmarks {rel="nofollow"} with hardware configuration details and reproduction scripts. The Product Hunt launch page is at https://www.producthunt.com/products/basert {rel="nofollow"}. Refer to the MLX documentation at https://mlx-swift.org {rel="nofollow"} and the llama.cpp Metal backend guide at https://github.com/ggerganov/llama.cpp/tree/master/metal {rel="nofollow"} for comparison baselines.