Models are programs
Readable XLang3 .py source describes symbolic tensor computation that Garnet can inspect and compile.
Developed by CantorAI and the XLang Foundation
Models are programs.
Compile them for the hardware you ship.
Garnet is an open-source, profile-based AI inference runtime and compiler. It captures XLang3 Tensor Expressions from backend-neutral xModel programs and specializes execution for your model, hardware, precision and workload.
Official public source repository: xlang-foundation/Garnet.
LLMVLMASRTTS
GARNET COMPILER / RUNTIMEENGINE READY
XMODEL IR
PAGED KV
NATIVE I/O
BackendTensorRT
PrecisionBF16 / INT4
ExecutionNative
Backend-neutral
xModel semantics stay separate from TensorRT, OpenVINO, native CPU, and custom hardware policy.
Hardware-specialized
Garnet lowers graphs, operators, precision, memory behavior, and kernels for the machine you deploy.
Models & measured performance
Find your model.
See how it runs.
Optimized research profiles first. Compare Garnet with tested vLLM modes on the same hardware; explore the other model programs below.
Optimized profiles
COMPLETED RESEARCH RUNS · OCT 26, 2026| Model | Garnet profile | Hardware / workload | Garnet | vLLM | Advantage | Details |
|---|---|---|---|---|---|---|
| Qwen3.5 2BVLMBEST TESTED LATENCY RATIO* | Native CUDA + MTPBF16 large weights | RTX 4080Batch 2 · 32 output tokens / sequence | 191.09 msRequest latency ↓ | 280.39 msMTP3 · request latency | 1.47×31.8% lower latency | Method ↗ |
| Qwen3.5 4BVLM | Native CUDA + MTPBF16 large weights | RTX 4080Batch 2 · 32 output tokens / sequence | 332.45 msRequest latency ↓ | 402.07 msMTP3 · request latency | 1.21×17.3% lower latency | Method ↗ |
| Qwen3.8 27BVLM · PILOT | Joined MLP + MTP3Mixed NVFP4 / FP8 · BF16 vision | RTX 5090 · 32 GBBatch 1 · 128 output tokens | 74.16 tok/sImage decode ↑64.72 msText first token ↓ | 68.50 tok/sOrdinary · image decode152.36 msOrdinary · text first token | +8.3%Image decode2.35×Text first-token ratio | Method ↗ |
27B text first-token pilot: Garnet measured 64.72 ms versus 152.36 ms for vLLM and 153.89 ms for SGLang: 2.35× and 2.38× faster time to first token, respectively. RTX 5090, batch 1, mixed NVFP4/FP8, 8 measured requests per cohort, 128 output tokens. Ordinary reference modes; broader tuning and quality qualification remain open. This is a text first-token result, not an overall inference or throughput speedup. Source measurements.
*Among the seven qualified Qwen3.5 request-latency profiles. Historical measurements; independent quality and current-build qualification remain open. 27B uses short-request pilots against ordinary vLLM. Different hardware and metrics are not a single speed ranking. Full conditions ↓
More model programs
LLM · VLM · ASR · TTS| Model | Modality | Execution / precision | Hardware | Status | Explore |
|---|---|---|---|---|---|
| GPT-OSS 120B | LLM | Optimization in progress | 2 × RTX 6000 PROTarget configuration | IN PROGRESS | Source ↗ |
| Qwen3 1.7B | LLM | TensorRT BF16 · OpenVINO INT4 | CPU / NVIDIA GPU31.87 decode tok/s · i9-14900K · INT4 | SOURCE INCLUDED | Source ↗ |
| Qwen3-VL 2B | VLM | TensorRT BF16 · JPEG input | NVIDIA GPU | SOURCE INCLUDED | Source ↗ |
| Qwen3-ASR 0.6B | ASR | TensorRT BF16 · native WAV input | NVIDIA GPU | SOURCE INCLUDED | Source ↗ |
| Qwen3-TTS 0.6B | TTS | TensorRT BF16 · 24 kHz output | NVIDIA GPU | SOURCE INCLUDED | Source ↗ |
| Qwen3-TTS 1.7B | TTS | TensorRT BF16 · instruction control | NVIDIA GPU | SOURCE INCLUDED | Source ↗ |
Garnet Runtime
Model code is not
a deployment plan.
Garnet executes Python-compatible xModel source in XLang3, captures an inspectable Tensor Expression graph, and lowers it into execution specialized for the selected backend and processor architecture.
- ✓ Native C++ execution—no Python process at inference
- ✓ TensorRT, OpenVINO, native CPU, and custom-kernel execution
- ✓ Paged KV memory, fixed decode buckets, and engine caching
- ✓ Validated model manifests and application-facing APIs
COMPILED MODEL PIPELINE
01
Selected modelSafetensors + configuration
Qwen302
xModel programXLang3 Tensor Expressions
.py03
Target profileBackend · precision · shape
JSONNVIDIATensorRTGPU engines
INTELOpenVINOCPU / GPU
Build Garnet from source
Inspect the graph.
Compile for your hardware.
Clone Garnet beside XLang3, build the native package, validate graph capture with a weight-free xModel, and then bring the checkpoint and vendor runtime for your target backend.
DEVELOPER FLOW4 STEPS
- 01Clone the source
Place Garnet and XLang3 in sibling directories.
- 02Build the runtime
Configure the C++ toolchain and the backend SDKs required for your target.
- 03Capture an xModel
Execute Python-compatible model source with symbolic tensors and inspect the graph.
- 04Compile and run
Lower the graph for TensorRT, OpenVINO, native CPU, or custom hardware execution.
Behind the results
How we measure.
The comparison table above contains the headline results. These are the workloads, evidence, and qualifications behind them.
Workloads, evidence, and limits
Qwen3.5 2B/4B: historical native build 14c, RTX 4080, sequential matched cohorts with 32 output tokens per sequence. Batch-2 results use 14 common groups for 2B and 19 for 4B. BF16 large weights are matched; preprocessing, API boundaries, small-parameter precision, and intermediate arithmetic differ. Garnet timings use bracketing runs. Each external engine is represented by its fastest tested ordinary or MTP3 mode.
27B: one pilot per mode, eight measured requests per cohort, 41 text or 104 image prompt tokens, and 128 output tokens. Capacity is 4,096 tokens; actual 4K prompts were not tested in this pilot. References are ordinary modes; tuned and MTP references, repetitions, long context, and concurrency remain pending.
Research status: independent quality, exact VLM parity, and current-build qualification remain incomplete. These historical measurements do not establish production readiness or precision-equivalent output quality. No error bars or overall 2× decode/throughput claim.
Download measurement data and provenance (JSON) · Request benchmark details
Internal engineering measurements from completed runs. Results vary with model, precision, prompt, hardware, software, and cache state. Snapshot: October 26, 2026.
Custom model engineering
Your model.
Your hardware.
A measured path to production.
Have a model or device target outside the catalog? CantorAI can evaluate the graph, memory budget, precision strategy, native I/O path, and application packaging.
Start an optimization review →- 01Profile
Model, workload, hardware, latency, memory, and quality targets.
- 02Engineer
Graph lowering, quantization, kernels, scheduling, and native I/O.
- 03Validate
Correctness, quality, throughput, latency, startup, and recovery.
- 04Package
A versioned model package ready for your application.
Ready to program closer to the hardware?