Agnostic

Quantized weights, built for the metal.

Agnostic publishes MLX-format, affine-quantized builds of recent language and multimodal models, shaped for fast local inference on Apple Silicon and packed for lisa — a pure-Rust Metal inference engine with no Python, no MLX runtime, and no tensor framework at load time. Built from scratch on the Metal C API, with an OpenAI-compatible server.

Every checkpoint here is a faithful re-pack of the public original, not a fine-tune: same weights, same tokenizer, same chat template, reduced precision. Quantization is deliberately boring — one bf16 scale/bias pair per group, packed u32 payloads, group 32/64 as required by the kernels that consume them.

Models

Model	Base	Bits	Group	Notes
Qwen3.8-Flash-Next-4bit	Qwen/Qwen3.8-Flash-Next	4	32	`qwen4_exp` text tower + MTP head + merged n-gram table
Qwen3.8-27B-4bit	Qwen/Qwen3.8-27B	4	64	vision tower included, MTP head bundled
Qwen3.8-27B-2bit	Qwen/Qwen3.8-27B	2	64	text-only, MTP head bundled
MiMo-V2.6-Distill-Qwen-9B-4bit	XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B	4	64	vision tower included, image-text-to-text

What's in the box

Affine quantization (mx.quantize): packed u32 weights plus one bf16 scale and bias per group along the input dimension.
Group size is fixed by the checkpoint and must be honored by the loader. Group 32 is required by lisa's fused Flash-Next decode kernels; other builds use group 64 unless noted.
Only linear layers are quantized. Embeddings, norms, convolutions and the vision patch embedder keep original precision.
MTP heads ride along where the base model has one, for lossless speculative decoding: mtp/ holds the multi-token-prediction sidecar.
Transformers are reworked to the engine's expected layout at publish time (prefix renames, MoE switch_mlp packing, conv1d transposes, RMSNorm fold) — loading is the contract test, not a separate transform pass.

Usage

With lisa, --model takes a repo id directly:

lisa serve --model agnosticeng/Qwen3.8-Flash-Next-4bit --addr 127.0.0.1:8000

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Hello"}],"max_tokens":64}'

With MLX / mlx-vlm these are ordinary quantized checkpoints:

from mlx_vlm import load, generate
model, processor = load("agnosticeng/MiMo-V2.6-Distill-Qwen-9B-4bit")

Greedy decoding is bit-exact against the reference engine's goldens (256/256, serial and MTP depths 1–6 on the public long-copy gate).

Conversion

Built from the public originals by streaming the checkpoint shard by shard (download → quantize → delete), so no host ever holds the full-precision model twice. Scripts and the exact transforms are documented on each model card.

Licensing