Skip to content
Codex
LLMs
LLM models

LLM models

How to find and host an LLM model on your computer

General recipes

Inference engines

  • exllamav3 : ExLlamaV3 is an inference library for the EXL3 quantization format (smaller bit per weight).
  • Project Maya : run GLM-5.3-Flash (321B MoE), DeepSeek-V4.1-Flash-1017 (748B MoE) and MiniMax-M3.1-Flash (428B MoE) privately on your own GPU(s)
  • FreeToken : Optimized for MoE models.
  • Strata : run Qwen3.8-Flash-Next and DeepSeekV4.1-Flash-1017 from 12GB+ VRAM with smart expert caching.
  • SGLang GitHub repo. See also SGLang Cookbook
  • TensorFold : LLM Inference Engine for Metal and CUDA.

llama.cpp

  • llama.cpp GitHub repo.
  • beellama.cpp : llama.cpp fork supporting KVarN KV cache format.
  • ik_llama.cpp : llama.cpp fork with new quants and improved performance for MoE models.

vLLM

Qwen3.8-27B

Running the Qwen3.8-27B model

  • ISTA: GSQ-RCO : small quants of ISTA lab.
  • MiaAI-Lab for one 16-24 GB GPU using the EXL3 quantization format.
  • ninfer-4090 : Qwen3.8-27B on one RTX 4090. Based on ninfer, a C++/CUDA inference engine for explicitly registered Qwen checkpoints on a single NVIDIA GeForce RTX 5090.
  • Swift-1.5-Qwen3.8-27b : UkisAI’s reasoning-efficient derivative of Qwen3.8-27B.
  • Unsloth

Qwen3.8-Flash-Next

Running the Qwen3.8-Flash-Next model

  • Unsloth
  • MiaAI-Lab on one DGX Spark using TensorFold.
  • Strata : run Qwen3.8-Flash-Next from 12GB+ VRAM with smart expert caching.

Gemma4

Use them with llama.cpp.

Speech recognition and text-to-speech (TTS)

Document processing

Biology

Last updated on