Models and Engines
How to install an LLM inference engine and host an LLM model
Find models and recipes
- Hugging face
- Modelscope
- Unsloth
- Club 3090 : recipes for 3090/4090/5090 owners.
- MiaAI-Lab : various recipes for popular models.
Deepseek v4.1 Flash
- sybil’s recipe : DeepSeek-V4.1-Flash EXL3 on one 24 GB RTX 3090 + DDR4 + NVMe.
- jayleaton’s recipe : DeepSeek-V4.1-Flash on 2x DGX Spark with TensorFold.
- MiaAI’s recipe : DeepSeek-V4.1-Flash-1017 EXL3 on 2x DGX Sparks with TensorFold.
Gemma4
Speech recognition and text-to-speech (TTS)
Document processing
Biology
Image generation
Inference engines
- Hyperflare : a local inference server for NVIDIA SM120/SM121 GPUs.
- Lucebox : LLM speculative inference server for heterogeneous hardware & consumer GPUs (esp. AMD GPUs).
- Magnitude : an open source inference engine that tunes its kernels on your hardware.
- SGLang a high-performance serving framework. See also SGLang Cookbook
- TensorFold : LLM Inference Engine for Metal and CUDA.
- Tensorsharp : a native .NET LLM inference engine and agent runtime for GGUF models.
- TabbyAPI the official API server for exllamav3 backend, an inference library for the EXL3 quantization format (smaller bit per weight).
- FreeToken : Local inference engine with Bandwidth-Adaptive Execution.
- Project Maya : supports GLM-5.3-Flash (321B MoE), DeepSeek-V4.1-Flash-1017 (748B MoE), and MiniMax-M3.1-Flash (428B MoE).
llama.cpp
llama.cpp : LLM inference in C/C++ supporting GGUF format.
- beellama.cpp :
llama.cppfork supporting the KVarN KV cache format. - ik_llama.cpp :
llama.cppfork with new quants and improved performance for MoE models. - KoboldCpp :
llama.cppfork with text, image, speech, music, and video generation.
- llama-optimize : Find good llama.cpp command-line parameters for a given GGUF model on your machine, automatically, using statistically-designed experiments.
vLLM
vllm : a high-throughput and memory-efficient inference and serving engine for LLMs.
- DeepInfer : High-performance, large-scale inference framework for DeepSeek models.
- Intel vllm a vLLM docker image for Intel GPUs.
- Paiton : optimized for AMD Radeon AI PRO R9700.
- spark-vllm-docker : a vLLM docker image for DGX sparks.
- vllm-radiance : a vLLM inference-server image for the AMD Radeon AI PRO R9700.
Last updated on