Skip to content
Models and Engines

Models and Engines

How to install an LLM inference engine and host an LLM model

Find models and recipes


Deepseek v4.1 Flash

Gemma4

Speech recognition and text-to-speech (TTS)

Document processing

Biology

Image generation

Inference engines

  • Hyperflare : a local inference server for NVIDIA SM120/SM121 GPUs.
  • Lucebox : LLM speculative inference server for heterogeneous hardware & consumer GPUs (esp. AMD GPUs).
  • Magnitude : an open source inference engine that tunes its kernels on your hardware.
  • SGLang a high-performance serving framework. See also SGLang Cookbook
  • TensorFold : LLM Inference Engine for Metal and CUDA.
  • Tensorsharp : a native .NET LLM inference engine and agent runtime for GGUF models.
  • TabbyAPI the official API server for exllamav3 backend, an inference library for the EXL3 quantization format (smaller bit per weight).
  • FreeToken : Local inference engine with Bandwidth-Adaptive Execution.
  • Project Maya : supports GLM-5.3-Flash (321B MoE), DeepSeek-V4.1-Flash-1017 (748B MoE), and MiniMax-M3.1-Flash (428B MoE).

llama.cpp

llama.cpp : LLM inference in C/C++ supporting GGUF format.

  • beellama.cpp : llama.cpp fork supporting the KVarN KV cache format.
  • ik_llama.cpp : llama.cpp fork with new quants and improved performance for MoE models.
  • KoboldCpp : llama.cpp fork with text, image, speech, music, and video generation.

  • llama-optimize : Find good llama.cpp command-line parameters for a given GGUF model on your machine, automatically, using statistically-designed experiments.

vLLM

vllm : a high-throughput and memory-efficient inference and serving engine for LLMs.

  • DeepInfer : High-performance, large-scale inference framework for DeepSeek models.
  • Intel vllm a vLLM docker image for Intel GPUs.
  • Paiton : optimized for AMD Radeon AI PRO R9700.
  • spark-vllm-docker : a vLLM docker image for DGX sparks.
  • vllm-radiance : a vLLM inference-server image for the AMD Radeon AI PRO R9700.
Last updated on