AirLLM Enables 70B Language Models on 4Gb GPUs

Lyogavin released AirLLM, a system that allows large language models, including those with 70 billion parameters, to run on single 4GB GPUs. The software achieves this by dramatically reducing memory usage without relying on techniques like quantization, distillation, or pruning.

Users can currently run models like Qwen3-235B on 8GB of VRAM and DeepSeek-V3 (671B) on approximately 12GB. AirLLM works by keeping only one model layer on the GPU at a time, scaling VRAM requirements to layer size rather than total model size.

The project supports a wide range of models including Llama, Qwen, DeepSeek, Mistral, and others, simplifying deployment. Recent updates have added support for FP8 models and CPU inference, alongside performance improvements through model compression.

The developers note that sufficient disk space is crucial during inference, as the model is initially decomposed and saved layer-wise. The project is open source and welcomes contributions.