From ee3a1f8c60a56dbec193d5d45bcc2c3c963bb8c3 Mon Sep 17 00:00:00 2001 From: "Aryan Singh K." <70511529+aryansk@users.noreply.github.com> Date: Thu, 6 Aug 2026 11:38:47 +0530 Subject: [PATCH 1/2] Clarify compression performance tradeoffs --- README.md | 14 ++++++-------- air_llm/airllm/utils.py | 5 ++++- 2 files changed, 10 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 160cffd5..b1baf0cd 100644 --- a/README.md +++ b/README.md @@ -55,7 +55,7 @@ [2023/12/02] added support for safetensors. Now support all top 10 models in open llm leaderboard. -[2023/12/01] airllm 2.0. Support compressions: **3x run time speed up!** +[2023/12/01] airllm 2.0. Added 4-bit and 8-bit model compression. [2023/11/20] airllm Initial version! @@ -71,7 +71,7 @@ ## Table of Contents * [Quick start](#quickstart) -* [Model Compression](#model-compression---3x-inference-speed-up) +* [Model Compression](#model-compression) * [Configurations](#configurations) * [Run on MacOS](#macos) * [Example notebooks](#example-python-notebook) @@ -137,13 +137,11 @@ print(output) Note: During inference, the original model will first be decomposed and saved layer-wise. Please ensure there is sufficient disk space in the huggingface cache directory. -## Model Compression - 3x Inference Speed Up! +## Model Compression -We just added model compression based on block-wise quantization-based model compression. Which can further **speed up the inference speed** for up to **3x** , with **almost ignorable accuracy loss!** (see more performance evaluation and why we use block-wise quantization in [this paper](https://arxiv.org/abs/2212.09720)) +Model compression uses block-wise quantization to reduce the size of the layer shards written to disk. It can make a model easier to store and move, but it does **not** guarantee faster inference: each layer is dequantized as it is loaded, and the result depends on the storage device, accelerator, model, and workload. Benchmark both modes on your target hardware before enabling compression for performance reasons. For background on the quantization method, see [this paper](https://arxiv.org/abs/2212.09720). -![speed_improvement](https://github.com/lyogavin/airllm/blob/main/assets/airllm2_time_improvement.png?v=2&raw=true) - -#### How to enable model compression speed up: +#### How to enable model compression: * Step 1. make sure you have [bitsandbytes](https://github.com/TimDettmers/bitsandbytes) installed by `pip install -U bitsandbytes ` * Step 2. make sure airllm verion later than 2.0.0: `pip install -U airllm` @@ -159,7 +157,7 @@ model = AutoModel.from_pretrained("garage-bAInd/Platypus2-70B-instruct", Quantization normally needs to quantize both weights and activations to really speed things up. Which makes it harder to maintain accuracy and avoid the impact of outliers in all kinds of inputs. -While in our case the bottleneck is mainly at the disk loading, we only need to make the model loading size smaller. So, we get to only quantize the weights' part, which is easier to ensure the accuracy. +While in some environments the bottleneck may be disk loading, the compression path only quantizes weights and still dequantizes them during layer loading. This keeps the stored representation smaller, but it also adds work during loading; accuracy and performance should be measured for the model and hardware you plan to use. ## Configurations diff --git a/air_llm/airllm/utils.py b/air_llm/airllm/utils.py index ad45f995..1519c577 100644 --- a/air_llm/airllm/utils.py +++ b/air_llm/airllm/utils.py @@ -495,7 +495,10 @@ def find_or_create_local_splitted_path(model_local_path_or_repo_id, layer_shards saved_layer_shards_path : str the path saved layer shards compression: str, optinal - setting to '4bit' or '8bit' to enable compression from 16 bits to 4 bits/8 bits which speeed up 4x or 2x inference time with a tiny accuracy loss. + setting to '4bit' or '8bit' to store weights in a smaller representation. + Compression can reduce checkpoint storage, but it does not guarantee faster + inference because layers are dequantized while they are loaded. Benchmark + the target model and hardware before relying on a performance improvement. hf_token: str, optional huggingface api token could be provided, by default None """ From 4883a6f00cf84d901116e958602f6d8c2d06ed23 Mon Sep 17 00:00:00 2001 From: "Aryan Singh K." <70511529+aryansk@users.noreply.github.com> Date: Sun, 9 Aug 2026 02:13:21 +0530 Subject: [PATCH 2/2] Document compression I/O and GPU tradeoffs --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index b1baf0cd..d1ec469a 100644 --- a/README.md +++ b/README.md @@ -139,7 +139,7 @@ Note: During inference, the original model will first be decomposed and saved la ## Model Compression -Model compression uses block-wise quantization to reduce the size of the layer shards written to disk. It can make a model easier to store and move, but it does **not** guarantee faster inference: each layer is dequantized as it is loaded, and the result depends on the storage device, accelerator, model, and workload. Benchmark both modes on your target hardware before enabling compression for performance reasons. For background on the quantization method, see [this paper](https://arxiv.org/abs/2212.09720). +Model compression uses block-wise quantization to reduce the size of the layer shards written to disk. It can make a model easier to store and move, but it does **not** guarantee faster inference: compression currently disables prefetching and adds GPU dequantization work for each layer, so it tends to help when disk I/O is the bottleneck and can hurt when it is not. The result also depends on the storage device, accelerator, model, and workload. Benchmark both modes on your target hardware before enabling compression for performance reasons. For background on the quantization method, see [this paper](https://arxiv.org/abs/2212.09720). #### How to enable model compression: