docs: clarify compression performance tradeoffs

README.md

@@ -55,7 +55,7 @@
 
 [2023/12/02] added support for safetensors. Now support all top 10 models in open llm leaderboard.
 
-[2023/12/01] airllm 2.0. Support compressions: **3x run time speed up!**
+[2023/12/01] airllm 2.0. Added 4-bit and 8-bit model compression.
 
 [2023/11/20] airllm Initial version!
 
@@ -71,7 +71,7 @@
 ## Table of Contents
 
 * [Quick start](#quickstart)
-* [Model Compression](#model-compression---3x-inference-speed-up)
+* [Model Compression](#model-compression)
 * [Configurations](#configurations)
 * [Run on MacOS](#macos)
 * [Example notebooks](#example-python-notebook)
@@ -137,13 +137,11 @@ print(output)
 Note: During inference, the original model will first be decomposed and saved layer-wise. Please ensure there is sufficient disk space in the huggingface cache directory.
 
 
-## Model Compression - 3x Inference Speed Up!
+## Model Compression
 
-We just added model compression based on block-wise quantization-based model compression. Which can further **speed up the inference speed** for up to **3x** , with **almost ignorable accuracy loss!** (see more performance evaluation and why we use block-wise quantization in [this paper](https://arxiv.org/abs/2212.09720))
+Model compression uses block-wise quantization to reduce the size of the layer shards written to disk. It can make a model easier to store and move, but it does **not** guarantee faster inference: compression currently disables prefetching and adds GPU dequantization work for each layer, so it tends to help when disk I/O is the bottleneck and can hurt when it is not. The result also depends on the storage device, accelerator, model, and workload. Benchmark both modes on your target hardware before enabling compression for performance reasons. For background on the quantization method, see [this paper](https://arxiv.org/abs/2212.09720).
 
-![speed_improvement](https://github.com/lyogavin/airllm/blob/main/assets/airllm2_time_improvement.png?v=2&raw=true)
-
-#### How to enable model compression speed up:
+#### How to enable model compression:
 
 * Step 1. make sure you have [bitsandbytes](https://github.com/TimDettmers/bitsandbytes) installed by `pip install -U bitsandbytes `
 * Step 2. make sure airllm verion later than 2.0.0: `pip install -U airllm` 
@@ -159,7 +157,7 @@ model = AutoModel.from_pretrained("garage-bAInd/Platypus2-70B-instruct",
 
 Quantization normally needs to quantize both weights and activations to really speed things up. Which makes it harder to maintain accuracy and avoid the impact of outliers in all kinds of inputs.
 
-While in our case the bottleneck is mainly at the disk loading, we only need to make the model loading size smaller. So, we get to only quantize the weights' part, which is easier to ensure the accuracy.
+While in some environments the bottleneck may be disk loading, the compression path only quantizes weights and still dequantizes them during layer loading. This keeps the stored representation smaller, but it also adds work during loading; accuracy and performance should be measured for the model and hardware you plan to use.
 
 ## Configurations
 

air_llm/airllm/utils.py

@@ -495,7 +495,10 @@ def find_or_create_local_splitted_path(model_local_path_or_repo_id, layer_shards
     saved_layer_shards_path : str
         the path saved layer shards
     compression: str, optinal
-        setting to '4bit' or '8bit' to enable compression from 16 bits to 4 bits/8 bits which speeed up 4x or 2x inference time with a tiny accuracy loss.
+        setting to '4bit' or '8bit' to store weights in a smaller representation.
+        Compression can reduce checkpoint storage, but it does not guarantee faster
+        inference because layers are dequantized while they are loaded. Benchmark
+        the target model and hardware before relying on a performance improvement.
     hf_token: str, optional
         huggingface api token could be provided, by default None
     """