Re-quantize the Official FP8 Model Using INC (vllm-project#959)

Re-quantize the Official FP8 Model Using INC

- Docs:
https://github.com/yiliu30/vllm-fork/blob/r1-woq-requant/scripts/DEEPSEEK_R1_ON_GAUDI.md#requantize-the-official-fp8-model-using-inc
gsm8k result:
``` bash
Running generate_until requests: 100%|█████████████| 64/64 [06:10<00:00,  5.78s/it]
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9688|±  |0.0219|
|     |       |strict-match    |     5|exact_match|↑  |0.9531|±  |0.0266|
```

cc @thuang6 @yangulei

---------

Signed-off-by: Yi Liu <[email protected]>
Signed-off-by: Yi <[email protected]>
Signed-off-by: yiliu30 <[email protected]>
Co-authored-by: Yi Liu <[email protected]>
Co-authored-by: KepingYan <[email protected]>
Co-authored-by: kepingyan <[email protected]>
Co-authored-by: Marceli Fylcek <[email protected]>
Co-authored-by: Youlei Yang <[email protected]>
Co-authored-by: Wei Lin <[email protected]>
Co-authored-by: Tony Lin <[email protected]>
Co-authored-by: Bob Zhu <[email protected]>
Co-authored-by: Bob Zhu <[email protected]>

Author: 10 people

requirements-hpu.txt

@@ -9,3 +9,5 @@ tabulate
 setuptools>=61
 setuptools-scm>=8
 vllm-hpu-extension @ git+https://github.com/HabanaAI/vllm-hpu-extension.git@ecb60e4
+# FIXME: (Yi) Replace it with the INC 3.4
+git+https://github.com/intel/neural-compressor.git@r1-woq

scripts/DEEPSEEK_R1_ON_GAUDI.md

@@ -93,3 +93,29 @@ ray start --address='${head_ip}:6379' --resources='{"HPU": 8, "TPU": 0}'
 python scripts/run_example_tp_2nodes.py --model ${YOUR_PATH}/DeepSeek-R1-static
 ```
 
+# Requantize the Official FP8 Model Using INC
+- INC: https://github.com/intel/neural-compressor/tree/r1-woq
+
+- Calibration
+> [!Note]
+> This step will take a while. You can skip it by downloading the pre-calibration result.
+> 
+> `huggingface-cli download Yi30/inc-woq-full-pile-512-1024-331  --local-dir ./scripts/nc_workspace_measure_kvache`
+
+```bash
+export OFFICIAL_FP8_MODEL=deepseek-ai/DeepSeek-R1
+cd ./scripts
+VLLM_REQUANT_FP8_INC=1 QUANT_CONFIG=inc_measure_with_fp8kv_config.json VLLM_ENABLE_RUNTIME_DEQUANT=1 python run_example_tp.py --model ${OFFICIAL_FP8_MODEL} --tokenizer ${OFFICIAL_FP8_MODEL} --osl 32 --max_num_seqs 1 --nprompts 512 --dataset pile
+```
+
+- Quantization
+```bash
+cd ./scripts
+VLLM_REQUANT_FP8_INC=1 QUANT_CONFIG=inc_quant_with_fp8kv_config.json VLLM_ENABLE_RUNTIME_DEQUANT=1 python run_example_tp.py --model ${OFFICIAL_FP8_MODEL} --tokenizer ${OFFICIAL_FP8_MODEL} --osl 32 --max_num_seqs 1 --fp8_kv_cache 
+```
+
+- Evaluation
+```bash
+cd ./scripts
+VLLM_REQUANT_FP8_INC=1 QUANT_CONFIG=inc_quant_with_fp8kv_config.json VLLM_ENABLE_RUNTIME_DEQUANT=1 python run_lm_eval.py  --model ${OFFICIAL_FP8_MODEL} --tokenizer ${OFFICIAL_FP8_MODEL} --fp8_kv_cache -l 64 --batch_size 1 
+```

scripts/benchmark-inc-staticfp8-i1k-o1k-ep8-bestperf.sh

@@ -0,0 +1,96 @@
+# FIXME: (Yi) remove it before merge
+#!/bin/bash
+tp_parrallel=8
+in_len=1024
+out_len=1024
+multi_step=1
+total_len=$((in_len + out_len))
+# if total_len is not multiple of 128, round up to the next multiple of 128
+if [ $((total_len % 128)) -ne 0 ]; then
+    echo 'round up for 128'
+    total_len=$(((total_len / 128 +  1) * 128 ))
+fi
+ep_size=8
+moe_n_slice=1
+gpu_utils=0.92
+bs=448
+num_prompts=448
+request_rate=inf
+log_name="[inc-331-moe-op-maxabs_hw-scalar-online-gaudi3-${gpu_utils}util-TPparallel${tp_parrallel}-EP${ep_size}-loop${moe_n_slice}moegroups-multistep${multi_step}_nprompt${num_prompts}_rrate${request_rate}_bs${bs}_i${in_len}_o${out_len}_mdllen${total_len}"
+
+VLLM_DECODE_BLOCK_BUCKET_MIN=$((in_len * bs / 128))
+VLLM_DECODE_BLOCK_BUCKET_MAX=$((total_len * bs / 128 + 128))
+# model="/data/models/DeepSeek-R1-static/"
+# tokenizer="/data/models/DeepSeek-R1-static/"
+model="/data/models/DeepSeek-R1/"
+tokenizer="/data/models/DeepSeek-R1/"
+model_name="DeepSeek-R1"
+
+
+QUANT_CONFIG="inc_quant_with_fp8kv_config.json" \
+VLLM_REQUANT_FP8_INC=1 \
+VLLM_ENABLE_RUNTIME_DEQUANT=1 \
+VLLM_DELAYED_SAMPLING=true \
+HABANA_VISIBLE_DEVICES="ALL" \
+VLLM_MOE_N_SLICE=${moe_n_slice} \
+VLLM_EP_SIZE=${ep_size} \
+VLLM_MLA_DISABLE_REQUANTIZATION=1 \
+PT_HPU_ENABLE_LAZY_COLLECTIVES=true \
+PT_HPU_WEIGHT_SHARING=0 \
+VLLM_PROMPT_BS_BUCKET_MIN=1 \
+VLLM_PROMPT_BS_BUCKET_MAX=16 \
+VLLM_PROMPT_SEQ_BUCKET_MIN=${in_len} \
+VLLM_PROMPT_SEQ_BUCKET_MAX=${in_len} \
+VLLM_DECODE_BS_BUCKET_MIN=${bs} \
+VLLM_DECODE_BS_BUCKET_MAX=${bs} \
+VLLM_DECODE_BLOCK_BUCKET_MIN=${VLLM_DECODE_BLOCK_BUCKET_MIN} \
+VLLM_DECODE_BLOCK_BUCKET_MAX=${VLLM_DECODE_BLOCK_BUCKET_MAX} \
+python -m vllm.entrypoints.openai.api_server \
+    --port 8080 \
+    --model ${model} \
+    --tensor-parallel-size ${tp_parrallel} \
+    --max-num-seqs ${bs} \
+    --disable-log-requests \
+    --dtype bfloat16 \
+    --use-v2-block-manager \
+    --num_scheduler_steps ${multi_step}\
+    --max-model-len 4096 \
+    --distributed_executor_backend mp \
+    --gpu_memory_utilization ${gpu_utils} \
+    --kv_cache_dtype "fp8_inc" \
+    --trust_remote_code 2>&1 | tee benchmark_logs/${log_name}_serving.log &
+pid=$(($!-1))
+
+until [[ "$n" -ge 1000 ]] || [[ $ready == true ]]; do
+    n=$((n+1))
+    if grep -q "Started server process" benchmark_logs/${log_name}_serving.log; then
+        break
+    fi
+    sleep 5s
+done
+sleep 10s
+echo ${pid}
+
+hl-smi -l > tee benchmark_logs/${log_name}_smi.log &
+hl_pid=$(($!-1))
+
+
+start_time=$(date +%s)
+echo "Start to benchmark"
+python ../benchmarks/benchmark_serving.py --backend vllm --model ${model} --tokenizer ${tokenizer} --dataset-name sonnet --dataset-path ../benchmarks/sonnet.txt --request-rate ${request_rate} --num-prompts ${num_prompts} --port 8080 --sonnet-input-len ${in_len} --sonnet-output-len ${out_len} --sonnet-prefix-len 100 2>&1 | tee benchmark_logs/${log_name}_run1.log
+end_time=$(date +%s)
+echo "Time elapsed: $((end_time - start_time))s"
+
+sleep 10
+
+# start_time=$(date +%s)
+# echo "Start to benchmark"
+# python benchmarks/benchmark_serving.py --backend vllm --model ${model} --tokenizer ${tokenizer} --dataset-name sonnet --dataset-path benchmarks/sonnet.txt --request-rate ${request_rate} --num-prompts ${num_prompts} --port 8080 --sonnet-input-len ${in_len} --sonnet-output-len ${out_len} --sonnet-prefix-len 100 2>&1 | tee benchmark_logs/${log_name}_run2.log
+# end_time=$(date +%s)
+# echo "Time elapsed: $((end_time - start_time))s"
+
+# sleep 10
+
+kill ${pid}
+kill ${hl_pid}
+#--backend openai-chat --endpoint "v1/chat/completions"

scripts/inc_measure_with_fp8kv_config.json

@@ -0,0 +1,15 @@
+{
+    "method": "HOOKS",
+    "mode": "MEASURE",
+    "observer": "maxabs",
+    "whitelist": {
+        "types": [],
+        "names": []
+    },
+    "blocklist": {
+        "types": [],
+        "names": ["lm_head", "mlp\\.gate\\b"]
+    },
+    "quantize_weight": false,
+    "dump_stats_path": "./nc_workspace_measure_kvache/inc_measure_output"
+}

scripts/inc_quant_per_channel_with_fp8kv_config.json

@@ -0,0 +1,22 @@
+{
+    "mode": "QUANTIZE",
+    "observer": "maxabs",
+    "scale_method": "ACT_MAXABS_HW_WEIGHTS_PCS_MAXABS_POW2",
+    "scale_format": "const",
+    "allowlist": {
+        "types": [],
+        "names": []
+    },
+    "blocklist": {
+        "types": [],
+        "names": [
+            "lm_head",
+            "mlp\\.gate\\b",
+            "matmul_qk",
+            "matmul_av",
+            "batch2block_matmul",
+            "block2batch_matmul"
+        ]
+    },
+    "dump_stats_path": "./nc_workspace_measure_kvache/inc_measure_output"
+}

scripts/inc_quant_with_fp8kv_config.json

@@ -0,0 +1,22 @@
+{
+    "mode": "QUANTIZE",
+    "observer": "maxabs",
+    "scale_method": "maxabs_hw",
+    "scale_format": "const",
+    "allowlist": {
+        "types": [],
+        "names": []
+    },
+    "blocklist": {
+        "types": [],
+        "names": [
+            "lm_head",
+            "mlp\\.gate\\b",
+            "matmul_qk",
+            "matmul_av",
+            "batch2block_matmul",
+            "block2batch_matmul"
+        ]
+    },
+    "dump_stats_path": "./nc_workspace_measure_kvache/inc_measure_output"
+}

scripts/run_example_tp.py

@@ -25,6 +25,7 @@
 parser.add_argument("--isl", type=int, default=1024, help="input sequence length.")
 parser.add_argument("--osl", type=int, default=1024, help="output sequence length.")
 parser.add_argument("--nprompts", type=int, default=4, help="The number of prompts.")
+parser.add_argument("--max_num_seqs", type=int, default=None, help="The max number of sequences.")
 parser.add_argument("--random", action="store_true", help="Randomly sample prompts.")
 parser.add_argument("--fp8_kv_cache", action="store_true", help="Use fp8 for kv cache.")
 args = parser.parse_args()
@@ -160,6 +161,70 @@ def sample_gsm8k_requests(
             tokenizer=tokenizer,
             do_random=args.random,
         )
+    elif args.dataset == "pile":
+
+        def reset_seed(seed=42):
+            import torch
+            import random
+            import numpy as np
+
+            torch.manual_seed(seed)
+            np.random.seed(seed)
+            random.seed(seed)
+
+        def get_prompt_token_ids(model_path, prompts, max_length=1024):
+            from transformers import AutoTokenizer
+
+            tokenizer = AutoTokenizer.from_pretrained(model_path)
+            prompt_token_ids = []
+            for prompt in prompts:
+                tokens = tokenizer(
+                    prompt,
+                    return_tensors="pt",
+                    truncation=True,
+                    max_length=max_length,
+                )
+                if len(tokens.input_ids[0]) < max_length:
+                    continue
+                prompt_token_ids.append([x.item() for x in tokens.input_ids[0]])
+            return prompt_token_ids
+
+        def get_pile_prompts(model_name, num_samples=512):
+            from datasets import load_dataset
+            from tqdm import tqdm
+            import transformers
+
+            least_tokens = 1024
+            seed = 42
+
+            reset_seed(seed)
+
+            dataset = load_dataset("NeelNanda/pile-10k", split="train")
+            dataset = dataset.shuffle(seed=seed)
+
+            tokenizer = transformers.AutoTokenizer.from_pretrained(
+                model_name, trust_remote_code=True
+            )
+            num_sample = 0
+            samples_lst = []
+            for data in tqdm(dataset):
+                prompt = data["text"]
+                tokens = tokenizer(prompt, return_tensors="pt")
+                if len(tokens.input_ids[0]) < least_tokens:
+                    continue
+                num_sample += 1
+                samples_lst.append(prompt)
+                if num_sample >= num_samples:
+                    break
+            return samples_lst
+        least_tokens = args.isl
+        num_samples = args.nprompts
+        prompts = get_pile_prompts(args.model, num_samples)
+        prompt_token_ids = get_prompt_token_ids(
+            args.model, prompts, least_tokens
+        )
+        print(f"Got {len(prompts)} prompts, length of first prompt: {len(prompt_token_ids[0])}.")
+        gt = None
     else:
         prompts = [
             "Hello, my name is",
@@ -178,6 +243,8 @@ def sample_gsm8k_requests(
     param = {}
     if args.fp8_kv_cache:
         param["kv_cache_dtype"] = "fp8_inc"
+    if args.max_num_seqs is not None:
+        param["max_num_seqs"] = args.max_num_seqs
     if args.tp_size == 1:
         llm = LLM(
             model=model, 
@@ -204,7 +271,12 @@ def sample_gsm8k_requests(
     # Generate texts from the prompts. The output is a list of RequestOutput objects
     # that contain the prompt, generated text, and other information.
     start = time.perf_counter()
-    outputs = llm.generate(prompts, sampling_params)
+    if args.dataset == "pile":
+        outputs = llm.generate(
+            prompts=None, sampling_params=sampling_params, prompt_token_ids=prompt_token_ids
+        )
+    else:
+        outputs = llm.generate(prompts, sampling_params)
     end = time.perf_counter()
     # Print the outputs.
     print(f"e2e took {end - start} seconds")
@@ -218,4 +290,6 @@ def sample_gsm8k_requests(
         print(f"Generated text: {generated_text!r}")
         print(f"Ground truth: {gt_i!r}")
         print("====================================")
+    if os.getenv("VLLM_REQUANT_FP8_INC", None) is not None:
+        llm.llm_engine.model_executor.shutdown()
     del llm

scripts/run_lm_eval.py

@@ -16,8 +16,9 @@
 parser.add_argument("--tokenizer", type=str, default=None, help="The model path.")
 parser.add_argument("--tp_size", type=int, default=8, help="Tensor Parallelism size.")
 parser.add_argument("--ep_size", type=int, default=8, help="Expert Parallelism size.")
-parser.add_argument("-l", "--limit", type=int, default=64, help="test request counts.")
+parser.add_argument("-l", "--limit", type=int, default=None, help="test request counts.")
 parser.add_argument("--batch_size", type=int, default=1, help="The batch size.")
+parser.add_argument("--fp8_kv_cache", action="store_true", help="Use fp8 for kv cache.")
 args = parser.parse_args()
 
 os.environ["VLLM_SKIP_WARMUP"] = "true"
@@ -44,6 +45,9 @@
     model = args.model
     if args.tokenizer is None:
         args.tokenizer = model
+    param = {}
+    if args.fp8_kv_cache:
+        param["kv_cache_dtype"] = "fp8_inc"
     if args.tp_size == 1:
         llm = VLLM(
             pretrained=model, 
@@ -65,17 +69,25 @@
             dtype="bfloat16",
             gpu_memory_utilization=0.8,
             batch_size=args.batch_size,
+            **param,
         )
 
 
     # Run the evaluation; you can adjust num_fewshot and batch_size as needed.
     start = time.perf_counter()
     if args.task == "gsm8k":
-        results = simple_evaluate(model=llm, tasks=["gsm8k"], num_fewshot=5, batch_size=8, limit=args.limit)
+        from lm_eval.utils import make_table
+
+        results = simple_evaluate(
+            model=llm,
+            tasks=["gsm8k"],
+            limit=args.limit,
+        )
         end = time.perf_counter()
         e2e = end - start
+        print(make_table(results))
         # save as json
-        with open(f"gsm8k_ep{args.ep_size}_result_samples_limit{args.limit}.jsonl", "w") as f:
+        with open(f"gsm8k_ep{args.ep_size}_result_samples_limit{str(args.limit)}.jsonl", "w") as f:
             json.dump(results['results'], f)
             json.dump({"e2e time(secs)": e2e}, f)
             f.write("\n")
@@ -86,7 +98,7 @@
         results = simple_evaluate(model=llm, tasks=["hellaswag"], num_fewshot=0, batch_size=8, limit=args.limit)
         end = time.perf_counter()
         e2e = end - start
-        with open(f"hallaswag_ep{args.ep_size}_result_samples_limit{args.limit}.jsonl", "w") as f:
+        with open(f"hallaswag_ep{args.ep_size}_result_samples_limit{str(args.limit)}.jsonl", "w") as f:
             json.dump(results['results'], f)
             json.dump({"e2e time(secs)": e2e}, f)
             f.write("\n")