File tree 2 files changed +8
-10
lines changed
docs/source/features/quantization
2 files changed +8
-10
lines changed Original file line number Diff line number Diff line change @@ -54,16 +54,15 @@ The quantization process involves three main steps:
54
54
55
55
### 1. Loading the Model
56
56
57
- Use ` SparseAutoModelForCausalLM ` , which wraps ` AutoModelForCausalLM ` , for saving and loading quantized models :
57
+ Load your model and tokenizer using the standard ` transformers ` AutoModel classes :
58
58
59
59
``` python
60
- from llmcompressor.transformers import SparseAutoModelForCausalLM
61
- from transformers import AutoTokenizer
60
+ from transformers import AutoTokenizer, AutoModelForCausalLM
62
61
63
62
MODEL_ID = " meta-llama/Meta-Llama-3-8B-Instruct"
64
-
65
- model = SparseAutoModelForCausalLM.from_pretrained(
66
- MODEL_ID , device_map = " auto " , torch_dtype = " auto " )
63
+ model = AutoModelForCausalLM.from_pretrained(
64
+ MODEL_ID , device_map = " auto " , torch_dtype = " auto " ,
65
+ )
67
66
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID )
68
67
```
69
68
Original file line number Diff line number Diff line change @@ -30,14 +30,13 @@ The quantization process involves four main steps:
30
30
31
31
### 1. Loading the Model
32
32
33
- Use ` SparseAutoModelForCausalLM ` , which wraps ` AutoModelForCausalLM ` , for saving and loading quantized models :
33
+ Load your model and tokenizer using the standard ` transformers ` AutoModel classes :
34
34
35
35
``` python
36
- from llmcompressor.transformers import SparseAutoModelForCausalLM
37
- from transformers import AutoTokenizer
36
+ from transformers import AutoTokenizer, AutoModelForCausalLM
38
37
39
38
MODEL_ID = " meta-llama/Meta-Llama-3-8B-Instruct"
40
- model = SparseAutoModelForCausalLM .from_pretrained(
39
+ model = AutoModelForCausalLM .from_pretrained(
41
40
MODEL_ID , device_map = " auto" , torch_dtype = " auto" ,
42
41
)
43
42
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID )
You can’t perform that action at this time.
0 commit comments