How to dequantize a specific layer from gguf into torch tensor?

[SweetGUOguo]

Hello, I am a beginner, and I appreciate your work. I would like to ask, if I want to dequantize a specific layer of a GGUF model and save it as a PyTorch tensor, how should I proceed? I tried step-by-step debugging of llama.cpp/examples/main/main.cpp, but it was difficult to achieve the desired result. Thank you!

[ejrydhfs]

Do you mean extract? Quantization is a lossy process because detail is lost so it is not reversible.

[SweetGUOguo]

Hi, thank you. And I want to see the gap between the quantization value and the before value.

[ejrydhfs]

This could be a feature request here: https://github.com/gpustack/gguf-parser-go/issues

[compilade]

I would like to ask, if I want to dequantize a specific layer of a GGUF model and save it as a PyTorch tensor, how should I proceed?

You can use the gguf-py library, which has dequantization support for most of the quantization types.

$ pip install gguf

(Assuming you also have PyTorch in your environment)

import gguf
import torch

reader: gguf.GGUFReader = gguf.GGUFReader("path/to/model.gguf")
# Assuming you know which tensor it is
index: int = 3
tensor: gguf.ReaderTensor = reader.get_tensor(index)
torch_tensor = torch.from_numpy(gguf.dequantize(tensor.data, tensor.tensor_type))

I did not test this, but hopefully this is close enough to what is needed.

Hmm, I think this should be made more convenient eventually. Notably, searching a tensor by name could be made easier (ideally there should be a way to get a dict[str, np.ndarray] or something).

[SweetGUOguo]

Thanks a lot!!!! It does help!

[SweetGUOguo]

Hello I have one more question, I wonder if there is a similar method for quantizing a specific layer of a .npy to quantize into q4k. Because when I dive into 'gguf/quants.py', there is no implements for q4k? Will this be supported in the future?