[Quant] Add quantization 2.0 document #2354

leslie-fang-intel · 2023-05-30T02:32:37Z

Description

Add the new document for Quantization 2.0 flow.

Checklist

The issue that is being fixed is referred in the description (see above "Fixes #ISSUE_NUMBER")
Only one issue is addressed in this pull request
Labels from the issue that this PR is fixing are added to this pull request
No unnessessary issues are included into this pull request.

cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10 @ZailiWang @ZhaoqiongZ @Xia-Weiwen @sekahler2 @CaoE @zhuhaozhe @Valentine233

leslie-fang-intel · 2023-05-30T02:36:29Z

@jgong5 @mingfeima @Xia-Weiwen Here is the draft of documents for Quantization 2.0.

netlify · 2023-05-30T02:37:10Z

✅ Deploy Preview for pytorch-tutorials-preview ready!

Name	Link
🔨 Latest commit	`397eaa9`
🔍 Latest deploy log	https://app.netlify.com/sites/pytorch-tutorials-preview/deploys/64838aa111bee10008372c33
😎 Deploy Preview	https://deploy-preview-2354--pytorch-tutorials-preview.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

prototype_source/quantization_2_0_tutotial.rst

svekars · 2023-05-30T16:19:46Z

Please make sure to add a customcarditem to https://github.com/pytorch/tutorials/blob/main/prototype_source/prototype_index.rst

leslie-fang-intel · 2023-05-31T02:16:21Z

Please make sure to add a customcarditem to https://github.com/pytorch/tutorials/blob/main/prototype_source/prototype_index.rst

@svekars Thanks for the suggestion. customcarditem has been added. But I keep seeing some preCI failures, the error message is ERROR: botocore 1.12.25 has requirement urllib3<1.24,>=1.20, but you'll have urllib3 2.0.2 which is incompatible. and Error response from daemon: manifest for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda12.1-cudnn8-py3-gcc7:2449172a5f2d2b623e0baab605db55c4105e35f5 not found: manifest unknown: Requested image not found Do you have any suggestions?

svekars · 2023-05-31T14:16:27Z

We fixed the issue and I re-kicked the build.

svekars

A few editorial suggestions.

prototype_source/quantization_2_0_tutotial.rst

svekars

A few editorial suggestions.

_static/img/quantization/pytorch_quantization_2_0_diagram.png

prototype_source/quantization_2_0_tutotial.rst

leslie-fang-intel · 2023-06-08T02:00:16Z

Hi @jerryzh168, the comments has been addressed. Please help to take a look again. Thanks.

prototype_source/quantization_in_pytorch_2_0_export_tutorial.rst

jerryzh168

LGTM, thanks!

jerryzh168 · 2023-06-08T21:38:56Z

@kimishpatel please take a look again as well

kimishpatel

I left several comments. I feel overall this is not painting the picture of why we are doing new API and thus the motivaiton. HOwever, since I am reviewing this too late, I dont want to block this on my behalf.

prototype_source/quantization_in_pytorch_2_0_export_tutorial.rst

leslie-fang-intel · 2023-06-09T01:54:39Z

Hi @jerryzh168 @kimishpatel, thanks for the suggestions. The comments has been addressed. Please help to take a look again.

jgong5 · 2023-06-09T06:10:52Z

prototype_source/quantization_in_pytorch_2_0_export_tutorial.rst

+-  Currently, in ``QConfig`` we are exposing observer/fake_quant classes as an object for user to configure quantization.
+   This increases the things that user needs to care about, e.g. not only the ``dtype`` but also how the observation should
+   happen. These could potentially be hidden from user to make user interface simpler.


A bit confused by this, I think the new API also has the observer setting in the QuantizationSpec. So it is also part of the new API, right?

Thanks for the comment. In creation of QuantizationSpec, we do still need to specify the class of observer. But I think it's simpler for users to use comparing with FX Quantization Mode.

In FX Quantization Mode, QConfig is created with observer. User needs to learn the constructor and specify the quantization parameters for different observers in order to create a QConfig for his use case.

Now the general quantization parameters such as dtype, qscheme for different observers are decoupled from observer to the QuantizationSpec at user interface. Although user still need to specify observer type, It decreases what's user need to know for different observers.

@jerryzh168 I think you may help to comment more about this bullet.

so in the new quantizer API, the idea is that configurability is controlled by each backend, the quantizer/annotation API is not facing common modeling user directly, it is an API for backend developers or advanced modeling users only. for common users, they will be interacting with each backend specific quantizer only, e.g. I can have a quantizer that only exposes "quantize/not quantize" option to users:

backend_quantizer = BackendQuantizer() # BackendQuantizer is interacting with QuantizationSpec, not modeling users, # modeling user is interacting configurations exposed by the BackendQuantizer only backend_quantizer.enable_quantization() model = prepare_pt2e(model, backend_quantizer) ...

see the graph in the end of motivation section: https://docs.google.com/document/d/1_jjXrdaPbkmy7Fzmo35-r1GnNKL7anYoAnqozjyY-XI/edit#heading=h.jtqauapwj95c for details

prototype_source/quantization_in_pytorch_2_0_export_tutorial.rst

jgong5

LGTM.

prototype_source/quantization_in_pytorch_2_0_export_tutorial.rst

supriyar · 2023-06-09T22:41:46Z

prototype_source/quantization_in_pytorch_2_0_export_tutorial.rst

+way by annotating the appropriate nodes. A backend specific quantizer inherited from base quantizer,
+some methods that need to be implemented:
+
+-  `annotate method <https://github.com/pytorch/pytorch/blob/3e988316b5976df560c51c998303f56a234a6a1f/torch/ao/quantization/_pt2e/quantizer/qnnpack_quantizer.py#L269>`__


is this the only method that needs to be implemented (above description says some methods)? Why create a separate bullet point for it in that case?

Actually there are some other methods needs to be implemented. However, the detail design for the other methods are not fully set per the discussion with @jerryzh168. So we only mention annotate method here which is also the most important method. Sure, I will merge this bullet into previous paragraph for now.

prototype_source/quantization_in_pytorch_2_0_export_tutorial.rst

leslie-fang-intel · 2023-06-09T23:54:14Z

@supriyar Thanks for the comments. I made another PR #2456 to following up your comments. Please help to take a look.

leslie-fang-intel added 3 commits May 25, 2023 17:02

add quantization 2.0 document

7afe407

add example of DerivedQuantizationSpec

0beaa6f

add example of FixedQParamsQuantizationSpec

927ce7d

facebook-github-bot added the cla signed label May 30, 2023

leslie-fang-intel changed the title ~~Add quantization 2.0 document~~ [Quant] Add quantization 2.0 document May 30, 2023

leslie-fang-intel marked this pull request as draft May 30, 2023 02:33

jgong5 reviewed May 30, 2023

View reviewed changes

svekars added the intel label May 30, 2023

leslie-fang-intel added 4 commits May 31, 2023 08:56

add customcarditem to Quantization 2.0

66a5bde

add more into prototype_index

89bccc9

add quantization 2.0 diagram

212e820

Modify the arch diagram

042fbbc

leslie-fang-intel added 4 commits May 31, 2023 10:22

add explain for QuantizationSpec

f1b911e

add explanations for QuantizationSpec and QuantizationConfig

e78bcef

unify to use module partitional API

238e2f0

Modify the descriptation

f7c7747

leslie-fang-intel requested a review from jgong5 May 31, 2023 04:16

leslie-fang-intel marked this pull request as ready for review May 31, 2023 04:16

svekars added the docathon-h1-2023 A label for the docathon in H1 2023 label May 31, 2023

svekars added the advanced label May 31, 2023

svekars reviewed May 31, 2023

View reviewed changes

jerryzh168 reviewed Jun 1, 2023

View reviewed changes

_static/img/quantization/pytorch_quantization_2_0_diagram.png Outdated Show resolved Hide resolved

jerryzh168 reviewed Jun 1, 2023

View reviewed changes

prototype_source/quantization_2_0_tutotial.rst Outdated Show resolved Hide resolved

fix comments

75d7493

update descriptation

ef4fa73

leslie-fang-intel requested a review from jerryzh168 June 8, 2023 01:34

fix typo

5edc1f2

jerryzh168 reviewed Jun 8, 2023

View reviewed changes

prototype_source/quantization_in_pytorch_2_0_export_tutorial.rst Outdated Show resolved Hide resolved

jerryzh168 approved these changes Jun 8, 2023

View reviewed changes

kimishpatel approved these changes Jun 8, 2023

View reviewed changes

leslie-fang-intel added 2 commits June 9, 2023 08:42

Merge branch 'main' into leslie/add_quantization_2_doc

3e632a6

add author

4b86060

add more explain of limitations

5a43584

leslie-fang-intel force-pushed the leslie/add_quantization_2_doc branch 2 times, most recently from f9ee24b to 5a43584 Compare June 9, 2023 02:01

adjust descriptation

d9d3245

jgong5 reviewed Jun 9, 2023

View reviewed changes

jgong5 approved these changes Jun 9, 2023

View reviewed changes

leslie-fang-intel and others added 6 commits June 9, 2023 16:34

Move QuantizationAnnotation to preface

1c0eead

fix descriptation

1fb6c57

fix typo

cfd095a

Merge branch 'main' into leslie/add_quantization_2_doc

ceb9843

Merge branch 'main' into leslie/add_quantization_2_doc

a8dd4b0

Merge branch 'main' into leslie/add_quantization_2_doc

397eaa9

supriyar reviewed Jun 9, 2023

View reviewed changes

prototype_source/quantization_in_pytorch_2_0_export_tutorial.rst Show resolved Hide resolved

supriyar reviewed Jun 9, 2023

View reviewed changes

prototype_source/quantization_in_pytorch_2_0_export_tutorial.rst Show resolved Hide resolved

supriyar reviewed Jun 9, 2023

View reviewed changes

prototype_source/quantization_in_pytorch_2_0_export_tutorial.rst Show resolved Hide resolved

svekars merged commit 0ef9a65 into pytorch:main Jun 9, 2023

leslie-fang-intel mentioned this pull request Jun 9, 2023

modify quantization in pytorch.2.0 export tutorial #2456

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Quant] Add quantization 2.0 document #2354

[Quant] Add quantization 2.0 document #2354

leslie-fang-intel commented May 30, 2023 •

edited by pytorch-bot bot

Loading

leslie-fang-intel commented May 30, 2023

netlify bot commented May 30, 2023 •

edited

Loading

svekars commented May 30, 2023

leslie-fang-intel commented May 31, 2023 •

edited

Loading

svekars commented May 31, 2023

svekars left a comment

svekars left a comment

leslie-fang-intel commented Jun 8, 2023

jerryzh168 left a comment

jerryzh168 commented Jun 8, 2023

kimishpatel left a comment

leslie-fang-intel commented Jun 9, 2023

jgong5 Jun 9, 2023

leslie-fang-intel Jun 9, 2023 •

edited

Loading

jerryzh168 Jun 9, 2023 •

edited

Loading

jgong5 left a comment

supriyar Jun 9, 2023

leslie-fang-intel Jun 9, 2023 •

edited

Loading

leslie-fang-intel commented Jun 9, 2023

[Quant] Add quantization 2.0 document #2354

[Quant] Add quantization 2.0 document #2354

Conversation

leslie-fang-intel commented May 30, 2023 • edited by pytorch-bot bot Loading

Description

Checklist

leslie-fang-intel commented May 30, 2023

netlify bot commented May 30, 2023 • edited Loading

✅ Deploy Preview for pytorch-tutorials-preview ready!

svekars commented May 30, 2023

leslie-fang-intel commented May 31, 2023 • edited Loading

svekars commented May 31, 2023

svekars left a comment

Choose a reason for hiding this comment

svekars left a comment

Choose a reason for hiding this comment

leslie-fang-intel commented Jun 8, 2023

jerryzh168 left a comment

Choose a reason for hiding this comment

jerryzh168 commented Jun 8, 2023

kimishpatel left a comment

Choose a reason for hiding this comment

leslie-fang-intel commented Jun 9, 2023

jgong5 Jun 9, 2023

Choose a reason for hiding this comment

leslie-fang-intel Jun 9, 2023 • edited Loading

Choose a reason for hiding this comment

jerryzh168 Jun 9, 2023 • edited Loading

Choose a reason for hiding this comment

jgong5 left a comment

Choose a reason for hiding this comment

supriyar Jun 9, 2023

Choose a reason for hiding this comment

leslie-fang-intel Jun 9, 2023 • edited Loading

Choose a reason for hiding this comment

leslie-fang-intel commented Jun 9, 2023

leslie-fang-intel commented May 30, 2023 •

edited by pytorch-bot bot

Loading

netlify bot commented May 30, 2023 •

edited

Loading

leslie-fang-intel commented May 31, 2023 •

edited

Loading

leslie-fang-intel Jun 9, 2023 •

edited

Loading

jerryzh168 Jun 9, 2023 •

edited

Loading

leslie-fang-intel Jun 9, 2023 •

edited

Loading