Adds support for PTQ through the PyTorch to_backend api. #398

narendasan · 2021-03-16T23:20:58Z

Description

Adds support for PTQ through the PyTorch to_backend api.

Type of change

Please delete options that are not relevant and/or add your own.

New feature (non-breaking change which adds functionality)

Checklist:

My code follows the style guidelines of this project (You can use the linters)
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas and hacks
I have made corresponding changes to the documentation
I have added tests to verify my fix or my feature
New and existing unit tests pass locally with my changes

INT8 calibrators Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>

tests Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>

Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>

github-actions

Code conforms to C++ style guidelines

github-actions

There are some changes that do not conform to Python style guidelines:

Reformatting /workspace/docsrc/conf.py
--- /workspace/tests/py/test_ptq_trt_calibrator.py	(original)
+++ /workspace/tests/py/test_ptq_trt_calibrator.py	(reformatted)
@@ -10,7 +10,9 @@
import torchvision.transforms as transforms
from model_test_case import ModelTestCase

+
class TRTEntropyCalibrator(trt.IInt8EntropyCalibrator2):
+
    def __init__(self, dataloader, **kwargs):
        trt.IInt8EntropyCalibrator2.__init__(self)

@@ -40,7 +42,6 @@
            batch = batch[0].to(self.device)
        return [batch.data_ptr()]

-
    def read_calibration_cache(self):
        # If there is a cache, use it instead of calibrating again. Otherwise, implicitly return None.
        if self.use_cache:
@@ -51,6 +52,7 @@
        if self.cache_file:
            with open(self.cache_file, "wb") as f:
                f.write(cache)
+

class TestAccuracy(ModelTestCase):

Reformatting /workspace/tests/py/test_to_backend_api.py
Reformatting /workspace/tests/py/model_test_case.py
Reformatting /workspace/tests/py/test_api.py
Reformatting /workspace/tests/py/test_ptq_trt_calibrator.py
Reformatting /workspace/tests/py/test_api_dla.py
Reformatting /workspace/tests/py/test_multi_gpu.py
Reformatting /workspace/tests/modules/hub.py
Reformatting /workspace/tests/py/test_ptq_to_backend.py
Reformatting /workspace/tests/py/test_ptq_dataloader_calibrator.py
ERROR: Some files do not conform to style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

There are some changes that do not conform to Python style guidelines:

Reformatting /workspace/docsrc/conf.py
--- /workspace/tests/py/test_ptq_trt_calibrator.py	(original)
+++ /workspace/tests/py/test_ptq_trt_calibrator.py	(reformatted)
@@ -10,7 +10,9 @@
import torchvision.transforms as transforms
from model_test_case import ModelTestCase

+
class TRTEntropyCalibrator(trt.IInt8EntropyCalibrator2):
+
    def __init__(self, dataloader, **kwargs):
        trt.IInt8EntropyCalibrator2.__init__(self)

@@ -40,7 +42,6 @@
            batch = batch[0].to(self.device)
        return [batch.data_ptr()]

-
    def read_calibration_cache(self):
        # If there is a cache, use it instead of calibrating again. Otherwise, implicitly return None.
        if self.use_cache:
@@ -51,6 +52,7 @@
        if self.cache_file:
            with open(self.cache_file, "wb") as f:
                f.write(cache)
+

class TestAccuracy(ModelTestCase):

Reformatting /workspace/tests/py/test_to_backend_api.py
Reformatting /workspace/tests/py/model_test_case.py
Reformatting /workspace/tests/py/test_api.py
Reformatting /workspace/tests/py/test_ptq_trt_calibrator.py
Reformatting /workspace/tests/py/test_ptq_to_backend.py
Reformatting /workspace/tests/py/test_ptq_dataloader_calibrator.py
Reformatting /workspace/tests/modules/hub.py
Reformatting /workspace/tests/py/test_api_dla.py
Reformatting /workspace/tests/py/test_multi_gpu.py
ERROR: Some files do not conform to style guidelines

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

peri044

LGTM

peri044 · 2021-03-17T01:26:02Z

tests/py/test_ptq_to_backend.py

+        with torch.no_grad():
+            idx = 0
+            for data, labels in testing_dataloader:
+                data, labels = data.cuda(), labels.cuda(non_blocking=True)


data.to(device) maybe to avoid warnings ?

this shouldnt throw warnings since its using the new api. the .cuda api wasnt deprecated it was just the async flag

peri044 · 2021-03-17T01:29:34Z

tests/py/test_ptq_to_backend.py

+
+def test_suite():
+    suite = unittest.TestSuite()
+    suite.addTest(TestAccuracy.parametrize(TestAccuracy, model=torch.jit.load('./trained_vgg16.jit.pt')))


Can you please add this comment here and also to test_ptq_dataloader_calibrator.py as well ?

# You need a pre-trained VGG cifar10 model to run this test. Please follow instructions at # https://github.com/NVIDIA/TRTorch/tree/master/cpp/ptq/training/vgg16 to export this model.

I added it to test_ptq_trt_calibrator.py but forgot it at other place.

peri044 · 2021-03-17T01:36:21Z

py/trtorch/csrc/tensorrt_classes.h

@@ -94,10 +94,18 @@ struct CompileSpec : torch::CustomClassHolder {
    input_ranges.push_back(*ir);
  }

+  int64_t getPTQCalibratorHandle() {
+    return (int64_t)ptq_calibrator;


Can you let me know why we do this cast to int64_t and type cast it back in the setPTQCalibratorViaHandle call ?

TorchBind cannot handle pointers as arguments so this was the cheapest way to get a pointer added to the struct. we get the int64_t casted pointer from the original struct and there is void setPTQCalibratorViaHandle(int64_t handle) to set the pointer from an int64_t in a struct owned by torchbind

Both these functions dont get exposed to the user, they are purely used internally

test Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>

github-actions

Code conforms to Python style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

There are some changes that do not conform to Python style guidelines:

Reformatting /workspace/docsrc/conf.py
Reformatting /workspace/tests/modules/hub.py
Reformatting /workspace/tests/py/test_ptq_to_backend.py
--- /workspace/tests/py/test_ptq_dataloader_calibrator.py	(original)
+++ /workspace/tests/py/test_ptq_dataloader_calibrator.py	(reformatted)
@@ -82,7 +82,7 @@
def test_suite():
    suite = unittest.TestSuite()
    # You need a pre-trained VGG cifar10 model to run this test. Please follow instructions at
-# https://github.com/NVIDIA/TRTorch/tree/master/cpp/ptq/training/vgg16 to export this model.
+    # https://github.com/NVIDIA/TRTorch/tree/master/cpp/ptq/training/vgg16 to export this model.
    suite.addTest(TestAccuracy.parametrize(TestAccuracy, model=torch.jit.load('./trained_vgg16.jit.pt')))

    return suite
Reformatting /workspace/tests/py/test_to_backend_api.py
Reformatting /workspace/tests/py/model_test_case.py
Reformatting /workspace/tests/py/test_api.py
Reformatting /workspace/tests/py/test_ptq_trt_calibrator.py
Reformatting /workspace/tests/py/test_api_dla.py
Reformatting /workspace/tests/py/test_multi_gpu.py
Reformatting /workspace/tests/py/test_ptq_dataloader_calibrator.py
ERROR: Some files do not conform to style guidelines

github-actions

Code conforms to C++ style guidelines

github-actions

Code conforms to Python style guidelines

narendasan and others added 3 commits March 16, 2021 15:50

feat(//py): Allowing people using the PyTorch backend to use TRTorch/TRT

6c3e0ad

INT8 calibrators Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>

Merge branch 'int8_py' of https://github.com/NVIDIA/TRTorch into int8_py

b4484b4

refactor(//tests): Couple of edits and organization for new int8 py

076bab0

tests Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>

narendasan requested a review from peri044 March 16, 2021 23:21

github-actions bot added component: api [Python] Issues re: Python API component: tests Issues re: Tests labels Mar 16, 2021

refactor(//tests): Apply linting

f309262

Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>

github-actions bot approved these changes Mar 16, 2021

View reviewed changes

github-actions bot requested changes Mar 16, 2021

View reviewed changes

github-actions bot approved these changes Mar 16, 2021

View reviewed changes

github-actions bot requested changes Mar 16, 2021

View reviewed changes

github-actions bot approved these changes Mar 16, 2021

View reviewed changes

peri044 requested changes Mar 17, 2021

View reviewed changes

chore(//tests): Adding instructions on where to get model for PTQ python

088d586

test Signed-off-by: Naren Dasan <[email protected]> Signed-off-by: Naren Dasan <[email protected]>

github-actions bot approved these changes Mar 17, 2021

View reviewed changes

narendasan force-pushed the to_backend_int8 branch 2 times, most recently from e19fe2e to 088d586 Compare March 17, 2021 19:46

github-actions bot approved these changes Mar 17, 2021

View reviewed changes

github-actions bot requested changes Mar 17, 2021

View reviewed changes

github-actions bot approved these changes Mar 17, 2021

View reviewed changes

narendasan merged commit a4e40ca into int8_py Mar 17, 2021

narendasan deleted the to_backend_int8 branch March 17, 2021 19:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds support for PTQ through the PyTorch to_backend api. #398

Adds support for PTQ through the PyTorch to_backend api. #398

narendasan commented Mar 16, 2021

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

peri044 left a comment

peri044 Mar 17, 2021

narendasan Mar 17, 2021

peri044 Mar 17, 2021

peri044 Mar 17, 2021

narendasan Mar 17, 2021

narendasan Mar 17, 2021

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

github-actions bot left a comment

Adds support for PTQ through the PyTorch to_backend api. #398

Adds support for PTQ through the PyTorch to_backend api. #398

Conversation

narendasan commented Mar 16, 2021

Description

Type of change

Checklist:

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

peri044 left a comment

Choose a reason for hiding this comment

peri044 Mar 17, 2021

Choose a reason for hiding this comment

narendasan Mar 17, 2021

Choose a reason for hiding this comment

peri044 Mar 17, 2021

Choose a reason for hiding this comment

peri044 Mar 17, 2021

Choose a reason for hiding this comment

narendasan Mar 17, 2021

Choose a reason for hiding this comment

narendasan Mar 17, 2021

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment